the time allocation of using battery energy and solar energy can extend the
flight time of SUAV [223]. In this case, another control variable is born that
increases the difficulty of the problem. The next topic can thus be introduced
from this, the Hybrid UAVs. There have been some studies on hybrid UAVs using
solar energy and some other forms of energy sources. For example, fossil fuel
and solar powered UAV [224], wind and solar energy powered UAV [225] and RF
and solar energy powered UAV [226, 227]. The energy management method is worth
being researched on hybrid UAVs.

####  VII-B9 Multi-SUAVs Self-collision Avoidance

Another important challenge in navigation of SUAV teams is that the developed
navigation algorithm should also guarantee avoiding collisions between SUAVs.
This make involved optimization problems far more difficult as the dimension
of the optimization space increases in $n$ times, where $n$ is the number of
SUAVs in the team. This greatly increases the computational complexity of
navigation algorithms, which is similar to [74]. All in all, this research
direction is still an immature field, with many novel applications and
algorithms to be developed and investigated.

Overall, the development of unmanned systems and renewable energy has been
rapid during the last few decades. Researching the navigation algorithms for
SUAVs has the potential to help with the development of a sustainable future.
There is no doubt that more research findings can accelerate the growth of
autonomous UAVs, potentially reducing labour work in the near future.

####  VII-B10 Energy Harvesting Model of SUAVs

Proper energy harvesting model is an essential factor that build the
foundation of the energy-aware path planning algorithm. Among all the
aforementioned path planning algorithm of SUAVs, the assumption was made that
SUAVs cannot harvest solar energy under the shadow area. However, this does
not fully reflect reality. The energy absorbed should be directly related to
the light intensity, the shadow is just a more intuitive dividing line. Since
it is not possible to quantify and simulate the light intensity easily, the
concept of shadow areas is introduced. SUAVs are still possible to harvest
solar energy from the dispersed sunlight to some degree. A sigmoid function is
a potential solution for mimicking their relation more accurately. More
research is needed to find a better energy harvesting model.

## References

  * [1] J. M. Sullivan, “Evolution or revolution? The rise of UAVs,” _IEEE Technology and Society Magazine_ , vol. 25, no. 3, pp. 43–49, 2006.
  * [2] S. K. Khan, U. Naseem, H. Siraj, I. Razzak, and M. Imran, “The role of unmanned aerial vehicles and mmwave in 5G: Recent advances and challenges,” _Transactions on Emerging Telecommunications Technologies_ , vol. 32, no. 7, p. e4241, 2021.
  * [3] Y. Zeng, R. Zhang, and T. J. Lim, “Wireless communications with unmanned aerial vehicles: Opportunities and challenges,” _IEEE Communications magazine_ , vol. 54, no. 5, pp. 36–42, 2016.
  * [4] A. Merwaday and I. Guvenc, “UAV assisted heterogeneous networks for public safety communications,” in _2015 IEEE wireless communications and networking conference workshops (WCNCW)_. IEEE, 2015, pp. 329–334.
  * [5] S. Asadzadeh, W. J. de Oliveira, and C. R. de Souza Filho, “UAV-based remote sensing for the petroleum industry and environmental monitoring: State-of-the-art and perspectives,” _Journal of Petroleum Science and Engineering_ , vol. 208, p. 109633, 2022.
  * [6] D. R. Green, J. J. Hagon, C. Gómez, and B. J. Gregory, “Using low-cost UAVs for environmental monitoring, mapping, and modelling: Examples from the coastal zone,” in _Coastal management_. Elsevier, 2019, pp. 465–501.
  * [7] R. Eskandari, M. Mahdianpari, F. Mohammadimanesh, B. Salehi, B. Brisco, and S. Homayouni, “Meta-analysis of unmanned aerial vehicle (UAV) imagery for agro-environmental monitoring using machine learning and statistical models,” _Remote Sensing_ , vol. 12, no. 21, p. 3511, 2020.
  * [8] L. P. Osco, J. M. Junior, A. P. M. Ramos, L. A. de Castro Jorge, S. N. Fatholahi, J. de Andrade Silva, E. T. Matsubara, H. Pistori, W. N. Gonçalves, and J. Li, “A review on deep learning in UAV remote sensing,” _International Journal of Applied Earth Observation and Geoinformation_ , vol. 102, p. 102456, 2021.
  * [9] J. Everaerts _et al._ , “The use of unmanned aerial vehicles (UAVs) for remote sensing and mapping,” _The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences_ , vol. 37, no. 2008, pp. 1187–1192, 2008.
  * [10] A. Ramachandran and A. K. Sangaiah, “A review on object detection in unmanned aerial vehicle surveillance,” _International Journal of Cognitive Computing in Engineering_ , vol. 2, pp. 215–228, 2021.
  * [11] D. Dissanayaka, T. R. Wanasinghe, O. De Silva, A. Jayasiri, and G. K. Mann, “Review of navigation methods for UAV-based parcel delivery,” _IEEE Transactions on Automation Science and Engineering_ , 2023.
  * [12] A. Li, M. Hansen, and B. Zou, “Traffic management and resource allocation for UAV-based parcel delivery in low-altitude urban space,” _Transportation Research Part C: Emerging Technologies_ , vol. 143, p. 103808, 2022.
  * [13] D. Wang, P. Hu, J. Du, P. Zhou, T. Deng, and M. Hu, “Routing and scheduling for hybrid truck-drone collaborative parcel delivery with independent and truck-carried drones,” _IEEE Internet of Things Journal_ , vol. 6, no. 6, pp. 10 483–10 495, 2019.
  * [14] L. Quan, L. Han, B. Zhou, S. Shen, and F. Gao, “Survey of UAV motion planning,” _IET Cyber-systems and Robotics_ , vol. 2, no. 1, pp. 14–21, 2020\.
  * [15] P. Oettershagen, A. Melzer, T. Mantel, K. Rudin, T. Stastny, B. Wawrzacz, T. Hinzmann, S. Leutenegger, K. Alexis, and R. Siegwart, “Design of small hand-launched solar-powered UAVs: From concept study to a multi-day world endurance record flight,” _Journal of Field Robotics_ , vol. 34, no. 7, pp. 1352–1377, 2017.
  * [16] N. El-Atab, R. B. Mishra, R. Alshanbari, and M. M. Hussain, “Solar powered small unmanned aerial vehicles: a review,” _Energy Technology_ , vol. 9, no. 12, p. 2100587, 2021.
  * [17] _Global Solar-powered UAV Market 2020-2024_. TechNavio, 2020.
  * [18] W. Engblom, “Novel Approach to High-Altitude, Long-Endurance Stationkeeping,” in _30th AIAA applied aerodynamics conference_ , 2012, p. 3203.
  * [19] M. Harris, “Project Skybender: Google’s secretive 5G internet drone tests revealed,” _the Guardian_ , vol. 29, 2016.
  * [20] (2023) Zephyr HAPS. [Online]. Available: https://www.airbus.com/en/products-services/defence/uas/uas-solutions/zephyr
  * [21] B. Patle, A. Pandey, D. Parhi, A. Jagadeesh _et al._ , “A review: On path planning strategies for navigation of mobile robot,” _Defence Technology_ , vol. 15, no. 4, pp. 582–606, 2019.
  * [22] S. M. LaValle, _Planning algorithms_. Cambridge university press, 2006.
  * [23] A. V. Goldberg and T. Radzik, _A heuristic improvement of the Bellman-Ford algorithm_. Citeseer, 1993\.
  * [24] D. Harabor and A. Grastien, “Improving jump point search,” in _Proceedings of the International Conference on Automated Planning and Scheduling_ , vol. 24, 2014, pp. 128–135.
  * [25] J. Petereit, T. Emter, C. W. Frey, T. Kopfstedt, and A. Beutel, “Application of hybrid A* to an autonomous mobile robot for path planning in unstructured outdoor environments,” in _ROBOTIK 2012; 7th German Conference on Robotics_. VDE, 2012, pp. 1–6.
  * [26] A. Stentz, _The D* algorithm for real-time planning of optimal traverses_. Carnegie Mellon University, the Robotics Institute, 1994.
  * [27] D. Dolgov, S. Thrun, M. Montemerlo, and J. Diebel, “Practical search techniques in path planning for autonomous driving,” _Ann Arbor_ , vol. 1001, no. 48105, pp. 18–80, 2008.
  * [28] L. Yang, J. Qi, J. Xiao, and X. Yong, “A literature review of UAV 3D path planning,” in _Proceeding of the 11th World Congress on Intelligent Control and Automation_. IEEE, 2014, pp. 2376–2381.
  * [29] S. LaValle, “Rapidly-exploring random trees: A new tool for path planning,” _Research Report 9811_ , 1998.
  * [30] S. Karaman and E. Frazzoli, “Sampling-based algorithms for optimal motion planning,” _The international journal of robotics research_ , vol. 30, no. 7, pp. 846–894, 2011.
  * [31] I. Noreen, A. Khan, and Z. Habib, “A comparison of RRT, RRT* and RRT*-smart path planning algorithms,” _International Journal of Computer Science and Network Security (IJCSNS)_ , vol. 16, no. 10, p. 20, 2016\.
  * [32] J. Kennedy and R. Eberhart, “Particle swarm optimization,” in _Proceedings of ICNN’95-international conference on neural networks_ , vol. 4. IEEE, 1995, pp. 1942–1948.
  * [33] M. Mitchell, _An introduction to genetic algorithms_. MIT press, 1998.
  * [34] M. Dorigo, M. Birattari, and T. Stutzle, “Ant colony optimization,” _IEEE computational intelligence magazine_ , vol. 1, no. 4, pp. 28–39, 2006\.
  * [35] R. Glasius, A. Komoda, and S. C. Gielen, “Neural network dynamics for path planning and obstacle avoidance,” _Neural Networks_ , vol. 8, no. 1, pp. 125–133, 1995.
  * [36] L. Dorst and K. Trovato, “Optimal path planning by cost wave propagation in metric configuration space,” in _Mobile Robots III_ , vol. 1007. SPIE, 1989, pp. 186–197.
  * [37] X. Xu, M. WANG, and Y. MAO, “Path planning of mobile robot based on improved artificial potential field method,” _Journal of Computer Applications_ , vol. 40, no. 12, p. 3508, 2020.
  * [38] H. Sang, Y. You, X. Sun, Y. Zhou, and F. Liu, “The hybrid path planning algorithm based on improved A* and artificial potential field for unmanned surface vehicle formations,” _Ocean Engineering_ , vol. 223, p. 108709, 2021\.
  * [39] B. Wang, Z. Liu, Q. Li, and A. Prorok, “Mobile robot path planning in dynamic environments through globally guided reinforcement learning,” _IEEE Robotics and Automation Letters_ , vol. 5, no. 4, pp. 6932–6939, 2020.
  * [40] X. Xiao, B. Liu, G. Warnell, and P. Stone, “Motion planning and control for mobile robot navigation using machine learning: a survey,” _Autonomous Robots_ , vol. 46, no. 5, pp. 569–597, 2022.
  * [41] X.-H. Liu, D.-G. Zhang, H.-R. Yan, Y.-y. Cui, and L. Chen, “A new algorithm of the best path selection based on machine learning,” _IEEE Access_ , vol. 7, pp. 126 913–126 928, 2019.
  * [42] M. Hoy, A. S. Matveev, and A. V. Savkin, “Algorithms for collision-free navigation of mobile robots in complex cluttered environments: a survey,” _Robotica_ , vol. 33, no. 3, pp. 463–497, 2015.
  * [43] C. Wang, A. V. Savkin, and M. Garratt, “A strategy for safe 3D navigation of non-holonomic robots among moving obstacles,” _Robotica_ , vol. 36, no. 2, p. 275–297, 2018.
  * [44] A. V. Savkin and M. Hoy, “Reactive and the shortest path navigation of a wheeled mobile robot in cluttered environments,” _Robotica_ , vol. 31, no. 2, p. 323–330, 2013.
  * [45] J. Sfeir, M. Saad, and H. Saliah-Hassane, “An improved artificial potential field approach to real-time mobile robot path planning in an unknown environment,” in _2011 IEEE international symposium on robotic and sensors environments (ROSE)_. IEEE, 2011, pp. 208–213.
  * [46] S. M. H. Rostami, A. K. Sangaiah, J. Wang, and X. Liu, “Obstacle avoidance of mobile robots using modified artificial potential field algorithm,” _EURASIP Journal on Wireless Communications and Networking_ , vol. 2019, no. 1, pp. 1–19, 2019.
  * [47] J. M. Toibero, F. Roberti, and R. Carelli, “Stable contour-following control of wheeled mobile robots,” _Robotica_ , vol. 27, no. 1, pp. 1–12, 2009.
  * [48] A. S. Matveev, C. Wang, and A. V. Savkin, “Real-time navigation of mobile robots in problems of border patrolling and avoiding collisions with moving and deforming obstacles,” _Robotics and Autonomous systems_ , vol. 60, no. 6, pp. 769–788, 2012.
  * [49] A. S. Matveev, H. Teimoori, and A. V. Savkin, “A method for guidance and control of an autonomous vehicle in problems of border patrolling and obstacle avoidance,” _Automatica_ , vol. 47, no. 3, pp. 515–524, 2011.
  * [50] S. Yang, S. Konam, C. Ma, S. Rosenthal, M. Veloso, and S. Scherer, “Obstacle avoidance through deep networks based intermediate perception,” _arXiv preprint arXiv:1704.08759_ , 2017.
  * [51] L. Dai, Q. Cao, Y. Xia, and Y. Gao, “Distributed MPC for formation of multi-agent systems with collision avoidance and obstacle avoidance,” _Journal of the Franklin Institute_ , vol. 354, no. 4, pp. 2068–2085, 2017\.
  * [52] D. Fox, W. Burgard, and S. Thrun, “The dynamic window approach to collision avoidance,” _IEEE Robotics & Automation Magazine_, vol. 4, no. 1, pp. 23–33, 1997.
  * [53] P. Ogren and N. E. Leonard, “A convergent dynamic window approach to obstacle avoidance,” _IEEE Transactions on Robotics_ , vol. 21, no. 2, pp. 188–195, 2005.
  * [54] R. Simmons, “The curvature-velocity method for local obstacle avoidance,” in _Proceedings of IEEE international conference on robotics and automation_ , vol. 4. IEEE, 1996, pp. 3375–3382.
  * [55] N. Y. Ko and R. G. Simmons, “The lane-curvature method for local obstacle avoidance,” in _Proceedings. 1998 IEEE/RSJ International Conference on Intelligent Robots and Systems. Innovations in Theory, Practice and Applications (Cat. No. 98CH36190)_ , vol. 3. IEEE, 1998, pp. 1615–1621.
  * [56] A. Chakravarthy and D. Ghose, “Obstacle avoidance in a dynamic environment: A collision cone approach,” _IEEE Transactions on Systems, Man, and Cybernetics-Part A: Systems and Humans_ , vol. 28, no. 5, pp. 562–574, 1998.
  * [57] X. Xu, W. Pan, Y. Huang, and W. Zhang, “Dynamic collision avoidance algorithm for unmanned surface vehicles via layered artificial potential field with collision cone,” _The Journal of Navigation_ , vol. 73, no. 6, pp. 1306–1325, 2020.
  * [58] Y. Zhu, T. Zhang, J. Song, and X. Li, “A new hybrid navigation algorithm for mobile robots in environments with incomplete knowledge,” _Knowledge-Based Systems_ , vol. 27, pp. 302–313, 2012.
  * [59] X. Li, X. Hu, Z. Wang, and Z. Du, “Path planning based on combinaion of improved A-star algorithm and DWA algorithm,” in _2020 2nd International Conference on Artificial Intelligence and Advanced Manufacture (AIAM)_. IEEE, 2020, pp. 99–103.
  * [60] S. Li, Z. Fang, S. C. Verma, J. Wei, and A. V. Savkin, “Navigation and Deployment of Solar-Powered Unmanned Aerial Vehicles for Civilian Applications: A Comprehensive Review,” _Drones_ , vol. 8, no. 2, p. 42, 2024\.
  * [61] P. Rajendran and H. Smith, “Implications of longitude and latitude on the size of solar-powered UAV,” _Energy conversion and management_ , vol. 98, pp. 107–114, 2015.
  * [62] S. Pal, A. Mishra, and P. Singh, “Recent trends in long endurance solar powered UAVs: A review,” 2020.
  * [63] X. Zhu, Z. Guo, and Z. Hou, “Solar-powered airplanes: A historical perspective and future challenges,” _Progress in aerospace sciences_ , vol. 71, pp. 36–53, 2014.
  * [64] Y. Lu, Z. Xue, G.-S. Xia, and L. Zhang, “A survey on vision-based UAV navigation,” _Geo-spatial information science_ , vol. 21, no. 1, pp. 21–32, 2018.
  * [65] Y. Chu, C. Ho, Y. Lee, and B. Li, “Development of a solar-powered unmanned aerial vehicle for extended flight endurance,” _Drones_ , vol. 5, no. 2, p. 44, 2021.
  * [66] X. Li and A. V. Savkin, “Networked unmanned aerial vehicles for surveillance and monitoring: A survey,” _Future Internet_ , vol. 13, no. 7, p. 174, 2021\.
  * [67] Y. Huang, J. Chen, H. Wang, and G. Su, “A method of 3D path planning for solar-powered UAV with fixed target and solar tracking,” _Aerospace Science and technology_ , vol. 92, pp. 831–838, 2019.
  * [68] Y. Huang, J. Chen, G. Su, H. Wang, R. Liu, and P. Du, “Energy-optimal path planning for solar-powered UAVs monitoring stationary target,” in _Proceedings of the 4th ACM SIGSPATIAL International Workshop on Safety and Resilience_ , 2018, pp. 1–8.
  * [69] J. Wu, H. Wang, Y. Huang, Z. Su, and M. Zhang, “Energy management strategy for solar-powered UAV long-endurance target tracking,” _IEEE Transactions on Aerospace and Electronic Systems_ , vol. 55, no. 4, pp. 1878–1891, 2018.
  * [70] H. Huang and A. V. Savkin, “Energy-efficient decentralized navigation of a team of solar-powered UAVs for collaborative eavesdropping on a mobile ground target in urban environments,” _Ad Hoc Networks_ , vol. 117, p. 102485, 2021.
  * [71] J. Wu, H. Wang, N. Li, P. Yao, Y. Huang, Z. Su, and Y. Yu, “Distributed trajectory optimization for multiple solar-powered UAVs target tracking in urban environment by adaptive grasshopper optimization algorithm,” _Aerospace Science and Technology_ , vol. 70, pp. 497–510, 2017.
  * [72] Y. Huang, H. Wang, and P. Yao, “Energy-optimal path planning for solar-powered UAV with tracking moving ground target,” _Aerospace Science and Technology_ , vol. 53, pp. 241–251, 2016.
  * [73] S. Hosseinisianaki and M. Mesbahi, “Energy aware aerial surveillance for a long endurance solar-powered UAV,” in _AIAA Guidance, Navigation, and Control (GNC) Conference_ , 2013, p. 4552.
  * [74] H. Huang and A. V. Savkin, “Energy-efficient autonomous navigation of solar-powered UAVs for surveillance of mobile ground targets in urban environments,” _Energies_ , vol. 13, no. 21, 2020.
  * [75] S. Herwitz, L. Johnson, J. Arvesen, R. Higgins, J. Leung, and S. Dunagan, “Precision agriculture as a commercial application for solar-powered unmanned aerial vehicles,” in _1st UAV Conference_ , 2002, p. 3404.
  * [76] S. R. Herwitz, S. Dunagan, D. Sullivan, R. Higgins, L. Johnson, J. Zheng, R. Slye, J. Brass, J. Leung, B. Gallmeyer _et al._ , “Solar-powered UAV mission for agricultural decision support,” in _IGARSS 2003. 2003 IEEE International Geoscience and Remote Sensing Symposium. Proceedings (IEEE Cat. No. 03CH37477)_ , vol. 3. IEEE, 2003, pp. 1692–1694.
  * [77] J. Malaver Rojas, F. Gonzalez, N. Motta, T. F. Villa, V. Etse, and E. Puig Garcia, “Design and flight testing of an integrated solar powered UAV and WSN for greenhouse gas monitoring emissions in agricultural farms,” in _Proceedings of the IROS Workshop on Agri-Food Robotics 2015_. Institute of Electrical and Electronics Engineers Inc., 2015, pp. 1–6.
  * [78] A. Malaver, N. Motta, P. Corke, and F. Gonzalez, “Development and integration of a solar powered unmanned aerial vehicle and a wireless sensor network to monitor greenhouse gases,” _Sensors_ , vol. 15, no. 2, pp. 4072–4096, 2015\.
  * [79] C. Thipyopas, V. Sripawadkul, and N. Warin, “Design and development of a small solar-powered UAV for environmental monitoring application,” in _2019 IEEE Eurasia Conference on IOT, Communication and Engineering (ECICE)_. IEEE, 2019, pp. 316–319.
  * [80] P. Oettershagen, T. Stastny, T. Mantel, A. Melzer, K. Rudin, P. Gohl, G. Agamennoni, K. Alexis, and R. Siegwart, “Long-endurance sensing and mapping using a hand-launchable solar-powered UAV,” in _Field and Service Robotics: Results of the 10th International Conference_. Springer, 2016, pp. 441–454.
  * [81] V. S. Dwivedi, J. Patrikar, A. Addamane, and A. Ghosh, “MARAAL: A low altitude long endurance solar powered UAV for surveillance and mapping applications,” in _2018 23rd International Conference on Methods & Models in Automation & Robotics (MMAR)_. IEEE, 2018, pp. 449–454.
  * [82] M. Karthik, S. Usha, B. Predeep, G. Saran, G. Sridhar, and R. Theeksith, “Design and development of solar powered unmanned aerial vehicle (UAV) for surveying, mapping and disaster relief,” in _AIP Conference Proceedings_ , vol. 2387, no. 1. AIP Publishing, 2021.
  * [83] L. Chettri and R. Bera, “A comprehensive survey on internet of things (IoT) toward 5G wireless systems,” _IEEE Internet of Things Journal_ , vol. 7, no. 1, pp. 16–32, 2019.
  * [84] I. Jawhar, N. Mohamed, J. Al-Jaroodi, D. P. Agrawal, and S. Zhang, “Communication and networking of UAV-based systems: Classification and associated architectures,” _Journal of Network and Computer Applications_ , vol. 84, pp. 93–108, 2017.
  * [85] A. S. Matveev and A. V. Savkin, _Estimation and control over communication networks_. Springer Science & Business Media, 2009.
  * [86] A. V. Savkin, “Analysis and synthesis of networked control systems: Topological entropy, observability, robustness and optimal control,” _Automatica_ , vol. 42, no. 1, pp. 51–62, 2006.
  * [87] A. S. Matveev and A. V. Savkin, “The problem of state estimation via asynchronous communication channels with irregular transmission times,” _IEEE transactions on automatic control_ , vol. 48, no. 4, pp. 670–676, 2003\.
  * [88] R. A. Gupta and M.-Y. Chow, “Networked control system: Overview and research trends,” _IEEE transactions on industrial electronics_ , vol. 57, no. 7, pp. 2527–2535, 2009.
  * [89] T. C. Yang, “Networked control system: a brief survey,” _IEE Proceedings-Control Theory and Applications_ , vol. 153, no. 4, pp. 403–412, 2006\.
  * [90] X. Liang and J. Xu, “Control for networked control systems with remote and local controllers over unreliable communication channel,” _Automatica_ , vol. 98, pp. 86–94, 2018.
  * [91] X.-M. Zhang, Q.-L. Han, X. Ge, D. Ding, L. Ding, D. Yue, and C. Peng, “Networked control systems: A survey of trends and techniques,” _IEEE/CAA Journal of Automatica Sinica_ , vol. 7, no. 1, pp. 1–17, 2019.
  * [92] G. E. G. Padilla, K.-J. Kim, S.-H. Park, and K.-H. Yu, “Flight path planning of solar-powered UAV for sustainable communication relay,” _IEEE Robotics and Automation Letters_ , vol. 5, no. 4, pp. 6772–6779, 2020.
  * [93] X. Song, Z. Chang, X. Guo, P. Wu, and T. Hämäläinen, “Energy efficient optimization for solar-powered UAV communications system,” in _2021 IEEE International Conference on Communications Workshops (ICC Workshops)_. IEEE, 2021, pp. 1–6.
  * [94] H. Huang and A. V. Savkin, “Autonomous navigation of a solar-powered UAV for secure communication in urban environments with eavesdropping avoidance,” _Future Internet_ , vol. 12, no. 10, p. 170, 2020.
  * [95] H. Huang, A. V. Savkin, and W. Ni, “Energy-efficient 3D navigation of a solar-powered UAV for secure communication in the presence of eavesdroppers and no-fly zones,” _Energies_ , vol. 13, no. 6, p. 1445, 2020.
  * [96] Y. Sun, D. W. K. Ng, D. Xu, L. Dai, and R. Schober, “Resource allocation for solar powered UAV communication systems,” in _2018 IEEE 19th International Workshop on Signal Processing Advances in Wireless Communications (SPAWC)_. IEEE, 2018, pp. 1–5.
  * [97] Y. Sun, D. Xu, D. W. K. Ng, L. Dai, and R. Schober, “Optimal 3D-trajectory design and resource allocation for solar-powered UAV communication systems,” _IEEE Transactions on Communications_ , vol. 67, no. 6, pp. 4281–4298, 2019.
  * [98] G. Romeo and G. Frulla, “HELIPLAT®: high altitude very-long endurance solar powered UAV for telecommunication and earth observation applications,” _The Aeronautical Journal_ , vol. 108, no. 1084, pp. 277–293, 2004.
  * [99] G. Romeo, G. Frulla, E. Cestino, and G. Corsino, “Heliplat: design, aerodynamic, structural analysis of long-endurance solar-powered stratospheric platform,” _Journal of Aircraft_ , vol. 41, no. 6, pp. 1505–1520, 2004.
  * [100] W. Woźniak and M. Jessa, “Selection of solar powered unmanned aerial vehicles for a long range data acquisition chain,” _Sensors_ , vol. 21, no. 8, p. 2772, 2021.
  * [101] C. Luo, J. Hu, Y. Hou, Y. Hong, Y. Zhu, and D. Li, “Maximizing the amount of data collected from wsn based on solar-powered UAV in urban environment,” _Journal of Combinatorial Optimization_ , vol. 45, no. 5, p. 132, 2023.
  * [102] K. Dorling, J. Heinrichs, G. G. Messier, and S. Magierowski, “Vehicle routing problems for drone delivery,” _IEEE Transactions on Systems, Man, and Cybernetics: Systems_ , vol. 47, no. 1, pp. 70–85, 2016.
  * [103] J. E. Scott and C. H. Scott, “Drone delivery models for healthcare,” in _Hawaii International Conference on System Sciences_ , 2017. [Online]. Available: https://api.semanticscholar.org/CorpusID:27433971
  * [104] Z. Tian, Z. J. Haas, and S. Shinde, “Routing in solar-powered UAV delivery system,” _Drones_ , vol. 6, no. 10, p. 282, 2022.
  * [105] H. Huang, A. V. Savkin, and C. Huang, “Reliable path planning for drone delivery using a stochastic time-dependent public transportation network,” _IEEE Transactions on Intelligent Transportation Systems_ , vol. 22, no. 8, pp. 4941–4950, 2020.
  * [106] H. Huang and A. V. Savkin, “A method of optimized deployment of charging stations for drone delivery,” _IEEE Transactions on Transportation Electrification_ , vol. 6, no. 2, pp. 510–518, 2020.
  * [107] P. Rajendran and H. Smith, “Review of solar and battery power system development for solar-powered electric unmanned aerial vehicles,” _Advanced Materials Research_ , vol. 1125, pp. 641–647, 2015.
  * [108] S. Morton, R. D’Sa, and N. Papanikolopoulos, “Solar powered UAV: Design and experiments,” in _2015 IEEE/RSJ international conference on intelligent robots and systems (IROS)_. IEEE, 2015, pp. 2460–2466.
  * [109] N. R. Khan, A. V. Raghorte, P. V. Nandankar, and J. A. Waware, “Solar powered UAV: A comprehensive review,” in _AIP Conference Proceedings_ , vol. 2753, no. 1. AIP Publishing, 2023.
  * [110] R. Rysdyk, “UAV path following for constant line-of-sight,” in _2nd AIAA” Unmanned Unlimited” Conf. and Workshop & Exhibit_, 2003, p. 6626.
  * [111] J. Kim and Y. Kim, “Moving ground target tracking in dense obstacle areas using UAVs,” _IFAC Proceedings Volumes_ , vol. 41, no. 2, pp. 8552–8557, 2008.
  * [112] I. Mohammed, I. B. Collings, and S. V. Hanly, “Line of sight probability prediction for UAV communication,” in _2021 IEEE International Conference on Communications Workshops (ICC Workshops)_. IEEE, 2021, pp. 1–6.
  * [113] J. Zhang, M. Lou, L. Xiang, and L. Hu, “Power cognition: Enabling intelligent energy harvesting and resource allocation for solar-powered UAVs,” _Future Generation Computer Systems_ , vol. 110, pp. 658–664, 2020.
  * [114] A. Kokhanovsky, “Optical properties of terrestrial clouds,” _Earth-Science Reviews_ , vol. 64, no. 3-4, pp. 189–241, 2004.
  * [115] A. Sayigh, _Solar energy engineering_. Elsevier, 2012.
  * [116] N. Baldock and M. Mokhtarzadeh-Dehghan, “A study of solar-powered, high-altitude unmanned aerial vehicles,” _Aircraft Engineering and Aerospace Technology_ , vol. 78, no. 3, pp. 187–193, 2006.
  * [117] W. Le, Z. Xue, J. Chen, and Z. Zhang, “Coverage path planning based on the optimization strategy of multiple solar powered unmanned aerial vehicles,” _Drones_ , vol. 6, no. 8, p. 203, 2022.
  * [118] X. Wei, P. Yao, and Z. Xie, “Comprehensive optimization of energy storage and standoff tracking for solar-powered UAV,” _IEEE Systems Journal_ , vol. 14, no. 4, pp. 5133–5143, 2020.
  * [119] A. T. Klesh and P. T. Kabamba, “Solar-powered aircraft: Energy-optimal path planning and perpetual endurance,” _Journal of guidance, control, and dynamics_ , vol. 32, no. 4, pp. 1320–1329, 2009.
  * [120] Y. Fu, H. Mei, K. Wang, and K. Yang, “Joint optimization of 3D trajectory and scheduling for solar-powered UAV systems,” _IEEE Transactions on Vehicular Technology_ , vol. 70, no. 4, pp. 3972–3977, 2021.
  * [121] J. A. Bondy, _Graph theory with applications_ , 1982.
  * [122] P. Hart, N. Nilsson, and B. Raphael, “A formal basis for the heuristic determination of minimum cost paths,” _IEEE Transactions on Systems Science and Cybernetics_ , vol. 4, no. 2, pp. 100–107, 1968. [Online]. Available: https://doi.org/10.1109/tssc.1968.300136
  * [123] S. M. LaValle, J. J. Kuffner, B. Donald _et al._ , “Rapidly-exploring random trees: Progress and prospects,” _Algorithmic and computational robotics: new directions_ , vol. 5, pp. 293–308, 2001.
  * [124] H. Huang and A. V. Savkin, “Path planning for a solar-powered UAV inspecting mountain sites for safety and rescue,” _Energies_ , vol. 14, no. 7, p. 1968, 2021.
  * [125] J. Wu, H. Wang, N. Li, P. Yao, Y. Huang, and H. Yang, “Path planning for solar-powered UAV in urban environment,” _Neurocomputing_ , vol. 275, pp. 2055–2065, 2018.
  * [126] L. Wirth, P. Oettershagen, J. Ambühl, and R. Siegwart, “Meteorological path planning using dynamic programming for a solar-powered UAV,” in _2015 IEEE aerospace conference_. IEEE, 2015, pp. 1–11.
  * [127] H. D. Tuan, A. A. Nasir, A. V. Savkin, H. V. Poor, and E. Dutkiewicz, “MPC-based UAV navigation for simultaneous solar-energy harvesting and two-way communications,” _IEEE Journal on Selected Areas in Communications_ , vol. 39, no. 11, pp. 3459–3474, 2021.
  * [128] J.-S. Lee and K.-H. Yu, “Optimal path planning of solar-powered UAV using gravitational potential energy,” _IEEE Transactions on Aerospace and Electronic Systems_ , vol. 53, no. 3, pp. 1442–1451, 2017.
  * [129] R. Dai, U. Lee, S. Hosseini, and M. Mesbahi, “Optimal path planning for solar-powered UAVs based on unit quaternions,” in _2012 IEEE 51st IEEE Conference on Decision and Control (CDC)_. IEEE, 2012, pp. 3104–3109.
  * [130] S. C. Spangelo and E. G. Gilbert, “Power optimization of solar-powered aircraft with specified closed ground tracks,” _Journal of Aircraft_ , vol. 50, no. 1, pp. 232–238, 2013.
  * [131] S. Hosseini, R. Dai, and M. Mesbahi, “Optimal path planning and power allocation for a long endurance solar-powered UAV,” in _2013 American Control Conference_. IEEE, 2013, pp. 2588–2593.
  * [132] Y. Lun, H. Wang, J. Wu, Y. Liu, and Y. Wang, “Target search in dynamic environments with multiple solar-powered UAVs,” _IEEE Transactions on Vehicular Technology_ , vol. 71, no. 9, pp. 9309–9321, 2022.
  * [133] S.-H. Kim, G. E. G. Padilla, K.-J. Kim, and K.-H. Yu, “Flight path planning for a solar powered UAV in wind fields using direct collocation,” _IEEE Transactions on Aerospace and Electronic Systems_ , vol. 56, no. 2, pp. 1094–1105, 2019.
  * [134] S. Aggarwal and N. Kumar, “Path planning techniques for unmanned aerial vehicles: A review, solutions, and challenges,” _Computer Communications_ , vol. 149, pp. 270–299, 2020.
  * [135] C. Di Franco and G. Buttazzo, “Energy-aware coverage path planning of UAVs,” in _2015 IEEE International Conference on Autonomous Robot Systems and Competitions_ , 2015, pp. 111–117.
  * [136] S. Vasisht and M. Mesbahi, “Trajectory design and coverage control for solar-powered UAVs,” in _AIAA guidance, navigation, and control conference_ , 2015, p. 1314.
  * [137] B. H. Wang, D. B. Wang, Z. A. Ali, B. Ting Ting, and H. Wang, “An overview of various kinds of wind effects on unmanned aerial vehicle,” _Measurement and Control_ , vol. 52, no. 7-8, pp. 731–739, 2019.
  * [138] Y. Zhi, Z. Fu, X. Sun, and J. Yu, “Security and privacy issues of UAV: a survey,” _Mobile Networks and Applications_ , vol. 25, pp. 95–101, 2020\.
  * [139] P. Blank, S. Kirrane, and S. Spiekermann, “Privacy-aware restricted areas for unmanned aerial systems,” _IEEE Security & Privacy_, vol. 16, no. 2, pp. 70–79, 2018.
  * [140] B. Ly and R. Ly, “Cybersecurity in unmanned aerial vehicles (UAVs),” _Journal of Cyber Security Technology_ , vol. 5, no. 2, pp. 120–137, 2021\.
  * [141] O. Westerlund and R. Asif, “Drone hacking with raspberry-pi 3 and wifi pineapple: Security and privacy threats for the internet-of-things,” in _2019 1st International Conference on Unmanned Vehicle Systems-Oman (UVS)_. IEEE, 2019, pp. 1–10.
  * [142] Y. Luo, Y. Yu, Z. Jin, Y. Li, Z. Ding, Y. Zhou, and Y. Liu, “Privacy-aware UAV flights through self-configuring motion planning,” in _2020 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2020, pp. 1169–1175.
  * [143] A. V. Savkin and C. Wang, “A simple biologically inspired algorithm for collision-free navigation of a unicycle-like robot in dynamic environments with moving obstacles,” _Robotica_ , vol. 31, no. 6, pp. 993–1001, 2013\.
  * [144] S. A. Gunawan, G. N. Pratama, A. I. Cahyadi, B. Winduratna, Y. C. Yuwono, and O. Wahyunggoro, “Smoothed a-star algorithm for nonholonomic mobile robot path planning,” in _2019 International Conference on Information and Communications Technology (ICOIACT)_. IEEE, 2019, pp. 654–658.
  * [145] C. Sun, Y.-C. Liu, R. Dai, and D. Grymin, “Two approaches for path planning of unmanned aerial vehicles with avoidance zones,” _Journal of Guidance, Control, and Dynamics_ , vol. 40, no. 8, pp. 2076–2083, 2017.
  * [146] B. Fan, Y. Li, R. Zhang, and Q. Fu, “Review on the technological development and application of UAV systems,” _Chinese Journal of Electronics_ , vol. 29, no. 2, pp. 199–207, 2020.
  * [147] D. C. Tsouros, S. Bibi, and P. G. Sarigiannidis, “A review on UAV-based applications for precision agriculture,” _Information_ , vol. 10, no. 11, p. 349, 2019.
  * [148] J. Ye, C. Zhang, H. Lei, G. Pan, and Z. Ding, “Secure UAV-to-UAV systems with spatially random UAVs,” _IEEE Wireless Communications Letters_ , vol. 8, no. 2, pp. 564–567, 2018.
  * [149] A. V. Savkin and H. Huang, “Asymptotically optimal path planning for ground surveillance by a team of UAVs,” _IEEE Systems Journal_ , vol. 16, no. 2, pp. 3446–3449, 2022.
  * [150] D. Dissanayaka, T. R. Wanasinghe, O. D. Silva, A. Jayasiri, and G. K. I. Mann, “Review of navigation methods for UAV-based parcel delivery,” _IEEE Transactions on Automation Science and Engineering_ , pp. 1–15, 2022.
  * [151] L. Gupta, R. Jain, and G. Vaszkun, “Survey of important issues in UAV communication networks,” _IEEE Communications Surveys & Tutorials_, vol. 18, no. 2, pp. 1123–1152, 2016.
  * [152] F. Nex and F. Remondino, “UAV for 3D mapping applications: a review,” _Applied geomatics_ , vol. 6, pp. 1–15, 2014.
  * [153] T. Elmokadem and A. V. Savkin, “Towards fully autonomous UAVs: A survey,” _Sensors_ , vol. 21, no. 18, 2021. [Online]. Available: https://www.mdpi.com/1424-8220/21/18/6223
  * [154] M. Arjomandi, S. Agostino, M. Mammone, M. Nelson, and T. Zhou, “Classification of unmanned aerial vehicles,” _Report for Mechanical Engineering class, University of Adelaide, Adelaide, Australia_ , pp. 1–48, 2006.
  * [155] M. B. Hayat, D. Ali, K. C. Monyake, L. Alagha, and N. Ahmed, “Solar energy—A look into power generation, challenges, and a solar-powered future,” _International Journal of Energy Research_ , vol. 43, no. 3, pp. 1049–1067, 2019.
  * [156] S. Morton, R. D’Sa, and N. Papanikolopoulos, “Solar powered UAV: Design and experiments,” in _2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , 2015, pp. 2460–2466.
  * [157] S. Shao, W. Shi, Y. Zhao, and Y. Du, “A new method of solving UAV trajectory planning under obstacles and multi-constraint,” _IEEE Access_ , vol. 9, pp. 161 161–161 180, 2021.
  * [158] I. R. Manchester and A. V. Savkin, “Circular-navigation-guidance law for precision missile/target engagements,” _Journal of guidance, control, and dynamics_ , vol. 29, no. 2, pp. 314–320, 2006.
  * [159] A. V. Savkin, S. C. Verma, and S. Anstee, “Optimal navigation of an unmanned surface vehicle and an autonomous underwater vehicle collaborating for reliable acoustic communication with collision avoidance,” _Drones_ , vol. 6, no. 01, p. 27, 2022.
  * [160] Y. Wu, “Coordinated path planning for an unmanned aerial-aquatic vehicle (UAAV) and an autonomous underwater vehicle (AUV) in an underwater target strike mission,” _Ocean Engineering_ , vol. 182, pp. 162–173, 2019.
  * [161] Y. Luo, A. Bai, and H. Zhang, “Distributed formation control of UAVs for circumnavigating a moving target in three-dimensional space,” _Guidance, Navigation and Control_ , vol. 1, no. 03, p. 2150014, 2021.
  * [162] S. A. Gunawan, G. N. P. Pratama, A. I. Cahyadi, B. Winduratna, Y. C. H. Yuwono, and O. Wahyunggoro, “Smoothed A-star algorithm for nonholonomic mobile robot path planning,” in _2019 International Conference on Information and Communications Technology (ICOIACT)_ , 2019, pp. 654–658.
  * [163] K. L. B. Cook, “The silent force multiplier: The history and role of UAVs in warfare,” in _2007 IEEE Aerospace Conference_ , 2007, pp. 1–7.
  * [164] L. Deng, Z. Mao, X. Li, Z. Hu, F. Duan, and Y. Yan, “UAV-based multispectral remote sensing for precision agriculture: A comparison between different cameras,” _ISPRS Journal of Photogrammetry and Remote Sensing_ , vol. 146, pp. 124–136, 2018.
  * [165] P. Ramon-Soria, M. Perez-Jimenez, B. Arrue, and A. Ollero, “Planning system for integrated autonomous infrastructure inspection using UAVs,” in _2019 International Conference on Unmanned Aircraft Systems (ICUAS)_. IEEE, 2019, pp. 313–320.
  * [166] A. Saboor, S. Coene, E. Vinogradov, E. Tanghe, W. Joseph, and S. Pollin, “Elevating the future of mobility: UAV-enabled intelligent transportation systems. arxiv 2021,” _arXiv preprint arXiv:2110.09934_.
  * [167] J. Li, R. Kacimi, T. Liu, X. Ma, and R. Dhaou, “Non-terrestrial networks-enabled internet of things: UAV-centric architectures, applications, and open issues,” _Drones_ , vol. 6, no. 4, 2022.
  * [168] X. Hu, B. Pang, F. Dai, and K. H. Low, “Risk assessment model for UAV cost-effective path planning in urban environments,” _IEEE Access_ , vol. 8, pp. 150 162–150 173, 2020.
  * [169] D. Shim, H. Chung, H. J. Kim, and S. Sastry, “Autonomous exploration in unknown urban environments for unmanned aerial vehicles,” in _AIAA Guidance, Navigation, and Control Conference and Exhibit_ , 2005, p. 6478.
  * [170] S. C. Verma, S. Li, and A. V. Savkin, “A Hybrid Global/Reactive Algorithm for Collision-Free UAV Navigation in 3D Environments with Steady and Moving Obstacles,” _Drones_ , vol. 7, no. 11, p. 675, 2023.
  * [171] T. Elmokadem and A. V. Savkin, “A hybrid approach for autonomous collision-free UAV navigation in 3D partially unknown dynamic environments,” _Drones_ , vol. 5, no. 3, p. 57, 2021.
  * [172] T. Elmokadem, “A hybrid strategy for safe navigation of mobile robots in partially-known dynamic environments,” in _2018 37th Chinese Control Conference (CCC)_ , 2018, pp. 4785–4791.
  * [173] P. Gia Luan and N. T. Thinh, “Real-time hybrid navigation system-based path planning and obstacle avoidance for mobile robots,” _Applied Sciences_ , vol. 10, no. 10, p. 3355, 2020.
  * [174] X. Zhong, J. Tian, H. Hu, and X. Peng, “Hybrid path planning based on safe A* algorithm and adaptive window approach for mobile robot in large-scale dynamic environment,” _Journal of Intelligent & Robotic Systems_, vol. 99, pp. 65–77, 2020.
  * [175] J. Li, G. Deng, C. Luo, Q. Lin, Q. Yan, and Z. Ming, “A hybrid path planning method in unmanned air/ground vehicle (UAV/UGV) cooperative systems,” _IEEE Transactions on Vehicular Technology_ , vol. 65, no. 12, pp. 9585–9596, 2016.
  * [176] R. C. Coulter _et al._ , _Implementation of the pure pursuit path tracking algorithm_. Carnegie Mellon University, The Robotics Institute, 1992.
  * [177] I. R. Petersen, V. A. Ugrinovskii, and A. V. Savkin, _Robust control design using H-infinity methods_. Springer Science & Business Media, 2000.
  * [178] A. Bansal and V. Sharma, “Design and analysis of robust h-infinity controller,” _Control theory and informatics_ , vol. 3, no. 2, pp. 7–14, 2013.
  * [179] A. T. Azar, F. E. Serrano, A. Koubaa, and N. A. Kamal, “Backstepping h-infinity control of unmanned aerial vehicles with time varying disturbances,” in _2020 First International Conference of Smart Systems and Emerging Technologies (SMARTTECH)_. IEEE, 2020, pp. 243–248.
  * [180] I. R. Petersen and A. V. Savkin, _Robust Kalman filtering for signals and systems with large uncertainties_. Springer Science & Business Media, 1999.
  * [181] P. N. Pathirana, N. Bulusu, A. V. Savkin, and S. Jha, “Node localization using mobile robots in delay-tolerant sensor networks,” _IEEE transactions on Mobile Computing_ , vol. 4, no. 3, pp. 285–296, 2005.
  * [182] A. V. Savkin and I. R. Petersen, “Robust state estimation and model validation for discrete-time uncertain systems with a deterministic description of noise and uncertainty,” _Automatica_ , vol. 34, no. 2, pp. 271–274, 1998.
  * [183] P. Marantos, Y. Koveos, and K. J. Kyriakopoulos, “UAV state estimation using adaptive complementary filters,” _IEEE Transactions on Control Systems Technology_ , vol. 24, no. 4, pp. 1214–1226, 2015.
  * [184] C. Hajiyev and H. E. Soken, “Robust estimation of UAV dynamics in the presence of measurement faults,” _Journal of Aerospace Engineering_ , vol. 25, no. 1, pp. 80–89, 2012.
  * [185] “ASHRAE, handbook of fundamentals 1979,” _American Society of Heating, Refrigeration, and Air-Conditioning Engineers_ , 1979.
  * [186] V. Utkin, J. Guldner, and J. Shi, _Sliding mode control in electro-mechanical systems_. CRC press, 2017.
  * [187] V. I. Utkin, “Sliding modes and their applications in variable structure systems,” _Mir, Moscow_ , 1978.
  * [188] A. V. Savkin and R. J. Evans, _Hybrid dynamical systems: controller and sensor switching problems_. Springer Science & Business Media, 2002.
  * [189] E. Skafidas, R. J. Evans, A. V. Savkin, and I. R. Petersen, “Stability results for switched controller systems,” _Automatica_ , vol. 35, no. 4, pp. 553–564, 1999.
  * [190] H. Huang and A. V. Savkin, “Deployment of heterogeneous UAV base stations for optimal quality of coverage,” _IEEE Internet of Things Journal_ , vol. 9, no. 17, pp. 16 429–16 437, 2022.
  * [191] A. V. Savkin, C. Huang, and W. Ni, “On-demand deployment of aerial base stations for coverage enhancement in reconfigurable intelligent surface-assisted cellular networks on uneven terrains,” _IEEE Communications Letters_ , vol. 27, no. 2, pp. 666–670, 2022.
  * [192] H. Huang and A. V. Savkin, “A method for optimized deployment of unmanned aerial vehicles for maximum coverage and minimum interference in cellular networks,” _IEEE Transactions on Industrial Informatics_ , vol. 15, no. 5, pp. 2638–2647, 2018.
  * [193] Y. Liu, X. Liu, X. Mu, T. Hou, J. Xu, M. Di Renzo, and N. Al-Dhahir, “Reconfigurable intelligent surfaces: Principles and opportunities,” _IEEE communications surveys & tutorials_, vol. 23, no. 3, pp. 1546–1577, 2021.
  * [194] A. V. Savkin, C. Huang, and W. Ni, “Collision-Free 3-D Navigation of a UAV Team for Optimal Data Collection in Internet-of-Things Networks With Reconfigurable Intelligent Surfaces,” _IEEE Systems Journal_ , vol. 17, no. 3, pp. 4070–4077, 2023.
  * [195] M. Eskandari and A. Savkin, “Trajectory Planning for UAVs Equipped With RISs to Provide Aerial LoS Service for Mobile Nodes in 5G/Optical Wireless Communication Networks,” _IEEE Transactions on Vehicular Technology_ , vol. 72, no. 6, pp. 8216–8221, 2023.
  * [196] M. Eskandari, H. Huang, A. V. Savkin, and W. Ni, “Model Predictive Control-Based 3D Navigation of a RIS-Equipped UAV for LoS Wireless Communication With a Ground Intelligent Vehicle,” _IEEE Transactions on Intelligent Vehicles_ , vol. 8, no. 3, pp. 2371–2384, 2023.
  * [197] P. Stodola, J. Drozd, J. Mazal, J. Hodickỳ, and D. Procházka, “Cooperative unmanned aerial system reconnaissance in a complex urban environment and uneven terrain,” _Sensors_ , vol. 19, no. 17, p. 3754, 2019\.
  * [198] A. V. Savkin, S. C. Verma, and W. Ni, “Autonomous uav 3d trajectory optimization and transmission scheduling for sensor data collection on uneven terrains,” _Defence Technology_ , vol. 30, pp. 154–160, 2023. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S2214914723000843
  * [199] J. Butzke, K. Gochev, B. Holden, E.-J. Jung, and M. Likhachev, “Planning for a ground-air robotic system with collaborative localization,” in _2016 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2016, pp. 284–291.
  * [200] J.-K. Lee, H. Jung, H. Hu, and D. H. Kim, “Collaborative control of UAV/UGV,” in _2014 11th International Conference on Ubiquitous Robots and Ambient Intelligence (URAI)_. IEEE, 2014, pp. 641–645.
  * [201] A. V. Savkin and H. Huang, “Navigation of a network of aerial drones for monitoring a frontier of a moving environmental disaster area,” _IEEE Systems Journal_ , vol. 14, no. 4, pp. 4746–4749, 2020.
  * [202] A. Nedjati, B. Vizvari, and G. Izbirak, “Post-earthquake response by small UAV helicopters,” _Natural Hazards_ , vol. 80, pp. 1669–1688, 2016.
  * [203] S. Amici, M. Turci, F. Giulietti, S. Giammanco, M. Buongiorno, A. La Spina, and L. Spampinato, “Volcanic environments monitoring by drones mud volcano case study,” _The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences_ , vol. 40, pp. 5–10, 2013.
  * [204] E. D. Beni, M. Cantarero, and A. Messina, “UAVs for volcano monitoring: A new approach applied on an active lava flow on Mt. Etna (Italy), during the 27 February–02 March 2017 eruption,” _Journal of Volcanology and Geothermal Research_ , vol. 369, pp. 250–262, 2019. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0377027318301756
  * [205] F. L. Bonali, A. Tibaldi, F. Marchese, L. Fallati, E. Russo, C. Corselli, and A. Savini, “UAV-based surveying in volcano-tectonics: An example from the Iceland rift,” _Journal of Structural Geology_ , vol. 121, pp. 46–64, 2019. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0191814118301998
  * [206] B. R. Jordan, “Collecting field data in volcanic landscapes using small UAS (sUAS)/drones,” _Journal of Volcanology and Geothermal Research_ , vol. 385, pp. 231–241, 2019. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0377027317302779
  * [207] J. C. Hodgson, S. M. Baylis, R. Mott, A. Herrod, and R. H. Clarke, “Precision wildlife monitoring using unmanned aerial vehicles,” _Scientific reports_ , vol. 6, no. 1, p. 22574, 2016.
  * [208] J. Linchant, J. Lisein, J. Semeki, P. Lejeune, and C. Vermeulen, “Are unmanned aircraft systems (UASs) the future of wildlife monitoring? A review of accomplishments and challenges,” _Mammal Review_ , vol. 45, no. 4, pp. 239–252, 2015.
  * [209] L. F. Gonzalez, G. A. Montes, E. Puig, S. Johnson, K. Mengersen, and K. J. Gaston, “Unmanned aerial vehicles (UAVs) and artificial intelligence revolutionizing wildlife monitoring and conservation,” _Sensors_ , vol. 16, no. 1, p. 97, 2016.
  * [210] A. V. Savkin and H. Huang, “Bioinspired bearing only motion camouflage UAV guidance for covert video surveillance of a moving target,” _IEEE Systems Journal_ , vol. 15, no. 4, pp. 5379–5382, 2020.
  * [211] Z. Yang, X. Yu, S. Dedman, M. Rosso, J. Zhu, J. Yang, Y. Xia, Y. Tian, G. Zhang, and J. Wang, “UAV remote sensing applications in marine monitoring: Knowledge visualization and review,” _Science of The Total Environment_ , vol. 838, p. 155939, 2022. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0048969722030364
  * [212] C. Wang, Z. Li, T. Wang, X. Xu, X. Zhang, and D. Li, “Intelligent fish farm—the future of aquaculture,” _Aquaculture International_ , pp. 1–31, 2021.
  * [213] D. Anggraeni, A. Udhiarto, A. Wirawan, A. Ramadian, and N. Poespawati, “Development and analysis characteristic solar powered system on LAPAN surveillance UAV 03 (LSU 03),” in _AIP Conference Proceedings_ , vol. 2088, no. 1. AIP Publishing, 2019.
  * [214] G. Romeo, E. Cestino, F. Borello, M. Pacino _et al._ , “Very-Long Endurance Solar Powered Autonomous UAVs: Role and Constraints for GMEs Applications,” in _28th International Congress of the Aeronautical Sciences–ICAS_ , 2012.
  * [215] X. Yang and X. Pei, “Hybrid system for powering unmanned aerial vehicles: Demonstration and study cases,” in _Hybrid Technologies for Power Generation_. Elsevier, 2022, pp. 439–473.
  * [216] A. P. Colefax, “Developing the use of drones for non-destructive shark management and beach safety,” Ph.D. dissertation, Southern Cross University, 2020.
  * [217] P. K. R. Maddikunta, S. Hakak, M. Alazab, S. Bhattacharya, T. R. Gadekallu, W. Z. Khan, and Q.-V. Pham, “Unmanned aerial vehicles in smart agriculture: Applications, requirements, and challenges,” _IEEE Sensors Journal_ , vol. 21, no. 16, pp. 17 608–17 619, 2021.
  * [218] K. Li, W. Ni, and F. Dressler, “Continuous maneuver control and data capture scheduling of autonomous drone in wireless sensor networks,” _IEEE Transactions on Mobile Computing_ , vol. 21, no. 8, pp. 2732–2744, 2021.
  * [219] P. S. Bithas, E. T. Michailidis, N. Nomikos, D. Vouyioukas, and A. G. Kanatas, “A survey on machine-learning techniques for UAV-based communications,” _Sensors_ , vol. 19, no. 23, p. 5170, 2019.
  * [220] X. Liu, Y. Liu, Y. Chen, and L. Hanzo, “Trajectory design and power control for multi-UAV assisted wireless networks: A machine learning approach,” _IEEE Transactions on Vehicular Technology_ , vol. 68, no. 8, pp. 7957–7969, 2019.
  * [221] H. B. Makineci, H. Karabörk, and A. Durdu, “ANN estimation model for photogrammetry-based UAV flight planning optimisation,” _International Journal of Remote Sensing_ , vol. 43, no. 15-16, pp. 5686–5708, 2022.
  * [222] M. E. Salgado, G. C. Goodwin, and S. F. Graebe, “Control System Design,” 2001\.
  * [223] L. Tao, Y. Zhou, L. Zicun, and X. Zhang, “State of art on energy management strategy for hybrid-powered unmanned aerial vehicle,” _Chinese Journal of Aeronautics_ , vol. 32, no. 6, pp. 1488–1503, 2019.
  * [224] P. Panagiotou, I. Tsavlidis, and K. Yakinthos, “Conceptual design of a hybrid solar MALE UAV,” _Aerospace Science and Technology_ , vol. 53, pp. 207–219, 2016.
  * [225] S. Sekander, H. Tabassum, and E. Hossain, “Statistical performance modeling of solar and wind-powered UAV communications,” _IEEE Transactions on Mobile Computing_ , vol. 20, no. 8, pp. 2686–2700, 2020.
  * [226] C. Van Nguyen, T. Van Quyen, A. M. Le, L. H. Truong, and M. T. Nguyen, “Advanced hybrid energy harvesting systems for unmanned aerial vehicles (UAVs),” _Advances in Science, Technology and Engineering Systems Journal_ , vol. 5, no. 1, pp. 34–39, 2020.
  * [227] X. Xu, Y. Zhao, L. Tao, and Z. Xu, “Resource allocation strategy for dual UAVs-assisted MEC system with hybrid solar and RF energy harvesting,” in _2021 3rd International Conference on Computer Communication and the Internet (ICCCI)_. IEEE, 2021, pp. 52–57.


∎

11institutetext: Bernhard Haslhofer 22institutetext: Cornell University,
Department of Information Science
301 College Avenue, Ithaca, NY 14850, USA
22email<EMAIL_ADDRESS>33institutetext: Robert Sanderson and
Herbert van de Sompel 44institutetext: Los Alamos National Laboratory
Los Alamos, NM 87544, USA
44email<EMAIL_ADDRESS>55institutetext: Rainer Simon
66institutetext: Austrian Institute of Technology
Donau-City-Str. 1, A-1220 Vienna, Austria
66email<EMAIL_ADDRESS>
# Open Annotations on Multimedia Web Resources

Bernhard Haslhofer  Robert Sanderson  Rainer Simon  Herbert van de Sompel

(Received: / Accepted: date)

###### Abstract

Many Web portals allow users to associate additional information with existing
multimedia resources such as images, audio, and video. However, these portals
are usually closed systems and user-generated annotations are almost always
kept locked up and remain inaccessible to the Web of Data. We believe that an
important step to take is the integration of multimedia annotations and the
Linked Data principles. We present the current state of the Open Annotation
Model, explain our design rationale, and describe how the model can represent
user annotations on multimedia Web resources. Applying this model in Web
portals and devices, which support user annotations, should allow clients to
easily publish and consume, thus exchange annotations on multimedia Web
resources via common Web standards.

###### Keywords:

Annotations Web Linked Data

††journal: Multimedia Tools and Applications

##  1 Introduction

Youtube and Flickr are examples of large-scale Web portals that allow users to
annotate multimedia resources by adding textual notes and comments to images
or videos. Figure 1 shows an
image111http://www.flickr.com/photos/library_of_congress/3175009412/ from the
Flickr Commons collection contributed by the Library of Congress. Flickr users
added several annotations to specific image segments, one of them telling us
that this picture shows a cathedral in Bergen.

Figure 1: An annotation example on Flickr.

Annotations describe resources with additional information, which is valuable
to other users, who are searching and browsing resource collections. They are
also important for underlying information systems, which can exploit the high-
level descriptive semantics of annotations in combination with automatically
extracted low-level features, such as image size and color, to implement
search and retrieval over multimedia resources. Taking the previous example,
users who are searching for “Bergen” or even “Bergen Cathedral” will now find
this particular image in Flickr, because some user provided this descriptive
information in textual form.

Annotations are also becoming an increasingly important component in scholarly
cyber-infrastructures (cf. Bradley:2008kx ), which are often realized as Web
systems. Therefore, a Web-based annotation model should fulfill several
requirements. In the age of video blogging and real-time sharing of geo-
located images, the notion of solely textual annotations has become obsolete.
Instead, _multimedia_ Web resources should be annotatable and also be able to
be annotated onto other resources. Users often discuss multiple segments of a
resource, or multiple resources, in a single annotation and thus the model
should support multiple targets. An annotation framework should also follow
the Linked Open Data guidelines Heath:2011uq  to promote annotation sharing
between systems. In order to avoid inaccurate or incorrect annotations, it
must take the ephemeral nature of Web resources into account.

Annotations on the Web have many facets: a simple example could be a textual
note or a tag (cf., Hunter2009 ) annotating an image or video. Things become
more complex when a particular paragraph in an HTML document annotates a
segment (cf., Hausenblas:LDOW09 ) in an online video or when someone draws
polygon shapes on tiled high-resolution image sets. If we further extend the
annotation concept, we could easily regard a large portion of Twitter tweets
as annotations on Web resources. Therefore, in a generic and Web-centric
conception, we regard an annotation as an association created between one
_body_ resource and other _target_ resources, where the body must be somehow
_about_ the target.

Annotea Kahan:2001vn  already defines a specification for publishing
annotations on the Web but has several shortcomings: (i) it was designed for
the annotation of Web pages and provides only limited means to address
segments in multimedia objects, (ii) if clients want to access annotations
they need to be aware of the Annotea-specific protocol, and (iii) Annotea
annotations do not take into account that Web resources are very likely to
have different states over time.

Throughout the years several Annotea extensions have been developed to deal
with these and other shortcomings: Koivunnen Koivunen:2006s  introduced
additional types of annotations, such as _bookmark_ and _topic_. Schroeter and
Hunter Schroeter:uq  proposed to express segments in media-objects by using
_context_ resources in combination with formalized or standardized
descriptions to represent the context, such as SVG or complex datatypes taken
from the MPEG-7 standard. Based on that work, Haslhofer et al.
Haslhofer:2009ve  introduce the notion of _annotation profiles_ as containers
for content- and annotation-type specific Annotea extensions and suggested
that annotations should be dereferencable resources on the Web, which follow
the Linked Data guidelines. However, these extensions were developed
separately from each other and inherit some of the above-mentioned Annotea
shortcomings.

In this article we describe the _Open Annotation Model_ 222Open Annotation
Model: Beta Data Model Guide http://www.openannotation.org/spec/beta/, which
is currently being developed in an international collaboration. It applies a
Web- and resource-centric view on annotations and defines a modular
architecture, which has a simple base line model in its core. The model also
provides means to address segments in multimedia resources either by encoding
segment information using the Media Fragment URI specification or by
introducing custom segment constraints for more complex annotation use cases.
By allowing fixity and timestamp information on the resources involved in an
annotation, it also takes into account the ephemeral nature of Web resources.
Pulling together the functionalities provided by various, partly independent
Annotea extensions is a major goal of this effort.

This article extends our work previously published in Haslhofer:2011fk  by the
following: it explains the design rationale that lead to the specification of
the current annotation model and also the technical aspects of the model in
more detail. It also contains an updated and extended related work section.

##  2 Design Rationale

In this section we outline the design rationale that drives the specification
of the Open Annotation Model. We give examples illustrating common
requirements we found in several real-world annotation use cases (e.g.,
Sanderson:2011a ; Verspoor:2005kx ; Simon:2011vn ) and describe the reasons
behind our design decisions. From each design decision we derived a set of
guiding design principles, which are reflected in the current Open Annotation
Model design.

###  2.1 Annotations are qualified associations between resources

Textual notes or tags on images, as in the Flickr example in Figure 1, occur
frequently on the Web and are the simplest examples for Web annotations. In
Open Annotation Model terms, these are _annotations_ that have a textual
_body_ and an image resource as _target_. However, in many scenarios the
prevailing view that annotation bodies are textual is insufficient. Figure 2
shows an annotation in which the body is not a textual note, but a video,
which itself is an addressable Web resource identified by a URI. In order to
cover such cases, we must abstract from purely textual annotation bodies and
model them as resources that can be of any media type.

From a conceptual point of view, an annotation is an association between two
resources, the _body_ and the _target_. However, in most cases this
association needs to be qualified and addressable in some way. Indicating the
creator and creation date of a resource, or replying to existing annotations,
are frequently occurring scenarios that require an annotation to be expressed
as a first-class entity. Web annotations are then instances of an annotation
and relate together the body and target resources that are involved in the
annotation association. With this approach we follow a common design pattern,
which is also known as Qualified Relation in Linked Data Dodds:uq  or
Association Class in UML.

Figure 2: A Youtube video annotating an image on the Web.

In contrast to existing tagging models (see KimEtAl2008 ) the annotation
_creator_ is not a mandatory core model entity. We believe that adopters of
the Open Annotation Model already have user models in place or rely on open
Web identities (e.g., a Google or Facebook account) and are most likely not
willing to map their user models against another user model, which is
specified as part of an annotation model. However, we encourage adopters to
relate annotations with existing Web resources, which identify users.

From these considerations we derive the following guiding principles:

  * •

All core entities (annotation, body, target) must be resources.

  * •

Annotations must allow for both body and target of any media type.

  * •

Annotations, bodies, and targets can have different authorship.

###  2.2 Annotations involve parts of multimedia resources

Our introductory Flickr example illustrates how annotations can target
specific segments in Web resources. In Figure 2 we gave an example that
involves multimedia resources: a video as body, and an image as target. There
is strong dependency between these requirements because a resource’s media
type affects the way segments need to be addressed. Annotating an area in an
image requires a different segment representation than those that target
segments in the spatial and temporal dimensions of a video. Similarly,
addressing text segments in a PDF document differs from addressing text in
plain text or HTML documents. Since we abstract from purely textual annotation
bodies, we must take into account that both the body and the target of an
annotation can be resource segments. We could, for instance, refine our
previous example by saying that some sequence within the video annotates a
certain image segment.

The problem of addressing media segments, also known as fragment
identification, is well known and will be explained in more detail in Section
6.3. Since the Open Annotation Model will be implemented in Web environments
and all resources involved in an annotation are Web resources identified by
URIs, it should reuse the fragment identification mechanisms that are already
defined as part of the Architecture of the World Wide Web Jacobs2004  and
extensions thereof, such as fragment construction rules for specific media
types. However, many annotation use cases require more complex segment
representations such as polygon regions in images, which cannot be expressed
with available standards. Therefore, our guiding principles with respect to
addressing segments in multimedia resources are:

  * •

Annotations must support resource segment addressing both on body and target
resources.

  * •

Preferably this should be done with (media) fragment URIs, but extensibility
must be provided for cases in which the use of URIs for segment addressing is
not possible.

###  2.3 Annotation resources are ephemeral Web resources

Previously we argued that all core entities of the Open Annotation Model must
be first class Web resources, which are identified by URIs, if possible HTTP
URIs. The great benefit of this approach is that existing technologies and
solutions that can be applied for Web resources (e.g., mime-types, fragments,
access control, etc.) also work for resources involved in a Web annotation
without the necessity to include these aspects in an annotation specification.
However, there is one big problem we inherit from the Web architecture and
which is severe in the context of annotations: URI-addressable Web resources
are ephemeral, which means that the representations obtained by dereferencing
their URIs may change over time.

The annotation example in Figure 3 shows a Twitter tweet annotating the CNN
web site and illustrates the ephemerality problem: the tweet refers to the CNN
page at a certain point in time and might be misinterpreted when the CNN main
web site changes. Measures should be provided that can help in avoiding
misinterpretations of annotations, including the expression of timestamps and
fixity information for body and target resources.

Figure 3: A Twitter tweet annotating the CNN website.

###  2.4 Annotations should be interoperable

The focus of the Open Annotation Model is on sharing annotations on scholarly
resources and therefore the model should have sufficient richness of
expression to satisfy scholars’ needs. However, in order to maximize the
likelihood of adoption, the model should also be an interoperability framework
readily applicable in other domains. One possibility to achieve this is to
follow a modular and extensible modeling approach, with a generally applicable
baseline model and domain- or media-type specific extensions. An annotation
client should implement at least the baseline model and, if possible, provide
fallback behavior for annotations that contain domain-specific extensions the
client might not be aware of.

In order to increase the likelihood of adoption, and in alignment with the
goal of sharing annotations, no client-server protocol for publishing,
updating, or deleting annotations will be specified. Rather, the
specifications will take a perspective whereby clients publish annotations to
the Web and make them discoverable using common Web approaches. Such an
approach does not require a preferred annotation server for a client, yet it
does not preclude one either.

Our guiding principles to achieve annotation interoperability and widespread
adoption are:

  * •

The Open Annotation Model should have a simple but expressive baseline model
defining top-level classes/entities and properties/relationships.

  * •

The baseline model should be extensible.

  * •

Annotation protocols are out of scope.

##  3 The Baseline Model

The Open Annotation data model draws from various extensions of Annotea to
form a cohesive whole. The Web architecture and Linked Data guidelines are
foundational principles, resulting in a specification that can be applied to
annotate any set of Web resources. At the time of this writing, the
specification, which is available at http://www.openannotation.org/spec/beta/,
is still under development. In the following, we describe the major technical
building blocks of the Open Annotation data model and use some simple examples
to illustrate how to apply the model in practice. For further examples, which
also cover different use cases, such as the annotation of medieval
manuscripts, we refer to the online documentation.

Following its predecessors, the Open Annotation Model, shown in Figure 4, has
three primary classes of resources. In all cases below, the oac namespace
prefix expands to http://www.openannotation.org/ns/.

Figure 4: Open Annotation baseline data model.

  * •

The oac:Body of the annotation (node B-1): This resource is the comment,
metadata or other information that is created about another resource. The body
can be any Web resource, of any media format, available at any URI. The model
allows for either one body per annotation, or an annotation without any body,
but not annotations with multiple bodies.

  * •

The oac:Target of the annotation (node T-1): This is the resource that the
body is about. Like the body, it can be any URI identified resource. The model
allows for one or more targets per annotation.

  * •

The oac:Annotation (node A-1): This resource is an RDF document, identified by
an HTTP URI, that describes at least the body and target resources involved in
the annotation as well as any additional properties and relationships (e.g.,
dcterms:creator). Dereferencing an annotation’s HTTP URI returns a
serialization in a permissible RDF format.

Figure 5: Additional properties and relationships in the Open Annotation
baseline model.

An annotation is a Web resource, which is identified by an HTTP URI and
returns an RDF document when being dereferenced. As with any RDF data,
additional properties and relationships can be associated with any of the
resources. It is recommended that an annotation has a timestamp of when the
annotation relationship was created (dcterms:created) and a reference to the
agent that created it (dcterms:creator). Resources referenced by additional
relationships may themselves have additional properties and relationships.
Figure 5 gives an example of recommended and other possible (e.g., dc:title)
properties and relationships that can be added to the Open Annotation baseline
model. The set of properties and relationship in this example is by no means
exhaustive. Properties and relationships from other vocabularies may also be
used. It is also important to note that the creator and created timestamp of
each of the three resource types above may be different. An annotation might
refer to an annotation body created by a third party, perhaps from before the
Open Annotation specification was published, and a target created by yet
another party.

Similarly, there may be additional subclasses of oac:Annotation that further
specify restrictions on the meaning of the annotation, such as a _reply_ ; an
annotation where the single target is itself an annotation. This allows
chaining of annotations into a threaded discussion model.

If the body of an annotation is identified by a dereferencable HTTP URI, as it
is the case in Twitter, various blogging platforms, or Google Docs, it can
easily be referenced from an annotation. If a client cannot create URIs for an
annotation body, for instance because it is an offline client, it can assign a
unique non-resolvable URI (called a URN) as the identifier for the body node.
This approach can still be reconciled with the Linked Data principles as
servers that publish such annotations can assign HTTP URIs they control to the
bodies, and express equivalence between the HTTP URI and the URN.

Figure 6: An example annotation with inline body.

The Open Annotation Model allows the inclusion of information directly in the
annotation document by adding the representation of a resource inline within
the RDF document using the _Content in RDF_ specification Koch:2009uq  from
the W3C. The example annotation in Figure 6 shows how to express this: the
representation is the object of the cnt:chars predicate, and its character
encoding the object of the cnt:characterEncoding predicate. Further classes
from this specification include Base64 encoded resources and XML encoded
resources.

##  4 Annotating Media Segments

Most of the use cases, which have been explored before specifying the model,
involved comments that were about a segment of a resource, rather than the
entire resource identified by a URI. The data model allows two different
methods of identifying and describing the region of interest of a resource;
either using a _fragment URI_ , or a more expressive _constraint_ resource. It
is clearly recommended to use fragment URIs whenever possible, because this
method relies on normative specifications, which brings interoperability with
other applications. Only if there are no appropriate URI fragment
specifications available, the creators should define their own constraints.

###  4.1 Describing segments with fragment URIs

A fragment URI normally identifies a part of a resource, and the method for
constructing and interpreting these URIs is dependent on the media type of the
resource. In general, fragment URIs are created by appending a fragment that
describes the section of interest to the URI of the full resource, separated
by a ’#’ character (see Berners-Lee:2005uq ).

There are two main sources for existing fragment URI specifications, which can
both identify and describe how to discover a segment of interest within a
resource. The first is the set of Mime Type specification RFCs from the IETF.
This includes X/HTML (RFC 2854/3236), XML (RFC 3023), PDF (RFC 3778) and Plain
Text (RFC 5147). The second is W3C Media URIs specification
fragmentsURI:2011ab , which is defined at a broader level to cover images,
video and audio resources, regardless of the exact format. The following
examples show how to apply these specifications to define media-type specific
URIs that describe a certain resource segment:

  * •

http://www.example.net/foo.html#namedSection identifies the section named as
“namedSection” in an HTML document.

  * •

http://www.example.net/foo.pdf#page=10&viewrect=20,100,50,60 identifies a
rectangle starting at 20 pixels in from the left, and 100 down from the top,
with a width of 50 and a height of 60 in a PDF document.

  * •

http://www.example.net/foo.png#xywh=160,120,320,240 identifies a 320 by 240
box, starting at x=160 and y=120 in an image.

  * •

http://www.example.net/foo.mpg#t=npt:10,20 identifies a sequence starting just
before the 10th second, and ending just before the 20th in a video.

We recommend that when a definition exists for how to construct a fragment URI
for a particular document format, and such a fragment would accurately
describe the section of interest for an annotation, then this technique should
be used. It is recommended to also use dcterms:isPartOf with the full resource
as the object, in order to make the annotation more easily discoverable.
Figure 7 shows an example in which a tweet annotates a rectangular section in
an image, which in turn is identified and described by a media fragment URI.

Figure 7: Annotating media segments using a fragment URI.

###  4.2 Describing segments via constraint resources

There are many situations when segments cannot be described with fragment
URIs, but it is still desirable to be able to annotate a segment of a
resource. For example, a non-rectangular section of an image, or a segment of
a resource with a format or media type that is not covered by either fragment
specification, such as a 3-dimensional model or a dataset. To handle these
situations, we introduce an oac:Constraint resource that describes the segment
of interest using an appropriate standard, and a oac:ConstrainedTarget
resource that identifies the segment of interest. This _constrained target_ is
the object of the oac:hasTarget predicate of the oac:Annotation, and
subsequently oac:constrains the full target resource. Figure 8 shows how
constrained targets extend the Open Annotation baseline model.

Figure 8: Constraint annotation targets in the Open Annotation Model.

The nature of the constraint description will be dependent on the type of the
resource for which the segment is being conveyed. It is then up to the
annotation client to interpret the segment description with respect to the
full resource. Figure 9 shows an example in which an area within an image is
described by an SVG path element. The document containing the SVG
specification is identified by a dereferencable HTTP URI and a specialized
type oac:SvgConstraint in combination with a dc:format property informs the
client about the type of constraint it needs to deal with.

Figure 9: Annotating media segments using an SVG constraint.

Alternatively, it is also possible to include the constraint information
inline within the annotation document using the same technique as used for
including the body. The oac:Constraint is given a URN (normally a urn:UUID)
and then the constraint information is included as the value of the cnt:chars
property. The requirements for doing this are the same as for including the
oac:Body inline within the annotation document. For more complex use cases it
also possible to express constraints in RDF and to apply constraints also on
the body of an annotation.

One goal of the ongoing Open Annotation demonstrator activities is to collect
real-world constraint definitions from various use cases and to specify them
in the context of the OA model. We hope that this also serves as as feedback
loop for possible enhancements or additional URI fragment specifications.

##  5 Robust Annotations over Time

It must be stressed that different agents may create the _annotation_ , _body_
and _target_ at different times. For example, Alice might create an annotation
saying that Bob’s YouTube video annotates Carol’s Flickr photo. Also, being
regular Web resources, the body and target are likely to have different
representations over time. Some annotations may apply irrespective of
representation, while others may pertain to specific representations. In order
to provide the ability to accurately interpret annotations past their
publication, the Open Annotation Model introduces three ways to express
temporal context. The manner in which these three types of annotations use the
oac:when property, which has a datetime as its value, distinguishes them.

A _Timeless Annotation_ applies irrespective of the evolving representations
of body and target; it can be considered as if the annotation references the
semantics of the resources. For example, an annotation with a body that says
“This is the front page of CNN” remains accurate as representations of the
target http://cnn.com/ change over time. Timeless annotations don’t make use
of the oac:when property.

A _Uniform Time Annotation_ has a single point in time at which all the
resources involved in the annotation should be considered. This type of
annotation has the oac:when property attached to the oac:Annotation. For
example, if Alice recurrently publishes a tweet that comments on a story on
the live CNN home page, an annotation that has the cartoon as body and the CNN
home page as target would need to be handled as a Uniform Time Annotation in
order to provide the ability to match up correct representations of body and
target. Figure 10 shows how Uniform Time Annotations can be represented using
the Open Annotation Model.

Figure 10: A Uniform Time Annotation example.

A _Varied Time Annotation_ has a body and target that need to be considered at
different moments in time. This type of Annotation uses the oac:when property
attached to an oac:WebTimeConstraint node, which is a specialization of
oac:Constraint, for both body and target. If, in the aforementioned example,
Alice would have the habit to publish a cartoon at http://example.org/cartoon
when the mocked article is no longer on the home page, but still use
http://cnn.com as the target of her annotation, the Varied Time Annotation
approach would have to be used.

This temporal information can be used to recreate the annotation as it was
intended by reconstructing it with the time-appropriate body and target(s).
Previous versions of Web resources exist in archives such as the Internet
Archive, or within content management systems such as MediaWiki’s article
history, however they are divorced from their original URI. Memento Van-de-
Sompel:2009fk ; Sompel:2010fk , which is a framework that proposes a simple
extension of HTTP in order to connect the original and archived resources, can
be applied for recreating annotations. It leverages existing HTTP capabilities
in order to support accessing resource versions through the use of the URI of
a resource and a datetime as the indicator of the required version. In the
framework, a server that host versions of a given resource exposes a TimeGate,
which acts as a gateway to the past for a given Web resource. In order to
facilitate access to a version of that resource, the TimeGate supports HTTP
content negotiation in the datetime dimension. Several mechanisms support
discovery of TimeGates, including HTTP links that point from a resource to its
TimeGate(s) Sanderson:2010fk .

##  6 Related Work

In this section we give an overview of existing work in the area of Web
annotations. After summarizing general works about annotations and annotation
interoperability, we analyze the features of existing Web annotation models
and compare them with those of the Open Annotation Model.

###  6.1 Annotations and annotation interoperability

Annotations have a long research history, and unsurprisingly the research
perspectives and interpretations of what an _annotation_ is supposed to be
vary widely. Agosti et al. Agosti:2007uq  provide a comprehensive study on the
contours and complexity of annotations. A representative discussion on how
annotations can be used in various scholarly disciplines is given by Bradley
Bradley:2008kx . He describes how annotations can support interpretation
development by collecting notes, classifying resources, and identifying novel
relationships between resources.

The different forms and functions that annotations can take are analyzed by
Marshall Marshall:2000kx . She distinguishes between _formal_ and _informal_
annotations, whereby formal annotations follow structural standards and
informal ones are unstructured. Furthermore, Marshall divides into _implicit_
annotations that are intended for sharing and _explicit_ annotations of
personal nature, often interpretable only by the original creator. Further
divisions defined by Marshall with regard to the function of an annotation
include _permanent_ vs. _transient_ , _annotation as writing_ vs. _annotation
as reading_ , _extensive_ vs. _intensive_ , _published_ vs. _private_ and
_institutional_ vs. _workgroup_ vs. _individual_. The difference between
personal and public annotations in a digital environment is further
investigated in a study by Marshall and Brush Marshall2004 . They derive
design implications for annotation systems, e.g. regarding find and filtering
requirements, and user interface strategies for processing and sharing
annotations.

A taxonomy of annotation types and marking symbols used by readers of
scholarly documents is presented by Qayyum Qayyum2008 . His taxonomy is
derived from the results of a user study conducted with students reading
research articles in a private as well as a collaborative digital setting. A
related recent effort is presented by Blustein et al. Blustein2011 . In their
field study, conducted over the course of three years, they identify six
purposes for scholarly annotation: _interpretation_ , _problem-working_ ,
_tracing progress_ , _procedural annotations_ , _place marking and aiding
memory_ and _incidental markings_.

###  6.2 Web annotation models

The idea of publishing user annotations on the Web is not new. Annotea
Kahan:2001vn  was specified more than a decade ago and defines a data model
and protocol for uploading, downloading, and modifying annotations. Since the
Web has changed over time and now also comprises non-document Web resources,
the Annotea model soon became insufficient for many annotation use cases, as
we explained at the beginning of this article. Annotea extensions, such as Co-
Annotea Hunter:2008ab  or LEMO Haslhofer:2009ve , were developed to deal with
the Annotea shortcomings and to take into account emerging architectural
styles, such as RESTful Web Services, or Linked Open Data. Recent Web
annotation model specification efforts include the M3O ontology
Saathoff:2010vn , the Annotation Ontology Ciccarese2011 , and the Open
Annotation Model, which we presented in this article.

Figure 11: Feature analysis of existing Web annotation models.

In Figure 11 we present the results of a feature comparison we performed
across the previously mentioned models. The models are timely ordered by their
publication year, and the feature selection is based on the requirements and
use case descriptions we found in the model documentations. Although we cannot
generalize from this representative set of annotation models and features, we
can observe hat annotation models have continually been adapted to emerging
standards, needs, and architectures: Linked Open Data is increasingly adopted
for publishing and sharing annotations, multimedia resources can now be
annotated also by other multimedia resources, extensibility has become a key
requirement, security and provenance are being outsourced to other models, and
standard segment identification mechanisms are being combined with custom
solutions (context, fragment, selectors) to capture complex domain-specific
needs.

The Annotation Ontology, which is an open ontology for the annotation of
scientific documents on the Web, is technically very similar to the Open
Annotation Model. However, they differ e.g., in terms of how fragments are
being expressed: by representing constraints and constraint targets as first-
class resources the Open Annotation Model supports direct addressing of
fragments, thus enabling use cases where different users annotate the same
fragment, or search scenarios where annotations are retrieved by fragment.
Furthermore, the Open Annotation Model supports structured annotation bodies
and allows to overlay semantic statements pertaining to one or more annotation
targets, which offers potentially more flexibility, e.g., for use cases of
entity and entity-relation extraction in scientific literature.

A related strand of research concerns models for _social tagging_ of Web
resources. Hunter Hunter2009  describes tags as “ _a subclass of annotations
that comprise simple, unstructured labels or keywords assigned to digital
resources to describe and classify the digital resource_ ”. A comparison of
tagging ontologies is presented by Kim et al. KimEtAl2008 . They survey the
state of the art in tagging models and identify three building blocks common
to existing tagging models: _taggers_ , the _tags_ themselves and the
_resources_ being tagged.

Semantic annotations features can also be found in multimedia metadata
frameworks such as MPEG-7 and multimedia metadata ontologies such as COMM
Arndt:2007uq  or the recent _W3C Ontology for Media Resources_
333http://www.w3.org/TR/mediaont-10/#example3 specification, which provides a
core metadata vocabulary for media resources on the Web. It defines two
metadata properties that can be used for the textual description of a media
resource (fragment) or for relating RDF files or named graphs to a media
resource. Other ontologies were designed to embed annotations directly into
the multimedia content representation. The M3O Ontology Saathoff:2010vn , for
instance, allows the integration of annotations with SMIL and SVG documents.
On the contrary, the Open Annotation Model is more in line with the previously
discussed Web Annotation models. It treats annotations as first class Web
resources, which can exist independently from the content or metadata
representations of media objects. This design choice is motivated by a set of
scholarly use cases, which require that multimedia content objects can be
annotated any time after the content production and metadata extraction
process.

###  6.3 Media segment identification

Early related work on the issue of describing segments in multimedia resources
can be traced back to research on linking in hypermedia documents (cf.
Hardman:1994zr ). For describing segments using a non-URI based mechanism one
can use MPEG-7 Shape Descriptors (cf. Nack:1999ly ) or terms defined in a
dedicated multimedia ontology. SVG svg:2003bh  and MPEG-21 ISO/IEC:2006qf
introduced XPointer-based URI fragment definitions for linking to segments in
multimedia resources. The Temporal URI specification Pfe07  addresses a
temporal segment in a time-based media resource through a defined URL query
parameter (’t=’). YouTube supports similar direct linking to a particular
point in time in a video using a fragment URI. The Media Fragments URI
Specification VanDeursen:2010  is a W3C Working Draft that introduces a
standard, URI-based approach for addressing temporal, spatial and track sub-
parts of any non-textual media content, thus making audiovisual media segments
first class citizens on the Web Hausenblas:LDOW09 .

###  6.4 Robustness of Web resources

The ephemeral nature of Web resources and methods to deal with that problem
have been studied from the early years of the Web on. Phelps and Wilensky
Phelps:2000ys  proposed to decorate hyperlinks with lexical signatures to re-
find disappeared web resources. Recent works include Klein et al. Klein:2011kx
, who proposed to compute lexical signatures from the link neighborhood of a
Web page, and Morishima et al. Morishima:2009vn  who describe a method to fix
broken links when link targets have moved. The problem has also been realized
in the Linked Data context and solutions like DSNotify Popitsch:2011zr  were
proposed to re-find resources by their representations.

##  7 Summary and Future Directions

We apply a generic and Web-centric conception to the various facets
annotations can have and regard an annotation as association created between
one _body_ resource and other _target_ resources, where the body must be
somehow _about_ the target. This conception lead to the specification of the
Open Annotation Model, which originates from activities in the Open Annotation
Collaboration and aims at building an interoperable environment for publishing
annotations on the Web.

At the time of this writing, the Open Annotation Model is still in beta stage
and is currently implemented in several demonstration projects covering use
cases ranging from annotating medieval manuscripts, over annotating online
maps, to annotating online video segments. As part of these demonstration
projects client APIs are being developed for various programming environments
to ease model adoption for developers. We expect to obtain user and developer
feedback from these projects, which should further refine the Open Annotation
Model.

Pursuing better integration of the proposed segment identification approach
with the W3C Media Fragment URI specification is on our research agenda.
However, this requires an extension mechanism in that specification, which is
not within the scope of the responsible working group at the moment. Also the
question of how to model multiple annotation targets and how to interpret this
information correctly, is currently being discussed. Finally, as a first
outcome of the demonstration projects, we observed that a common set of use
cases is annotating data with other data, rather than with information
intended for human consumption. This raises the question of how to model _Data
Annotations_ in an interoperable way.

As a final result, we expect a data model, which provides an interoperable
method of expressing annotations such that they can easily be shared between
platforms, with sufficient richness of expression to satisfy also complex
annotation scenarios.

###### Acknowledgements.

The work has partly been supported by the European Commission as part of the
eContentplus program (EuropeanaConnect) and by a Marie Curie International
Outgoing Fellowship within the 7th Europeana Community Framework Program. The
development of OAC is funded by the Andrew W. Mellon foundation.

## References

  * (1) Maristella Agosti, Giorgetta Bonfiglio-Dosio, and Nicola Ferro.  A historical and contemporary study on annotations to derive key features for systems design.  Int. J. on Digital Libraries, 8(1):1–19, 2007.
  * (2) Richard Arndt, Raphaël Troncy, Steffen Staab, Lynda Hardman, and Miroslav Vacura.  COMM: designing a well-founded multimedia ontology for the web.  In The Semantic Web, 6th International Semantic Web Conference, 2nd Asian Semantic Web Conference, ISWC 2007 + ASWC 2007, pages 30–43, Busan, Korea, November 2007.
  * (3) Tim Berners-Lee, Roy Fielding, and Larry Masinter.  Uniform Resource Identifier (URI): Generic Syntax.  IETF, 2005.  Available at: http://tools.ietf.org/html/rfc3986.
  * (4) James Blustein, David Rowe, and Ann-Barbara Graff.  Making sense in the margins: a field study of annotation.  In Proceedings of the 15th international conference on Theory and practice of digital libraries: research and advanced technology for digital libraries, TPDL’11, pages 252–259, Berlin, Heidelberg, 2011. Springer-Verlag.
  * (5) John Bradley.  Pliny: a model for digital support of scholarship.  Journal of Digital Information, 9(1), 2008.
  * (6) Paolo Ciccarese, Marco Ocana, Leyla Jael Garcia Castro, Sudeshna Das, and Tim Clark.  An open annotation ontology for science on web 3.0.  Journal of Biomedical Semantics, 2(2), 2011.
  * (7) Herbert Van de Sompel, Robert Sanderson, Michael L. Nelson, Lyudmila Balakireva, Harihar Shankar, and Scott Ainsworth.  An http-based versioning mechanism for linked data.  CoRR, abs/1003.3661, 2010.
  * (8) Leigh Dodds and Ian Davis.  Linked Data Patterns.  Available at: http://patterns.dataincubator.org/book/index.html.
  * (9) Lynda Hardman, Dick C. A. Bulterman, and Guido van Rossum.  The amsterdam hypermedia model: adding time and context to the dexter model.  Commun. ACM, 37(2):50–62, 1994.
  * (10) Bernhard Haslhofer, Wolfgang Jochum, Ross King, Christian Sadilek, and Karin Schellner.  The LEMO annotation framework: weaving multimedia annotations with the Web.  Int. J. on Digital Libraries, 10(1):15–32, 2009.
  * (11) Bernhard Haslhofer, Rainer Simon, Robert Sanderson, and Herbert van de Sompel.  The Open Annotation Collaboration (OAC) model.  In Workshop on Multimedia on the Web 2011, Graz, Austria, September 2011. IEEE.
  * (12) Michael Hausenblas, Raphaël Troncy, Yves Raimond, and Tobia Bürger.  Interlinking multimedia: How to apply linked data principles to multimedia fragments.  In WWW 2009 Workshop: Linked Data on the Web (LDOW2009), Madrid, Spain, 2009.
  * (13) Tom Heath and Christian Bizer.  Linked Data: evolving the web into a global data space.  Synthesis Lectures on the Semantic Web: Theory and Technology, 1:1, 1-136. Morgan & Claypool, 1st edition edition, 2011.  Available at: http://linkeddatabook.com/editions/1.0/.
  * (14) Jane Hunter.  Collaborative semantic tagging and annotation systems.  Annual Review of Information Science and Technology, 43(1):1–84, 2009.
  * (15) ISO/IEC.  Multimedia framework (MPEG-21) – Part 17: Fragment Identification of MPEG Resources.  International Organization for Standardization, Geneva, Switzerland, December 2006.
  * (16) Ian Jacobs and Norman Walsh.  Architecture of the World Wide Web, Volume One, December 2004.
  * (17) José Kahan and Marja-Ritta Koivunen.  Annotea: an open RDF infrastructure for shared web annotations.  In Proceedings of the 10th international conference on World Wide Web, WWW ’01, pages 623–632, New York, NY, USA, 2001. ACM.
  * (18) Hak L. Kim, Simon Scerri, John G. Breslin, Stefan Decker, and Hong G. Kim.  The state of the art in tag ontologies: a semantic model for tagging and folksonomies.  In DCMI ’08: Proceedings of the 2008 International Conference on Dublin Core and Metadata Applications, pages 128–137. Dublin Core Metadata Initiative, 2008.
  * (19) Martin Klein, Jeb Ware, and Michael L. Nelson.  Rediscovering missing web pages using link neighborhood lexical signatures.  In Proceeding of the 11th annual international ACM/IEEE joint conference on Digital libraries, JCDL ’11, pages 137–140, New York, NY, USA, 2011. ACM.
  * (20) Johannes Koch and Carlos A Velasco.  Representing Content in RDF 1.0.  W3C, 2009.  Available at: http://www.w3.org/TR/Content-in-RDF10/.
  * (21) Marja-Riitta Koivunen.  Semantic authoring by tagging with Annotea social bookmarks and topics.  In In The 5th International Semantic Web Conference (ISWC2006) - 1st Semantic Authoring and Annotation Workshop (SAAW2006), 2006.
  * (22) Catherine C. Marshall.  The future of annotation in a digital (paper) world.  In Harum and Twidale, editors, Successes and Failures of Digital Libraries, pages 97–117. Urbana-Champaign: University of Illinois, 2000.  Available at: http://www.csdl.tamu.edu/~marshall/uiucpaper-complete.pdf.
  * (23) Catherine C. Marshall and A. J. Bernheim Brush.  Exploring the relationship between personal and public annotations.  In Proceedings of the 2004 Joint ACM/IEEE conference on Digital Libraries, pages 349–357, 2004.
  * (24) Atsuyuki Morishima, Akiyoshi Nakamizo, Toshinari Iida, Shigeo Sugimoto, and Hiroyuki Kitagawa.  Bringing your dead links back to life: a comprehensive approach and lessons learned.  In Proceedings of the 20th ACM conference on Hypertext and hypermedia, HT ’09, pages 15–24, New York, NY, USA, 2009. ACM.
  * (25) Frank Nack and Adam T. Lindsay.  Everything You Wanted to Know About MPEG-7: Part 2.  IEEE MultiMedia, pages 64–73, October - December 1999.
  * (26) Silvia Pfeiffer.  Hyperlinking to time offsets: The temporal uri specification, 12 2007\.
  * (27) Thomas A. Phelps and Robert Wilensky.  Robust hyperlinks cost just five words each.  Technical Report UCB/CSD-00-1091, EECS Department, University of California, Berkeley, 2000.
  * (28) Niko Popitsch and Bernhard Haslhofer.  Dsnotify - a solution for event detection and link maintenance in dynamic datasets.  Web Semantics: Science, Services and Agents on the World Wide Web, 9(3):266 – 283, 2011.
  * (29) Asim Qayyum.  Analysing markings made on e-documents.  Canadian Journal of Information & Library Sciences, 32(1/2):35–53, 2008.
  * (30) Carsten Saathoff and Ansgar Scherp.  Unlocking the semantics of multimedia presentations in the web with the multimedia metadata ontology.  In Proceedings of the 19th international conference on World wide web, WWW ’10, pages 831–840, New York, NY, USA, 2010. ACM.
  * (31) Robert Sanderson, Benjamin Albritton, Rafael Schwemmer, and Herbert Van de Sompel.  Sharedcanvas: a collaborative model for medieval manuscript layout dissemination.  In Proceeding of the 11th annual international ACM/IEEE joint conference on Digital libraries, JCDL ’11, pages 175–184, New York, NY, USA, 2011. ACM.
  * (32) Robert Sanderson and Herbert Van de Sompel.  Making web annotations persistent over time.  In Proceedings of the 10th annual joint conference on Digital libraries, JCDL ’10, pages 1–10, New York, NY, USA, 2010. ACM.
  * (33) Ronald Schroeter, Jane Hunter, and Andrew Newman.  Annotating relationships between multiple mixed-media digital objects by extending annotea.  In Enrico Franconi, Michael Kifer, and Wolfgang May, editors, ESWC, volume 4519. Springer, 2007.
  * (34) Jane Hunter and Ronald Schroeter.  Co-Annotea: A system for tagging relationships between multiple mixed- media objects  IEEE Multimedia, 15(3), pages 42–53, 2008.
  * (35) Rainer Simon, Bernhard Haslhofer, Werner Robitza, and Elaheh Momeni.  Semantically augmented annotations in digitized map collections.  In Proceeding of the 11th annual international ACM/IEEE joint conference on Digital libraries, JCDL ’11, pages 199–202, New York, NY, USA, 2011. ACM.
  * (36) Herbert Van de Sompel, Michael L. Nelson, Robert Sanderson, Lyudmila L. Balakireva, Scott Ainsworth, and Harihar Shankar.  Memento: Time travel for the web.  Arxiv preprint., arxiv:0911.1112, November 2009.
  * (37) Davy Van Deursen, Raphaël Troncy, Erik Mannens, Silvia Pfeiffer, Yves Lafon, and Rik Van de Walle.  Implementing the media fragments uri specification.  In Proceedings of the 19th international conference on World wide web, WWW ’10, pages 1361–1364, New York, NY, USA, 2010. ACM.
  * (38) Karin Verspoor, Judith Cohn, Cliff Joslyn, Sue Mniszewski, Andreas Rechtsteiner, Luis Rocha, and Tiago Simas.  Protein annotation as term categorization in the gene ontology using word proximity networks.  BMC Bioinformatics, 6(1), 2005.
  * (39) W3C Media Fragments Working Group.  Media Fragments URI 1.0.  W3C, 2011.  Available at: http://www.w3.org/TR/media-frags/.
  * (40) W3C SVG Working Group.  Scalable Vector Graphics (SVG) — XML Graphics for the Web.  W3C, 2003.  Available at: http://www.w3.org/Graphics/SVG/.