Abstract
This paper presents four vision-based tracking system architectures for marine surface objects using a fixed-wing unmanned aerial vehicle (UAV) with a thermal camera mounted in a pan/tilt gimbal. The tracking systems estimate the position and velocity of an object in the North-East (NE) plane, and differ in how the measurement models are defined. The first tracking system measures the position and velocity of the target with georeferencing and optical flow. The states are estimated in a Kalman filter. A Kalman filter is also utilized in the second architecture, but only the georeferenced position is used as a measurement. A bearing-only measurement model is the basis for the third tracking system, and because the measurement model is nonlinear, an extended Kalman filter is used for state estimation. The fourth tracking system extends the bearing-only tracking system to let navigation uncertainty in the UAV position affect the target estimates in a Schmidt-Kalman filter. All tracking architectures are evaluated on data gathered at a flight experiment near the Azores islands outside of Portugal. The results show that various marine vessels can be tracked quite accurately.
Article PDF
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Avoid common mistakes on your manuscript.
References
COLREGs - convention on the international regulations for preventing collisions at sea, International Maritime Organization (IMO) (1972)
Bar-Shalom, Y., Li, X.R., Kirubarajan, T.: Estimation with Applications to Tracking and Navigation: Theory Algorithms and Software. Wiley, Hoboken (2004)
Barber, D.B., Redding, J.D., McLain, T.W., Beard, R.W., Taylor, C.N.: Vision-based target geo-location using a fixed-wing miniature air vehicle. J. Intell. Robot. Syst. 47(4), 361–382 (2006)
Horn, B.K.P., Schunk, B.G.: Determining optical flow. Artif. Intell. 17, 185–204 (1981)
Bradski, G.: The opencv library. Dr Dobb’s Journal of Software Tools (2000)
Bryson, M., Sukkarieh, S.: Bearing-Only Slam for an Airborne Vehicle. In: Proceedings of the Australasian Conference on Robotics and Automation, vol. 4 (2005)
Bryson, M., Sukkarieh, S.: Building a robust implementation of bearing-only inertial slam for a uav. J. Field Rob. 24(1-2), 113–143 (2007)
Doherty, P., Rudol, P.: A uav search and rescue scenario with human body detection and geolocalization. Lect. Notes Comput. Sci (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) 4830, 1–13 (2007)
Dumble, S.J., Gibbens, P.W.: Efficient terrain-aided visual horizon based attitude estimation and localization. J. Intell. Robot. Syst. 78(2), 205–221 (2015)
Egeland, O., Gravdahl, J.T.: Modeling and Simulation for Automatic Control. Marine Cybernetics Trondheim, Norway (2002)
Elkins, L., Sellers, D., Monach, W.R.: The autonomous maritime navigation (amn) project: Field tests, autonomous and cooperative behaviors, data fusion, sensors, and vehicles. J. Field Rob. 27(6), 790–818 (2010)
Fossen, T.: Handbook of Marine Craft Hydrodynamics and Motion Control. Wiley, Hoboken (2011)
Fusini, L., Hosen, J., Helgesen, H.H., Johansen, T.A., Fossen, T.I.: Experimental validation of a uniformly semi-globally exponentially stable non-linear observer for gnss- and camera-aided inertial navigation for fixed-wing Uavs. In: Proceedings of the International Conference on Unmanned Aircraft Systems, pp. 851–860 (2015)
Gomez-Balderas, J.E., Flores, G., García Carrillo, L.R., Lozano, R.: Tracking a ground moving target with a quadrotor using switching control. J. Intell. Robot. Syst. 70(1), 65–78 (2013)
Hansen, J., Fossen, T., Johansen, T.: Nonlinear observer for Ins aided by time-delayed Gnss measurements: implementation and Uav experiments. In: Proceedings of the International Conference on Unmanned Aircraft Systems, pp. 157–166 (2015)
Helgesen, H.H., Leira, F.S., Johansen, T.A., Fossen, T.I.: Tracking of marine surface objects from unmanned aerial vehicles with a pan/tilt unit using a thermal camera and optical flow. In: Proceedings of the International Conference on Unmanned Aircraft Systems, pp. 107–117 (2016)
Helgesen, H.H., Leira, F.S., Johansen, T.A., Fossen, T.I.: Detection and tracking of floating objects using a Uav with thermal camera. In: Fossen, T.I., Pettersen, K.Y., Nijmeijer, H. (eds.) Sensing and Control for Autonomous Vehicles: Applications to Land, Water and Air Vehicles, pp. 289–316. Springer International Publishing (2017)
Hemerly, E.M.: Automatic georeferencing of images acquired by uav’s. Int. J. Autom. Comput. 11(4), 347–352 (2014)
Hiba, A., Zsedrovits, T., Bauer, P., Zarandy, A.: Fast horizon detection for airborne visual systems. In: Proceedings of the International Conference on Unmanned Aircraft Systems, pp. 886–891 (2016). https://doi.org/10.1109/ICUAS.2016.7502661
Hosen, J., Helgesen, H.H., Fusini, L., Fossen, T.I., Johansen, T.A.: A vision-aided nonlinear observer for fixed-wing Uav navigation. In: Proceedings of the AIAA Guidance, Navigation, and Control Conference (2016)
Hosen, J., Helgesen, H.H., Fusini, L., Fossen, T.I., Johansen, T.A.: Vision-aided nonlinear observer for fixed-wing unmanned aerial vehicle navigation. J. Guid. Control. Dyn. 39(8), 1777–1789 (2016)
Huntsberger, T., Aghazarian, H., Howard, A., Trotz, D.C.: Stereo vision–based navigation for autonomous surface vessels. J. Field Rob. 28(1), 3–18 (2011)
Hutchinson, S., Hager, G., Corke, P.: A tutorial on visual servo control. IEEE Trans. Robot. Autom. 12(5), 651–670 (1996)
Johansen, T., Fossen, T.: Nonlinear filtering with exogenous Kalman filter and double Kalman filter. In: Proceedings of the European Control Conference, Aalborg (2016)
Johansen, T.A., Perez, T.: Unmanned aerial surveillance system for hazard collision avoidance in autonomous shipping. In: Proceedings of the International Conference on Unmanned Aircraft Systems (2016)
Kadyrov, A., Yu, H., Liu, H.: Ship detection and segmentation using image correlation. In: Proceedings of the IEEE International Conference on Systems, Man, and Cybernetics, pp. 3119–3126 (2013)
Kim, J.H., Sukkarieh, S.: Airborne simultaneous localisation and map building. In: Proceedings of the IEEE International Conference on Robotics and Automation, vol. 1 (2003)
Leira, F.S., Johansen, T.A., Fossen, T.I.: Automatic detection, classification and tracking of objects in the ocean surface from Uavs using a thermal camera. In: Proceedings of the IEEE Aerospace Conference, Big Sky, US (2015)
Leira, F.S., Trnka, K., Fossen, T.I., Johansen, T.A.: A ligth-weight thermal camera payload with georeferencing capabilities for small fixed-wing Uavs. In: Proceedings of the International Conference on Unmanned Aircraft Systems, pp. 485–494 (2015)
Li, X.R., Jilkov, V.P.: Survey of maneuvering target tracking: dynamic models. In: Proceedings of SPIE, vol. 4048, pp. 212–235 (2000)
Lowe, D.: Object recognition from local scale-invariant features. In: Proceedings of the International Conference on Computer Vision, pp. 1150–1157 (1999)
Mammarella, M., Campa, G., Fravolini, M.L., Napolitano, M.R.: Comparing optical flow algorithms using 6-dof motion of real-world rigid objects. IEEE Trans. Syst. Man Cybern. 42(6), 1752–1762 (2012)
Martínez, C., Mondragón, I.F., Campoy, P., Sánchez-López, J.L., Olivares-Méndez, M.A.: A hierarchical tracking strategy for vision-based applications on-board uavs. J. Intell. Robot. Syst. 72(3), 517–539 (2013)
Muja, M., Lowe, D.G.: Flann, fast library for approximate nearest neighbors. In: Proceedings of the International Conference on Computer Vision Theory and Applications (2009)
Novoselov, R.Y., Herman, S.M., Gadaleta, S.M., Poore, A.B.: Mitigating the effects of residual biases with Schmidt-Kalman filtering. In: Proceedings of the 8Th International Conference on Information Fusion, vol. 1 (2005)
Ortiz, A., Bonnin-Pascual, F., Garcia-Fidalgo, E.: Vessel inspection: a micro-aerial vehicle-based approach. J. Intell. Robot. Syst. 76(1), 151–167 (2014)
Rudol, P., Doherty, P.: Human body detection and geolocalization for Uav search and rescue missions using color and thermal imagery. In: Proceedings of the IEEE Aerospace Conference (2008)
Wilthil, E.F., Brekke, E.F.: Compensation of navigation uncertainty for target tracking on a moving platform. In: Proceedings of the 19Th International Conference on Information Fusion, pp. 1616–1621 (2016)
Wolf, M.T., Assad, C., Kuwata, Y., Howard, A., Aghazarian, H., Zhu, D., Lu, T., Trebi-Ollennu, A., Huntsberger, T.: 360-degree visual detection and target tracking on an autonomous surface vehicle. J. Field Rob. 27(6), 819–833 (2010)
Yang, C., Blasch, E., Douville, P.: Design of Schmidt-Kalman filter for target tracking with navigation errors. In: Proceedings of the IEEE Aerospace Conference (2010)
Yilmaz, A., Javed, O., Shah, M.: Object tracking: a survey. ACM Comput. Surv. 38(13), 1–45 (2006)
Yu, X., Zhang, Y.: Sense and avoid technologies with applications to unmanned aircraft systems: review and prospects. Prog. Aerosp. Sci. 74, 152–166 (2015)
Zhao, S., Lin, F., Peng, K., Dong, X., Chen, B.M., Lee, T.H.: Vision-aided estimation of attitude, velocity, and inertial measurement bias for uav stabilization. J. Intell. Robot. Syst. 81(3), 531–549 (2016)
Zhou, G., Li, C., Cheng, P.: Unmanned Aerial Vehicle (Uav) real-time video registration for forest fire monitoring. In: Proceedings of IEEE International Geoscience and Remote Sensing Symposium, vol. 3, pp. 1803–1806 (2005)
Acknowledgements
This work has been carried out at the NTNU Center for Autonomous Marine Operations and Systems. This work was supported by the Research Council of Norway through the Centers of Excellence funding scheme, Project number 223254. The authors are grateful for the support from the University of Porto, the University of the Azores, UAV operators Lars Semb and Krzysztof Cisek, REP15 coordinators João Tasso Sousa and Kanna Rajan and valuable feedback from Edmund Førland Brekke.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Helgesen, H.H., Leira, F.S., Fossen, T.I. et al. Tracking of Ocean Surface Objects from Unmanned Aerial Vehicles with a Pan/Tilt Unit using a Thermal Camera. J Intell Robot Syst 91, 775–793 (2018). https://doi.org/10.1007/s10846-017-0722-3
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10846-017-0722-3