Autonomous drone hunter operating by deep learning and all-onboard computations in GPS-denied environments

Autoři: Philippe Martin Wyder aff001;  Yan-Song Chen aff002;  Adrian J. Lasrado aff001;  Rafael J. Pelles aff001;  Robert Kwiatkowski aff002;  Edith O. A. Comas aff002;  Richard Kennedy aff002;  Arjun Mangla aff002;  Zixi Huang aff003;  Xiaotian Hu aff003;  Zhiyao Xiong aff001;  Tomer Aharoni aff002;  Tzu-Chan Chuang aff002;  Hod Lipson aff001
Působiště autorů: Department of Mechanical Engineering, Columbia University, New York, New York, United States of America aff001;  Department of Computer Science, Columbia University, New York, New York, United States of America aff002;  Department of Electrical Engineering, Columbia University, New York, New York, United States of America aff003
Vyšlo v časopise: PLoS ONE 14(11)
Kategorie: Research Article
doi: 10.1371/journal.pone.0225092


This paper proposes a UAV platform that autonomously detects, hunts, and takes down other small UAVs in GPS-denied environments. The platform detects, tracks, and follows another drone within its sensor range using a pre-trained machine learning model. We collect and generate a 58,647-image dataset and use it to train a Tiny YOLO detection algorithm. This algorithm combined with a simple visual-servoing approach was validated on a physical platform. Our platform was able to successfully track and follow a target drone at an estimated speed of 1.5 m/s. Performance was limited by the detection algorithm’s 77% accuracy in cluttered environments and the frame rate of eight frames per second along with the field of view of the camera.

Klíčová slova:

Algorithms – Cameras – Computer imaging – Computers – Flight testing – Machine learning algorithms – Neural networks – Target detection


1. Federal Aviation Administration. FAA Aerospace Forecast, FY2018-38. 2018. doi: 10.1017/CBO9781107415324.004

2. Wall R. U.K. Airport Remains Closed After Drones Disrupt Travel—WSJ. In: The Wall Street Journal [Internet]. 2018 [cited 11 Jan 2019]. Available:

3. Rizzo C. Drones Force London Airport to Shut Down, Leaving Thousands Stranded Throughout Europe | Travel + Leisure. In: Travel+Leisure [Internet]. 2018 [cited 11 Jan 2019]. Available:

4. FOX 5. Delta jet spots drone near JFK Airport—Story | WNYW. 2015 [cited 11 Jan 2019]. Available:

5. Wong D. Simmering debate over drone use in condos, as more move to ban the flying objects, Housing News & Top Stories—The Straits Times. In: The Straits Times [Internet]. 2018 [cited 11 Jan 2019]. Available:

6. Richey E. Peeping drones spying on people in St. Louis | In: KSDK [Internet]. 2018 [cited 11 Jan 2019]. Available:

7. Gibbons-Neff T. ISIS drones are attacking U.S. troops and disrupting airstrikes in Raqqa, officials say—The Washington Post. In: The Washington Post [Internet]. 2017 [cited 10 Jan 2019]. Available:

8. Lohr D. Gang Used Drone Swarm To Thwart FBI Hostage Raid | HuffPost. In: Huffington Post [Internet]. 2018 [cited 10 Jan 2019]. Available:

9. Dinan S. Mexican drug cartels using drones to smuggle heroin, meth, cocaine into U.S.—Washington Times. In: The Washington Times [Internet]. 2017 [cited 11 Jan 2019]. Available:

10. BBC. Gang who flew drones carrying drugs into prisons jailed—BBC News. 2018 [cited 11 Jan 2019]. Available:

11. Lu Y, Xue Z, Xia GS, Zhang L. A survey on vision-based UAV navigation. Geo-Spatial Inf Sci. 2018. doi: 10.1080/10095020.2017.1420509

12. Balamurugan G, Valarmathi J, Naidu VPS. Survey on UAV navigation in GPS denied environments. International Conference on Signal Processing, Communication, Power and Embedded System, SCOPES 2016—Proceedings. 2017. doi: 10.1109/SCOPES.2016.7955787

13. Rady S, Kandil AA, Badreddin E. A hybrid localization approach for UAV in GPS denied areas. 2011 IEEE/SICE International Symposium on System Integration, SII 2011. 2011. doi: 10.1109/SII.2011.6147631

14. Battelle Memorial Institute. Battelle DroneDefender® Counter-UAS Device. [cited 10 Jan 2019]. Available:

15. Wyant RT. Skynet is real! An anti-drone round. Small Arms Def J. 2016;8. Available:

16. OPENWORKS ENGINEERING LTD. SkyWall. [cited 10 Jan 2019]. Available:

17. AIRSPACE SYSTEMS I. Airspace. [cited 10 Jan 2019]. Available:

18. Mannes J. Airspace Systems’ ‘Interceptor’ can catch high-speed drones all by itself | TechCrunch. In: Tech Crunch [Internet]. 2017 [cited 9 Jan 2019]. Available:

19. Honegger D, Meier L, Tanskanen P, Pollefeys M. An open source and open hardware embedded metric optical flow CMOS camera for indoor and outdoor applications. Proceedings—IEEE International Conference on Robotics and Automation. 2013. doi: 10.1109/ICRA.2013.6630805

20. ARDUPILOT Dev Team. Companion Computers—Dev documentation. [cited 11 Jan 2019]. Available:

21. Dronecode Project Inc. Companion Computers · PX4 Developer Guide. [cited 11 Jan 2019]. Available:

22. Ong T. Dutch police will stop using drone-hunting eagles since they weren’t doing what they’re told. In: The Verge [Internet]. 2017 [cited 10 Jan 2019] p. 1. Available:

23. Selk A. Terrorists are building drones. France is destroying them with eagles.—The Washington Post. In: The Washington Post [Internet]. 2017 [cited 10 Jan 2019]. Available:

24. Opromolla R, Fasano G, Accardo D. A vision-based approach to uav detection and tracking in cooperative applications. Sensors (Switzerland). 2018;18. doi: 10.3390/s18103391 30309035

25. Xiang T, Jiang F, Lan G, Sun J, Liu G, Hao Q, et al. UAV based target tracking and recognition. IEEE Int Conf Multisens Fusion Integr Intell Syst. 2017; 400–405. doi: 10.1109/MFI.2016.7849521

26. Nvidia. Jetson TX2 Module | NVIDIA Developer. In: [Internet]. [cited 11 Jan 2019]. Available:

27. Dronecode Project Inc. Open Source for Drones—PX4 Open Source Autopilot. [cited 11 Jan 2019]. Available:

28. Redmon JSDRGAF. (YOLO) You Only Look Once. Cvpr. 2016. doi: 10.1109/CVPR.2016.91

29. Shah S, Dey D, Lovett C, Kapoor A. AirSim: High-Fidelity Visual and Physical Simulation for Autonomous Vehicles. 2017 [cited 10 Jan 2019]. doi: 10.1007/978-3-319-67361-5_40

30. Arreola L, Gudiño G, Flores G. Object recognition and tracking using Haar-like Features Cascade Classifiers: Application to a quad-rotor UAV. 2019 [cited 27 Aug 2019]. Available:

31. Girshick R, Donahue J, Darrell T, Malik J. Rich feature hierarchies for accurate object detection and semantic segmentation. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition. 2014. doi: 10.1109/CVPR.2014.81

32. Ren S, He K, Girshick R, Sun J. Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks. IEEE Trans Pattern Anal Mach Intell. 2017. doi: 10.1109/TPAMI.2016.2577031 27295650

33. Lin TY, Dollár P, Girshick R, He K, Hariharan B, Belongie S. Feature pyramid networks for object detection. Proceedings - 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017. 2017. doi: 10.1109/CVPR.2017.106

34. Redmon J, Farhadi A. YOLO. Plant Soil. 2001. doi: 10.1023/A:1011909414400

35. Redmon J, Farhadi A. YOLO9000: Better, faster, stronger. Proceedings - 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017. 2017. doi: 10.1109/CVPR.2017.690

36. Redmon J, Farhadi A. yolov3. Proc - 30th IEEE Conf Comput Vis Pattern Recognition, CVPR 2017. 2017. doi: 10.1109/CVPR.2017.690

37. Fu C-Y, Liu W, Ranga A, Tyagi A, Berg AC. DSSD: Deconvolutional Single Shot Detector. CoRR. 2017;abs/1701.0. Available:

38. Liu W, Anguelov D, Erhan D, Szegedy C, Reed S, Fu CY, et al. SSD: Single shot multibox detector. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). 2016. doi: 10.1007/978-3-319-46448-0_2

39. Lin TY, Goyal P, Girshick R, He K, Dollar P. Focal Loss for Dense Object Detection. Proceedings of the IEEE International Conference on Computer Vision. 2017. doi: 10.1109/ICCV.2017.324

40. Redmon J. Darknet: Open Source Neural Networks in C. [cited 1 Sep 2018]. Available:

41. Stereolabs Inc. SDK Introduction | ZED API. [cited 10 Jan 2019]. Available:

42. Marvelmind Robotics. Starter Set—HW v4.9 + IMU—Marvelmind Robotics. [cited 11 Jan 2019]. Available:

43. Open Source Robotics Foundation. | Powering the world’s robots. [cited 11 Jan 2019]. Available:

44. Nvidia Corp. Jetson AGX Xavier Developer Kit | NVIDIA Developer. In: [Internet]. 2018 [cited 12 Jan 2019]. Available:

45. Wu Y, Sui Y, Wang G. Vision-Based Real-Time Aerial Object Localization and Tracking for UAV Sensing System. IEEE Access. 2017;5: 23969–23978. doi: 10.1109/ACCESS.2017.2764419

46. Nikolic J, Rehder J, Burri M, Gohl P, Leutenegger S, Furgale P, et al. A Synchronized Visual-Inertial Sensor System with FPGA Pre-Processing for Accurate Real-Time SLAM. IEEE Int Conf Robot Autom (ICRA), 2014 May 31, 2014—June 7, 2014, Hong Kong, China. 2014; 431–437. doi: 10.3929/ETHZ-A-010061790

47. Zhang Z, Suleiman AA, Carlone L, Sze V, Karaman S. Visual-Inertial Odometry on Chip: An Algorithm-and-Hardware Co-design Approach. Sze. 2017 [cited 30 Aug 2019]. Available:

Článek vyšel v časopise


2019 Číslo 11