Computational methods for autonomous multirotor drone landing

Abstract

This dissertation presents findings on the topic of autonomous multirotor drone landing — one basic area of multirotor drone flight that is not yet fully automated — with emphasis on real world proofs of concept. We conduct two phases of research, focusing first on landing sites that are structured with fiducial markers, and next on unstructured landing sites, where the drone cannot expect to detect existing infrastructure. The first phase is a continuation of the author’s master thesis which proposes an autonomous landing method based on fiducial markers and a gimbal mounted camera that is tested in simulation. We migrate the method from simulation to the real world and then expand on it. The initial migration revealed problems in recognizing the orientation of the fiducial markers in the real world, which was obscured by idealized graphics in simulation. We quantify this orientation ambiguity in several fiducial systems and carry out a real world landing experiment without mitigating the orientation ambiguity to gauge its effects. Finally, we develop a method for avoiding this issue entirely by directing the drone based solely on the angle from the drone to the landing pad. We demonstrate this method in the real world with both visual and infrared fiducial markers. The infrared markers can serve as landing site identification infrastructure both at daytime and nighttime, and can even be unpowered. Our overall contributions on this front are 1. some modifications to existing fiducial systems to mitigate orientation ambiguity and decrease runtime computational requirements, 2. a test of the effects of orientation ambiguity on the feasibility of autonomous landing when depending primarily on fiducial pose estimation, and 3. a minimalistic method for autonomous drone landing that uses fewer data points than existing methods (primarily the pixel position of the marker and orientation of the gimbal, and avoiding use of the altitude and range) to avoid the orientation ambiguity issue and allow flexibility in changing between modalities (visible and infrared) and changing the landing pad size without reconfiguration. The second phase involves analyzing the terrain beneath the drone to determine if it is safe for landing. We focus on suburban environments as an easier test case, and on lava fields as a more challenging test case that is plentiful and relevant in Iceland. We develop a pipeline for creating appearance based terrain classifiers to automatically locate safe landing sites with a typical, monocular camera using supervised learning methods for image segmentation. To avoid hand labeling thousands of images, we generate a synthetic training data set from geometric surveys in analog environments — similar to the target landing environments, but not exactly the same. We train a U Net to do this task, but the pipeline is modular such that it is possible to use other methods. We evaluate the method synthetically and validate it on real world data to ensure that it has not overfitted to the synthetic data. The method shows success in classifying the training and validation sets in both scenarios, but can provide erroneous classifications when obstacles are too far away, or when obstacles cannot be easily identified by their appearance (e.g., safe, level gravel safe versus unsafe gravel slanted at 45 deg). We therefore supplement this method with a geometric check that uses a depth camera at low altitude (3.5 m – 5 m) which can prevent unsafe landing at erroneously classified sites. Our contributions on this front are: 1. a full pipeline to create visual terrain classifiers from the data collection stage to the real world deployment stage, 2. guidelines and scripts for automatically creating the custom synthetic data sets (particularly useful for environments with few or no publicly available topographical data sets), 3. a successful terrain classifier that goes beyond previous methods by being relatively tiny (1 2 MB), operating at varying camera angles and which is applicable to any environment for which it has training data (not limited to, e.g., urban environments), and 4. a real world demonstration of the method onboard a drone in real time. Real world testing requires a lot of engineering overhead, which itself is useful for other researchers. Therefore, we describe our systems and payloads in detail so they can be reproduced by others. We carry out autonomous landing experiments using a DJI Spark and the DJI Mobile SDK (MSDK) for autonomous control, which gives insight into how to add custom computers into the control loop when using the MSDK. Finally, we develop 3 payloads for the DJI Matrice 350 that add onboard computers with varying degrees of integration and autonomous control, and we provide guidelines for reproducing them. We target the Raspberry Pi series as the main computing board for these payloads in the hopes that the wider community of drone users can take advantage of them in a number of different fields with relative ease, given the already booming Raspberry Pi community that ensures compatibility with a huge range of other hardware and software. Our contributions on this front are: 1. guidelines for creating a fully integrated payload for the DJI Matrice series and other DJI drones, designs for the 3D printed case for such payload with quick release brackets for easy installation and removal in the field, and many field tests and demonstrations of the payload while testing the autonomous landing methods mentioned above.