Decentralized swarms of unmanned aerial vehicles for search and rescue operations without explicit communication

Abstract

In this paper, we introduce a distributed autonomous flocking behavior of Unmanned Aerial Vehicles (UAVs) in demanding outdoor conditions, motivated by search and rescue applications. We propose a novel approach for decentralized swarm navigation in the direction of a candidate object of interest (OOI) based on real-time detections from onboard RGB cameras. A novel self-adaptive communication strategy secures an efficient change of swarm azimuth to a higher priority direction based on the real-time detections. We introduce a local visual communication channel that establishes a network connection between neighboring robots without explicit communication to achieve high reliability and scalability of the system. As a case study, this novel method is applied for the deployment of a UAV swarm towards detected OOI for closer inspection and verification. The results of simulations and real-world experiments have verified the intended behavior of the swarm system for the detection of true positive and false positive OOI, as well as for cooperative environment exploration.

1 Introduction

Unmanned Aerial Vehicles (UAVs) are applicable for various scenarios related to this work of which most promise great potential for multi-robot deployment, including mapping of inaccessible areas (Miiller et al., 2018), inspection of industrial complexes (Eudes et al., 2018), and area surveillance (Stasinchuk et al., 2021). Swarm systems (Chung et al., 2018) increase the scalability and reliability required for applications such as environmental exploration, monitoring (Carpentiero et al., 2017), and search and rescue operations (Couceiro et al., 2013).

In this paper, we present the results of a novel decentralized swarming approach for search and rescue operations in demanding outdoor environments, specifically that of desert and dunes (Fig. 1). The proposed work builds on some enabling technologies for compact swarm stabilization (Saska et al. 2017; Petracek et al. 2020; Ahmad et al. 2021) designed by the Multi-robot Systems Group at Czech Technical University in Prague. This proposed approach surpasses the preliminary works by designing a new decentralized control and coordination rules for the exploration of an area containing a set of OOI, which are difficult to identify with onboard sensors. The swarm must also deal with multiple false positive detections in a strictly decentralized way. The deployment of robots in real time without any explicit communication places the demand on research for a decentralized, decision-making mechanism based on behavioral observation of local teammates within large swarms. Practical motivations for this research can include missions for locating missing persons or in searching for debris, such as in the event of an aircraft crash where the swarm may be tasked to locate and mark OOI (e.g., survivors, or a flight data recorder). To fully exploit the distributed sensory capabilities of an agile UAV swarm for such applications, an online object detector within the feedback of the swarm control architecture is required. The appearance of false positive detections frequently occurs in the real-world deployment of autonomous sensory units. As such, swarm intelligence can increase system reliability in the event of such phenomena. We propose using flexible swarm architecture for verification of the positivity of any OOI detections, while seeking to preserving the cohesion and reliability of the group. The detections are efficiently verified by the approaching swarm using distributed sensory data, such as camera images from different view angles and distances. To achieve this, a consensus on the swarm’s flight direction must be reached to secure the cohesiveness of the swarm during a change of the group target. Such an agreement may be reached using decentralized navigation rules based on limited communication among swarm members.

A radio channel (Chen et al., 2020) is a standard way to establish a communication channel between cooperating flying robots. However, using explicit radio communication to provide relative localization between UAVs (e.g., by sharing GPS data) requires all-to-all communication, which brings an unnecessary communication load. Furthermore, a radio communication channel can be unreliable, particularly in demanding environmental conditions, or it may become saturated in a swarm with a high number of agents. A visual local communication channel is an appropriate alternative to a radio channel. This approach secures connection and data transmission between swarm neighbors in local proximity only, providing system reliability, robustness, and, primarily, scalability (Chowdhury et al. 2018; Cheng et al. 2017). Regardless of the theoretical communication capabilities of modern radio modules, local visual communication represents a general solution independent of environmental conditions and number of robots.

Fig. 1

UAV swarm exploring a desert environment in a decentralized search and rescue task without wireless communication among the robots

It is desirable to secure system responsiveness in the case of a verified true positive OOI detection (Nascimento et al., 2013). In SAR missions, one of the applicable options for human first responders is to deploy a swarm agent towards an OOI. This agent can serve as a static beacon for the human SAR team, can inspect the detected location in detail, or deliver first aid supplies (see Fig. 1). The rest of the swarm can effectively continue field exploration in order to search for remaining OOIs. To preserve swarm cohesion, it is necessary to dynamically update the cohesion and separation influence between swarm agents in the UAV deployment area. This necessity is further addressed in this paper.

1.1 Related work

In Cardona et al. (2019), the authors showed the ability of a swarm system to fulfill a SAR task, including exploration in areas that are potentially dangerous for human workers. A multi-UAV path planning with explicit communication for task allocation within SAR was introduced in (Hayat et al., 2020). Another paper dealing with the deployment of swarms in SAR (Arnold et al., 2018) is focused on the simulation of a group of UAVs cooperatively searching for survivors. The authors designed reactive swarm behavior, including collision avoidance, formation control, and a method for optimizing the coverage area of cameras. In comparison to these papers, we consider relative localization of neighboring UAVs, implicit mutual communication, and onboard OOI detection with problems created by demanding outdoor conditions in real-world missions.

One of the crucial tasks of swarm exploration is navigation of the flocking system. In Zhao et al. (2018), GNSS localized swarms of robots were tasked to follow a pre-planned path where a virtual guidance force was introduced to lead swarm members along the path. Navigation agreement among agents was secured by following the informed agent. Another approach introduced in Madridano et al. (2021) and tested in simulations proposes a multi-layer architecture for the coordinated navigation of a UAV swarm. Leader-follower decentralized navigation for group of ground robots relying on motion capture localization infrastructure and explicit communication was proposed and experimentally verified in Yoshimoto et al. (2018). In our proposed approach, we consider the realistic assumption of unavailable external global localization. The swarm system was designed to rely on relative-localization of neighboring UAVs with a decentralized navigation consensus using real-time selection of leaders. We rely on a navigation consensus algorithm that was proposed for ground robots in Ferrante et al. (2014), where only some swarm agents are informed about the desired goal location.

Another aspect of swarm SAR operations is communication between neighbors, which is important for decision-making and swarm behavior consensus. In Wiltsche et al. (2013), an asynchronous protocol for communication between SAR robots was presented. The protocol was used together with cooperatively-controlled robots searching for a moving target. A routing protocol above Wi-Fi standard, called CHOPIN, was proposed in Araujo et al. (2014) for communication between the places of collaborating robots where a third-party communication network was unavailable. An approach based on the minimalistic communication between fireflies was introduced in Varughese et al. (2020), Hornischer et al. (2021). The presented single-bit information is diffused in a wave through the swarm to secure self-organization. The proposed approach goes beyond these works by offering an enhanced localization-communication system suitable for large scale swarm decision making without an explicit communication network.

We found the Cardona and Calderon (2019) paper to be the most relevant work compared to this approach. A swarm system for performing SAR missions in a simulated environment was introduced in Cardona and Calderon (2019). In addition to proximal control, the authors use a virtual attraction force towards victims. The paper also describe the generation of a sub-swarm to fulfill a subtask around a victim. However, agent localization, mutual communication, and specific object detection were not considered. Furthermore, we consider the negative effects of demanding outdoor environments within the design of the flocking system and demonstrate the success of our solution in several different real-world experiments.

1.2 Contribution

The purpose of the proposed paper is to bridge the laboratory-reality gap in swarm deployment, as well as to show the potential in deploying a team of UAVs for the solution of demanding tasks. Herein, we explore a novel flocking approach for the autonomous exploration of demanding outdoor environments using robots able to relatively localize neighboring robots. The proposed method enables decentralized navigation agreement and decision-making using only implicit communication by observing blinking markers. Another important contribution of this work is the achievement of a task-driven decentralized heading alignment securing the distributed sensing of workspace. To support our statement of bridging the reality gap, the reliability and robustness of the overall system were verified through repeated real-world testing of the proposed flocking system in the demanding conditions of desert, dunes, and hill environments.

1.3 Problem statement

The problem tackled in this paper regards the multi-robot exploration of demanding environments for SAR purposes. We utilize a swarm of UAVs to maintain the coverage of a large given area and take advantage of its distributed sensing capabilities in order to increase the reliability of OOI detection. The task of the swarm is to detect and locate a set of OOIs with unknown positions and to reject any false positive detections obtained by the onboard sensors. It is assumed that the proposed flocking system relies exclusively upon onboard sensory equipment with the use of minimal inter-agent communication. Agents can exchange a piece of information with neighboring agents for navigation agreement purposes. We further assume that the environment is unknown, but obstacle-free at the chosen flight level. In order to tackle this problem, the requirements for the UAV platform include a relative mutual agent localization system, a rangefinder for height estimation, an onboard computer, and a forward-facing RGB camera.The waypoints for sweeping trajectories are manually set by a human operator. We define the sweeping trajectory as the reference trajectory for the swarm when any OOI is not being approached by the swarm.

The reason behind the use of a relative localization system in GPS-available conditions is straightforward: shared GPS may not provide sufficient results with standard GPS precision and drift when used with compact swarms. Guaranteed GPS accuracy may vary in the real world concerning a given environment, additional onboard equipment (due to interference), or weather. Further, relative localization can be designed as a backup in the case of GPS signal or communication performance loss, due to signal reflection from nearby buildings or jamming.

2 Preliminaries

2.1 UVDAR-COM

In our swarming solution, each UAV agent is equipped with the UVDAR (UltraViolet Direction And Ranging) system developed by Walter et al. (2019), Walter et al. (2018b) for estimation of the relative position of surrounding swarm agents. The system consists of UV LED markers and UV-sensitive cameras carried onboard each UAV. Computer vision is used to localize the UV LED markers attached to neighboring agents that are observed by onboard UV cameras. A linear Kalman filter is applied to improve the position tracking of neighboring UAVs.

In this paper, we used an extended version of the UVDAR system utilized as an alternate communication system to radio communication. The extension, UVDAR-COM system (Horyna et al., 2022), is inspired by free-space optical (FSO) communication (Khalighi & Uysal, 2014; Kaushal & Kaddoum, 2016), which has been used in recent years for data transfer between terrestrial and space systems. UVDAR-COM transmits the message frame through blinking UV markers and using UV-sensitive cameras as optical signal receivers. The used visual communication channel creates a mesh network over the UAVs (nodes), where each node can directly communicate with other nodes with visual contact. Additionally, the positions of the UV markers in the camera image frame are processed in the same way as is done by the default UVDAR system. Thus, the extended system provides relative localization, as well as low-bandwidth local communication. The spatial separation of visual signals and the Hough Transform used for tracking of UV markers makes the UVDAR-COM system resistant to UV reflections.

2.2 Object detection and localization

The aim of the proposed swarm approach is to explore a given environment and to search for multiple OOI, such as lost people, wrecked cars, or airplane debris. To provide realistic properties of onboard real-time sensing, the proposed system was designed and tested with a Convolutional Neural Network (CNN) “person detector” built upon the OpenVINO toolkit (Intel Corporation, 2018).

Fig. 2

Onboard RGB camera object detection using a CNN

The output of the person detector (see Fig. 2) is the relative position and bounding box of the object in the camera frame, as well as a confidence score. The confidence score is necessary to determine whether the detection is a true positive or a false positive. The distance of the detection from the UAV is estimated from the size of the detected OOI in the image. The detections are filtered by a Kalman filter and the position of the OOI is estimated, even when the OOI is outside the field of view.

2.3 Swarm hierarchy

The novel flocking behavior (see Sect. 3) proposed in this paper is based on the observation of the surrounding workspace by a particular agent. The agent neighborhood (or just neighborhood) is a subset of the swarm containing agents around a particular agent. Neighborhood states are used to achieve decentralized flocking motion and decision making. Neighborhood selection is introduced in Sect. 3.7.

We suppose that every swarm agent is deployed with the same sensory equipment. Nevertheless, the significance of individual agents can vary in time. A swarm member having more relevant information than a regular agent is called the informed agent. We deal with two types of informed agents:

• An informed agent that knows the swarm trajectory waypoints in the deployed area. Agents of this type represent around $10\,\%$ of the total number of swarm members. They are decentralized and dynamically chosen based on an ID given by the UVDAR-COM system (e.g, selected by the lowest ID in the agent neighborhood).

• An informed agent that has valid OOI detection. Agents of this type arise based on observation of the surroundings.

2.4 Frames of reference

In this work, variables and computations are expressed in the local frame of a UAV. The local frame is fixed to the environment and aligned to the initial position of the UAV. The local frame is oriented such that its x-axis is aligned with the east direction, and the z-axis is pointing upward. In our experiments, the fixed orientation of local frames is ensured by using magnetometers for heading estimation.

3 Flocking control

Following the basic principles of decentralized swarming behavior, the control law for each swarming member is expressed as:

$$\begin{aligned} {{\dot{\textbf{q}}}_d} = \alpha {\textbf{p}} + \beta {\textbf{n}} + \gamma {\textbf{a}}, \end{aligned}$$

(1)

where ${{\dot{\textbf{q}}}_d}\in {\mathbb {R}}^3$ is the control vector containing the desired velocities in the horizontal plane and the desired heading rate. Vectors ${\textbf{p}}, {\textbf{n}}, {\textbf{a}} \in {\mathbb {R}}^3$ represent the proximal, navigation, and heading alignment vectors, respectively. Scalars $\alpha , \beta , \gamma \in {\mathbb {R}}_{>0}$ are weights of the corresponding vectors.

The proposed flocking control law is shown in Fig. 3 within the overall flocking system architecture of a single agent. The data from onboard sensors are processed in the Data processing part of the system. Processed data are inputs of both control layers (the Low-level control and the high-level Flocking control). The Navigation control is dependent on the OOI detections and the proposed self-adaptive communication system.

Fig. 3

Architecture of the proposed flocking system of a single agent

3.1 Proximal vector

The proximal control vector keeps the focal agent (an arbitrary agent from the swarm) at the desired distance from surrounding agents. It includes cohesion control, which keeps the swarm together, and the separation control, which prevents collisions between agents. We assume that each robot is able to measure the relative position of its neighbors by the UVDAR sensor. The proximity vector (inspired by potential functions from (Merheb et al., 2016; Gazi & Passino, 2002) is thus proposed as:

$$\begin{aligned} {\textbf{p}} =&\sum _{i=1}^{N} \mathbf {u_i}\left[ 1-\frac{d_{des}}{d_i}\right] , \end{aligned}$$

(2)

where $\mathbf {u_i}$ is the unit vector pointing from the focal agent to the i th neighbor, $d_{i}$ is the relative range of the i th neighbor, $d_{des}$ is the desired distance between the agents, and N is the number of agents in the given neighborhood (see Sect. 3.7). The unit vector

$$\begin{aligned} \mathbf {u_i}&= \begin{bmatrix} \frac{\mathbf {p_i}}{\left\Vert \mathbf {p_i} \right\Vert } \\ 0 \\ \end{bmatrix} \end{aligned}$$

(3)

depends on the relative position $\mathbf {p_i}\in {\mathbb {R}}^2$ of the i th neighbor from the focal agent in the horizontal plane.

3.2 Navigation vector

The navigation control vector ${\textbf{n}}$ determines the direction of motion of the focal agent towards a desired goal position. The goal direction may not be known by every agent. Furthermore, the goal direction can be changed in real-time should there be a change of swarm trajectory or visual detection of an OOI. It is desirable to make a decentralized agreement between all robots for the goal direction, regarding the current state of agents. Such an agreement, together with an appropriate inter-agent communication strategy, secures the dynamic response of the swarm behavior without risking the loss of cohesion or becoming stuck in a local minimum.

This extends the self-adaptive communication strategy (SCS) proposed in Ferrante et al. (2014) to enable navigation agreement among robots. The proposed navigation approach assumes that each agent can communicate some direction information to its neighbors (see Sect. 3.3). The received directional information of the UAV’s neighbors is averaged together with transmitted directional information:

$$\begin{aligned} \overline{\eta } = {{\,\textrm{atan2}\,}}\left[ \frac{ \eta _0 + \sum _{i=1}^{N} \sin {\eta _i}}{N+1}, \frac{ \eta _0 + \sum _{i=1}^{N} \cos {\eta _i} }{N+1} \right] , \end{aligned}$$

(4)

where $\eta _0$ is transmitted directional information, $\eta _i$ is directional information received from the i th neighbor, and N is the number of agents in the neighborhood of the focal agent. Directional information are computed onboard UAVs according to Eq. (6). The navigation vector is obtained as:

$$\begin{aligned} {\textbf{n}}&= \begin{bmatrix} \cos {\overline{\eta }} \\ \sin {\overline{\eta }} \\ 0 \\ \end{bmatrix}. \end{aligned}$$

(5)

3.3 Self-adaptive communication system

The novel self-adaptive communication system (SCS) is designed in the proposed flocking system (Fig. 3) for the dynamic and decentralized adjustment of the navigation vector. Such an approach allows for the change of flight direction without inter-agents oscillations or loss of cohesion. The communication system defines the transmitted information based on a comparison of the current UAV state (an internal information) and the averaged information received from neighboring agents. The communication system achieves an agreement on the common group goal direction with the highest priority (e.g., direction to OOI). The overall communicated information is composed of two types:

• Directional type of information (a directional angle) necessary for navigation agreement among robots, according to the Eq. (4).

• Visual type of information from the range $\left[ 0, 1\right]$. Determines if the UAVs have an OOI detection at the given time. The higher the value is, the higher the detection probability.

The directional and visual information transmitted by the focal agent ($\eta _0$ and $\nu _0$, respectively) are defined as:

$$\begin{aligned} \eta _0 =&w_{scs}\left( k\right) \eta _g + \left( 1-w_{scs}\left( k\right) \right) \overline{\eta }, \end{aligned}$$

(6)

$$\begin{aligned} \nu _0 =&{\left\{ \begin{array}{ll} 1 &{} \quad \text {if } \nu _{w} > \tau _{\nu }, \\ 0 &{} \quad \text {if } \nu _{w} \le \tau _{\nu }, \end{array}\right. } \end{aligned}$$

(7)

where $w_{scs}\in \left[ 0, 1 \right]$ is the weighting coefficient (see Eq. 10), and $\eta _g \in \left[ 0 , 2\pi \right)$ is the internal directional information. In other words, $\eta _g$ is the goal flight direction of the focal robot (e.g., azimuth towards waypoint or OOI detection). Further, $\overline{\eta } \in \left[ 0 , 2\pi \right)$ is averaged directional information from Eq. (4), and $\tau _{\nu } \in \left[ 0 , 1 \right]$ is the threshold of transmitted visual information, which maps values from the range of $\left[ 0, 1\right]$ to a single bit value. It determines the ratio between neighboring agents with and without visual contact on an OOI. The threshold $\tau _{\nu }$ determines how fast the information about OOI detection spreads through the swarm by uninformed agents. Finally, $\nu _w \in \left[ 0 , 1 \right]$ is the weighted visual information:

$$\begin{aligned} \nu _{w}\;=\;&w_{scs}\left( k\right) I_{Iv} + \left( 1-w_{scs}\left( k\right) \right) I_{Rv}, \end{aligned}$$

(8)

where $I_{Iv} \in \{0, 1\}$ is internal visual information (1—valid detection of OOI, 0—no onboard detection of OOI at the given time) and $I_{Rv} \in \left[ 0 , 1 \right]$ is the averaged received visual information:

$$\begin{aligned} I_{Rv} =&\frac{1}{N} \sum _{i=1}^{N}\nu _i, \end{aligned}$$

(9)

where $\nu _i$ is the visual information received from the i th neighbor, and N is the number of agents in the neighborhood of the focal agent.

As mentioned above, the interpolation of the internal information $\eta _g, I_{Iv}$ (estimated by onboard sensors of the focal UAV), with averaged information received from neighboring agents $\overline{\eta }, I_{Rv}$, is achieved by the weighting coefficient $w_{scs} \in \left[ 0 , 1 \right]$. In other words, $w_{scs}$ in Eqs. (6) and (8) determines how much priority the focal UAV gives to its own intention over the intention of the group. The weighting coefficient provides dynamic and decentralized change of influence of individual swarm members on the goal flight direction of the swarm. The parameter $w_{scs}$ is defined as:

$$\begin{aligned} w_{scs}\left( k\right) =&{\left\{ \begin{array}{ll} w_{scs}\left( k-1\right) + \Delta w_{scs} &{} \quad \text {if } \xi > \xi _0, \\ w_{scs}\left( k-1\right) - \Delta w_{scs} &{} \quad \text {if } \xi \le \xi _0, \end{array}\right. } \end{aligned}$$

(10)

where $\Delta w_{scs}$ is the weight increment between two time-steps, $\xi \in \left[ 0, 1\right]$ is the local information consensus, and $\xi _0$ is its threshold. We propose the local information consensus as a quantity representing the difference between received and transmitted information frames. $\xi$ is equal to 1, only if the transmitted piece of information is the same received piece of information. We define the local consensus as:

$$\begin{aligned} \xi =&\frac{w_v d_{Hv}\left( I_{Rv}, I_{Iv}\right) + w_d d_{Hd}\left( I_{Rd}, I_{Id}\right) }{w_v + w_d m}, \end{aligned}$$

(11)

where $d_{Hv}\left( I_{Rv}, I_{Iv}\right)$ and $d_{Hd}\left( I_{Rd}, I_{Id}\right)$ are Hamming distances between averaged received and internal information. The subscripts v, d indicate visual or directional types of information. The weights $w_v, w_d$ determine the importance of the individual type of information. Constant m represents the number of bits in the directional information after conversion from Eq. (6) to the binary form. To achieve swarm convergence in the higher priority direction, we introduce two special cases of information available for informed agents:

• Informed agent that knows the swarm waypoint in the deployed area and determines the internal directional information $I_{Id}$ as the angle from the common swarm heading to the following waypoint.

• Informed agent that obtains a valid object detection and determines the internal visual information $I_{Iv}$ regarding the true positivity of the current detection.

The visual information aspect of the local consensus in Eq. (11) provides the sensitivity of the system on insignificant desired directional change. Conversely, the directional information aspect of the local consensus results in a response to the change of swarm direction in the case of change of the system waypoint. Moreover, this ensures smooth navigation transition without significant directional differences between individual robots.

3.4 Heading alignment vector

We propose a novel heading alignment strategy, which assures planar omnidirectional sensing in the workspace. This strategy differs from the commonly used strategy of heading agreement among robots (De Benedetti et al., 2017). The proposed self-organizing approach supports data collection and processing. Omnidirectional sensing of the workspace is required for deployment of directional sensors or sensors with a limited field of view, such as cameras. Thus, we take full advantage of aerial multirotor robots in comparison to car-like vehicles (Ferrante et al., 2012), where the heading alignment of robots influences flocking cohesiveness.

To achieve the proposed unique heading alignment behavior, each UAV needs to evaluate itself as a peripheral or inner swarm member by observing its neighboring agents. To estimate the position of a UAV within the swarm, we define the parameter $\delta$ to determine the largest angle among two neighbors as:

$$\begin{aligned} \delta = \underset{A\left( i\right) , A\left( i+1\right) \in A}{\arg \max }\ \{ f (A\left( i\right) , A\left( i+1\right) ) \}, \end{aligned}$$

(12)

where bearings of neighboring agents $\alpha _{i}$ and $\alpha _{i+1}$ are elements of the set

$$\begin{aligned} A = \{ \alpha _{1}, \alpha _{2}, \alpha _{3},\ldots , \alpha _{n}, \alpha _{1} \}; \alpha _{j} < \alpha _{j+1}. \end{aligned}$$

(13)

The function $f (\alpha _{i}, \alpha _{i+1})$ provides the angle between the two given bearings. We define the desired heading $\psi _d$ of an agent as:

$$\begin{aligned} \psi _d = {\left\{ \begin{array}{ll} \overline{\alpha } + \pi &{} \quad \text {if } \delta > \epsilon , \\ \overline{\eta } &{} \quad \text {else}, \end{array}\right. } \end{aligned}$$

(14)

where $\overline{\alpha }$ is the mean bearing of neighbors of the focal UAV. The desired heading is set to the opposite of the angle $\overline{\alpha }$ if the parameter $\delta$ exceeds the given threshold $\epsilon$, otherwise the desired heading is set to the averaged directional information defined in the Eq. (5).

According to the approach described above, we define the heading alignment vector as:

$$\begin{aligned} {\textbf{a}} =&\begin{bmatrix} 0 \\ 0 \\ 1 \\ \end{bmatrix} \left( \psi _d - \psi \right) , \end{aligned}$$

(15)

where $\psi$ is the heading of the UAV and $\psi _d$ is the desired heading. With respect to the Eq. (1), the alignment vector has the meaning of heading control error with $\gamma$ as the proportional feedback gain.

3.5 UAV separation in OOI proximity

As discussed above, one such required strategy for SAR missions with possible multiple OOI is to separate one of the swarm members from the other UAVs in the event of an OOI detection confirmation. This UAV can serve as a static beacon, inspect the area of detection, or deliver first aid supplies. Separation of a UAV for a new task requires decentralized distribution of team tasks. To fulfill demands on the required decentralized and dynamical decision making that is currently unavailable in state-of-the-art methods designed for UAV swarming in real conditions, we propose the control law as an extension of the Eq. (1) in the form:

$$\begin{aligned} {{\dot{\textbf{q}}}}_d =&\alpha \mathbf {p'} + \beta \left( \rho _0{\textbf{n}} + \left( 1-\rho _0\right) \mathbf {n_g}\right) + \gamma \rho _0{\textbf{a}}, \end{aligned}$$

(16)

where $\mathbf {n_g}$ is the navigation vector pointing towards the detected object and $\mathbf {p'}$ is the enhanced proximal control

$$\begin{aligned} \mathbf {p'} =&\rho _0\sum _{i=1}^{N} \rho _i\mathbf {u_i}\left[ 1-\frac{d_{des}}{d_i}\right] . \end{aligned}$$

(17)

Coefficients $\rho _0, \rho _i$ are inverse distance weighting coefficients that are computed based on the known relative position of individual neighbor agents, and the relative position of the detected object. The coefficient $\rho _0$ affects the influence of the swarm on the particular agent, whereas the coefficient $\rho _i$ affects the influence of the agent on the rest of the swarm. The designed control strategy ensures a smooth transition from swarm control into a single UAV control around the detected OOI area.

3.6 False positive detection handling

To achieve dynamic response on false positive detections, the focal UAV, when having visual contact on a possible OOI, computes the visual consensus $\nu _c \in \left[ 0 ,1 \right]$ of neighbors:

$$\begin{aligned} \nu _{c} =&\frac{1}{N_c}\sum _{i=1}^{N_c} \nu _i, \end{aligned}$$

(18)

where $\nu _i$ is the visual information received from i th UAV that is closer to the OOI than the focal UAV, and $N_c$ is the number of UAV fulfilling this condition. The OOI is removed from the list of OOI candidates of the focal UAV if the visual consensus is below the threshold. The threshold expresses the ratio between neighbors who evaluated the OOI as a true positive and neighbors who evaluated the OOI as a false positive.

3.7 Selection of neighbors considered in swarming rules

The above-mentioned control vectors are computed based on states of neighboring agents given by observing the surrounding workspace. The Lessons learned section (Sect. 5) discusses motivation beyond the novel design of the neighbor selection, as we had found the commonly used selection methods (radius-based (Krishnanand & Ghose, 2009)—Fig. 4a, K-nearest method (Akat & Gazi, 2008)—Fig. 4b) to be inappropriate for deployment in demanding outdoor environments. An inappropriate design of neighbor selection can lead to inter-agents oscillations, destabilization of the swarm, or loss of cohesion.

Fig. 4

Commonly used neighbors selection methods with the proposed strategy. With respect to the blue UAV, green UAVs represent neighbors while red UAVs represent outlying robots. Yellow agent represents a neighbor candidate (Color figure online)

We propose an iterative neighbors selection method based on the bearings of neighbors selection (Fig. 4c). The aim of the method is to get a set of agents close to the focal agent, while having the difference between two arbitrary neighbors’ bearings not exceed the given angle limit. In other words, we define the desired neighbors’ bearing density.

The iterative neighbors selection method is proposed as follows. First, the closest agent is assigned as a neighbor. In every iteration step, a circular sector is determined around the agents that are already assigned as neighbors (Fig. 4d). Another robot from the list of visible agents is sorted by distance, then considered a neighbor if it does not lie in the circular sector of any neighbor. Thus, UAVs are assigned as neighbors if

$$\begin{aligned} \beta _{n, h}> \beta _{s} > \beta _{n, l}, \end{aligned}$$

(19)

where $\beta _{n, h}=\beta _{n}+\beta _i$ and $\beta _{n, l}=\beta _{n}-\beta _i$ are higher and lower angle limits of the circular sector of n-th neighbor, respectively, and $\beta _{s}$ is the bearing of the neighbor candidate. Features of the selection method are given by the size of the internal angle $\beta _i$ of the circular sector.

The proposed approach of the neighbors selection secures equal coverage of the neighbors around a UAV. Furthermore, it provides cohesive behavior between distant swarm members. Such behavior supports the robustness of the swarm system, since it is resistant to short-term outages of the relative localization data (e.g., due to an occlusion by an obstacle). Moreover, the proposed method rejects agents nearby if they are behind a closer neighbor. Such neighbors are sources of imbalances in attractive and repulsive forces.

4 Results

Experiments verifying the proposed flocking algorithms were performed in both simulations and the real world. The hardware platform used during the experimental campaigns is shown in Fig. 5 and described with basic MRS control system in Baca et al. (2021). A Quadcopter 450 frame with rotors, Pixhawk attitude rate controller, and Intel NUC onboard computer are the basis of the aerial vehicle. The sensory equipment consists of:

• Downward-facing rangefinder for estimating height above the ground.

• GPS receiver for self-localization.

• Forward-looking RGB camera for detection of an object of interest.

• UV LEDs and UV cameras (parts of the UVDAR (Walter et al., 2018a) localization system) for localization of neighboring agents and communication.

The MpcTracker from Baca et al. (2018, 2021) is used as the reference tracker in the proposed swarm system to control the motion of individual agents, according to the Eq. (1). A flocking controller was designed above the speed tracker to achieve the required flocking behavior.

Fig. 5

Sensory equipment of the aerial vehicles used for experimental verification

Fig. 6

Demonstration of the proposed self-alignment mechanism (Color figure online)

The goal of the presented experiments was to explore a given area, detect objects of interest, confirm the detections, and, if positive, allow one UAV to be separated from the swarm at the location of each object. As mentioned in the introduction, this UAV may be used for closer inspection, to act as a static beacon, to detail the space around found objects for human first responders, or to deliver first aid supplies.

Verification of the proposed methods consists of three main domains. First, the flocking control system analysis showed swarm behavior in large-scale applications composed of tens of robots. Second, navigation control, self-adaptive communication strategy (SCS), and extension of the UVDAR system are verified in the realistic Gazebo simulator. Finally, the verification and analysis of the proposed system deployed on the hardware is verified in various environments, complemented by the motivation of an experiment with onboard person detection in the control feedback.

4.1 Large-scale swarm simulations

The first set of experiments were performed using a ROS node simulating 100 robots. A simulator neglecting the dynamic model of UAV and interaction with other agents was designed in C++ with visualization in RViz¹ to achieve reasonable computational expenses. The purpose of this experiments is to verify the scalability of the swarm system in proximal and alignment control, as well as neighbors and leaders selection. The desired distance between robots was set to 7 m.

The histograms in Fig. 7 compare the influence of methods of neighbor selection (introduced in Sect. 3.7) on robots behavior in its surroundings. The first two rows represent an experiment with radius neighbor selection. The radius threshold r (Fig. 4a) was set to $125\,\%$ and $200\,\%$ of the distance to the closest neighbor. In the lower threshold, up to $50\,\%$ of robots were evaluated as the leader. However, the low value of the selection radius brings a high number of outlying neighbors, especially in the case of unequally distributed UAVs during initialization. This leads to slow transmission of navigation information, local instability, and formation split. Setting a high value of radius increases the number of detected neighbors, leading to better transmission of the navigation message. Moreover, the number of informed agents decreases to $15\,\%$, thereby reducing information redundancy. An excessive number of neighbors affects the proximal control, such that attractive forces outbalance repulsive forces. This leads to the creation of smaller swarm groups and forces UAVs towards others in the group (Fig. 7— the lower bound of distance to neighbors is low). Tuning the outlier radius between the tested values would lead to the desired performance. However, such tuning is challenging, especially when the uncertainty of relative localization is taken into account. This phenomena will be discussed in Sect. 5. The K-nearest neighbor selection (principle in Fig. 4b) shows a similar pattern in the comparison of distances to neighbors. However, attractive and repulsive forces are outbalanced, especially around the edges of the swarm.

Fig. 7

Histograms representing the number of detected neighbors (left column) and minimal, average, and maximal distance to neighbors (right column). Methods from Sect. 3.7 are compared in individual rows

Histograms of results obtained by the proposed method of neighbor selection based on bearing density of neighbors (Fig. 4d) are introduced in the last row of Fig. 7. The number of detected neighbors varies based on the position of the UAV in the swarm, while also keeping the minimal distance to neighbors sufficiently high. Moreover, the proposed strategy detects neighbors at a far proximity without creating local subgroups and outbalancing attractive and repulsive forces. Informed agents represented $15\,\%$ within the high radius neighbors selection.

The simulation results presented in this section showed bottlenecks for some commonly used neighbor selection methods, which are highly dependent on initial conditions and tuning parameters (e.g., thresholds, number of neighbors). It was verified that the proposed method of bearing selection improves these properties. Specifically, the proposed method allows for detecting a high number of neighbors regardless of distance from the UAVs, while also keeping them close to the desired distance using the proximal control. This helps to keep cohesion between swarm members if asymmetrical distance estimations have occurred. Moreover, the focal agent can not detect two UAVs with similar bearing, which helps to balance cohesion and separation forces.

4.1.1 Self-alignment demonstration

Alignment control and novel planar omnidirectional sensing were tested with 20 swarm members in the same simulation setup as the previous experiment. The initial position and orientation of agents were set randomly, as shown in Fig. 6a. The goal position of the experiment was set to align the navigation vector (5) along the x-axis. Figure 6b represents the steady-state of swarm heading with the fundamental navigation vector alignment. Planar omnidirectional coverage of the workspace introduced in Sect. 3.4 is shown in Fig. 6c. The blue vectors represent robots, which were evaluated as peripheral agents according to the rule (12). Peripheral UAVs are aligned with a vector pointing to the opposite direction of the mean of bearing vectors. The rest of the swarm (red vectors) are aligned with the navigation vector, as is seen in Fig. 6b.

4.2 Gazebo simulations

For the simulations presented in this section, seven UAVs were stabilized and navigated through four given waypoints using a UVDAR-COM system, a rangefinder for estimation of height above ground, and a forward-facing RGB camera for object detection (see Fig. 8a for one of the simulations results). The simulations were performed in ROS and the realistic Gazebo simulator. Only one UAV is informed (red path) and knows the positions of the defined waypoints.

Fig. 8

Simulation of the designed system with seven UAVs in the realistic Gazebo simulator (Color figure online)

All UAVs are capable of detecting an object of interest by an onboard RGB camera. Three artificial objects were added to the simulation. Two of them represent an object of interest (green circle) while the third object (red circle) represents an object to be detected by the camera as a false positive. Figure 8a (where one of the simulation trial is depicted) shows the behavior of the swarm in the xy-plane. Part of the graph between timestamps $t_1$ and $t_2$ shows the swarm approach to the first detected objects based on the utilized SCS method. After reaching the detection verification range, the UAV closest to the detected object is separated and grounded to act as a static beacon. The second object is evaluated to be a false positive detection. In this case, the swarm returns to the sweeping trajectory as shown in the proximity of the red spot in Fig. 8a.

The internal states of swarm members in the time between timestamps $t_1$ and $t_2$ are shown in Fig. 8b. The upper graph represents the evolution of the local consensus variable over a given time. The bottom graph shows the weight for the computation of transmitted information (SCS weight). The influence of individual agents on swarm navigation can be observed in both graphs. Primarily, the loss of influence of an informed agent becomes visible when a collision between transmitted and received information occurs after the first timestamp. Alternatively, $w_{scs}$ rises when the local consensus of a particular UAV exceeds its threshold. The analogous behavior is represented in Fig. 8b around timestamp $t_2$, where the swarm reaches the object of interest and one UAV is deployed.

4.3 Real-world experiments

Multiple varying experiments in different environments were conducted to verify the robustness and usability of the proposed approach. Videos of real robot experiments are available at http://mrs.felk.cvut.cz/auro-2022-sar. In this section, we highlight two sets of experiments in desert dunes and grass hills to demonstrate the performance of the approach over uneven terrain. During the first tests (for a record of one of the runs, see Fig. 9a), we used shared GPS positions as the localization technique.

A swarm of 14 UAVs was sent to fly in the direction of the positive x-axis of a local world frame that was given by predefined waypoints. A group of aerial vehicles fulfilled the goal without any collision, while also keeping in formation. The desired distance among robots was $7\,m$ and maximum velocity was set to $1\,m.s^{-1}$. The bottom of Fig. 9a shows the histogram analyses analogical to the results presented in Fig. 7. In the case of a real-world experiment, the count of detected neighbors in the neighborhood is generally lower. This is caused by the higher ratio of peripheral swarm members in comparison to simulated experiments in Sect. 4.1.

In the second set of outdoor experiments (Fig. 10a), we used UVDAR as the localization technique and artificial objects to focus on verification of the proposed navigation and decision-making approach in real conditions. The swarm system consisted of nine UAVs with one informed agent equipped with a known reference path. The swarm system behaved as was expected from the simulations. Information about OOI detections by peripheral UAVs was propagated by the self-adaptive communication system to the rest of the swarm, leading to an approach to the OOI detection. After the OOI detection was verified, the swarm member closest to the OOI was deployed to the detection and landed nearby (dark green trajectory in Fig. 10a). The choice of which deployed UAV to send was performed in a decentralized manner by using the flocking control rule described by the Eq. (16). The rest of the swarm continued its sweeping trajectory towards the location where a false positive detection was obtained. As planned, the swarm left the trajectory again, but once the detection was verified as false, the swarm returned to the sweeping trajectory. The same scenario as with the first approach to the OOI detection was observed close to the third artificial OOI. As in the first set of experiments, the swarm explored the area and detected OOIs as expected, without any collision or loss of cohesiveness. The extreme lighting conditions of the desert influenced the accuracy of the UVDAR system. Nevertheless, the overall performance of the flocking system was not significantly influenced. This was thanks to the resistance to asymmetric UVDAR detections of the used neighbors’ selection method, as well as to the distributed communication strategy.

Fig. 9

Results of the real-world flocking with 14 UAVs. a UAVs’ trajectories and histograms showing formation properties, as in Fig. 7

Fig. 10

Real experiment with nine UAVs relying on the UVDAR system. Artificial objects of interest were used (Color figure online)

The order metric and the accuracy metric are used as metrics for evaluating navigation performance, which we proposed in Ferrante et al. (2014). We define order as:

$$\begin{aligned} \Omega = \frac{1}{M}\left\Vert \sum _{i=1}^{M} {\textbf{n}}_i \right\Vert = \frac{1}{M}\left\Vert {\textbf{b}} \right\Vert , \end{aligned}$$

(20)

where M is the total number of UAVs and ${\textbf{n}}_i$ is the navigation vector of i-th agent. The order is equal to 1 when the UAVs have a common direction of the navigation vector. Otherwise, the order is less than 1. The accuracy is defined as:

$$\begin{aligned} \theta = 1 - \frac{1 - \Omega \cos \left( \angle {\textbf{b}} - \angle {\textbf{g}} \right) }{2}, \end{aligned}$$

(21)

where $\angle {\textbf{b}}$ is the direction of ${\textbf{b}}$ and $\angle {\textbf{g}}$ is the current goal direction (to a waypoint or OOI detection). The accuracy indicates how close to the goal direction UAVs are moving. The accuracy is equal 1 when the order is equal to 1, and UAVs move along the goal direction. The results of the presented metric analysis are shown in Fig. 11a. The sharp drops in the accuracy correspond to the change of direction of the navigation vector. These changes have three sources: approach to an object of interest (red arrows), recovery back to the sweeping trajectory (blue arrows), and change of waypoint (green arrow). The order and accuracy analysis shows convergent behavior of the flocking navigation system. An accuracy drop below 0.5 indicates directional change higher than $90^{\circ }$.

Fig. 11

Real experiment with nine UAVs relying on the UVDAR-COM system and artificial objects of interest (Color figure online)

The last set of tests was successfully performed with the online CNN person detector (Fig. 2) in the control feedback. A swarm of four UAVs was supposed to detect two people near a given path (Fig. 12a). The deployed swarm members were not allowed to directly approach the position of the detected people, as had been the case in the tests with artificial objects, but a safety margin was introduced to keep the UAVs at a safe distance. The results show the motivational deployment of the proposed SAR system in real-world conditions.

Fig. 12

Real experiment with 4 UAVs. CNN person detector was used to detect human victims (Color figure online)

5 Lessons learned

Herein, we introduce the lessons learned during our experimental campaigns in the demanding outdoor environments which have negatively influenced the deployment of swarm systems in reality. Such study, observed failure cases, and proposed solutions of these cases may benefit the robotic community in designing swarm systems resistant to demanding conditions. We find the following to have been the most crucial.

5.1 Relative localization & communication

We rely on the onboard UVDAR-COM (Sect. 2.1) system that was determined to be necessary for large scale deployment, as GPS tends to be unsuitable with higher counts of agents in a compact group. Multiple experiments behind the scenes for the swarm system deployed with GNSS localization and an explicit radio communication proved unreliable. This problem is partially solvable by properly choosing the radio communication network. Then, the proposed solution can be used as a backup system in the case of GNSS signal or explicit communication performance loss. This may be an issue due to signal reflection from nearby buildings, or due to jamming.

Failure cases:

• Collisions of UAVs due to precision and drift of GPS or unreliable explicit communication.

• Loss of cohesion of UAVs.

5.2 HW redundancy

Demanding environments (e.g., desert) may present conditions that reach the limits of the sensory equipment of the UAVs. Thus, we find sensory redundancy to be a vital demand (e.g., rangefinder-barometer redundancy for height estimation that was used).

Failure cases:

• The UAV gets to the failsafe mode due to exceeded temperature limit of the rangefinder.

• Height oscillations above the edges of the dunes.

5.3 SW redundancy

We propose also using software redundancy for decentralized decision-making in order to mitigate the uncertainty of sensory data. E.g., the UAV separation strategy (see Sect. 3.5) was enhanced by the neighbors’ state monitoring for filtration of UAVs incorrectly evaluated for separation.

Failure cases:

• Less or more than the preferred number of UAV separated from the group near an OOI, due to uncertainty of distance estimation towards the OOI or neighbors.

5.4 Failure detection and handling

We used a failure detection and handling system as part of the Low-level Control from Fig. 3 in order to perform safe experiments without damaging the UAVs or their surroundings. The UAV is switched to the failsafe mode and lands immediately once a failure not corresponding with safe flight is detected.

Failure cases:

• UAV exceeding the threshold of the maximal safe thrust or tilt (e.g., due to a strong wind).

5.5 Neighbor selection

We recommend emphasising a robust method of neighbor selection (Sect. 3.7) in outdoor conditions where neighbor detection based on camera observation is influenced by weather and lighting conditions. Such influences are a source of asymmetry in detections. Thus, we designed and proposed a unique neighbors selection method robust against adverse effects that are not present in laboratory conditions and experimental results available in the literature.

Failure cases:

• Loss of cohesion when neighboring UAVs are outside the detection radius of the radius-based selection method.

6 Conclusion

In this paper, a novel distributed autonomous flocking behavior for UAVs was introduced and motivated by search and rescue applications. A fully decentralized approach was intended, including swarm navigation with the minimum amount of information exchanged among robots possible with implicit communication integrated into an enhanced direct localization system. Moreover, we proposed a unique task-driven decentralized heading alignment rule for distributed sensing of a surrounding workspace. Thus, complex swarming behavior triggered by online inputs from onboard RGB cameras in SAR applications was achieved without the need to share GNSS positions or use explicit wireless communication, both of which appeared to bottleneck large compact swarming systems in real conditions. Successful real-world experiments with our UAVs using the onboard UVDAR-COM system demonstrated reliable behavior of the swarming system and flight in a compact shape with small mutual distances.