1 Introduction

A real disaster response takes longer than a single sortie into the area. As witnessed recently for example in Japan (Fukushima) and in Northern Italy (Emilia Romagna) deployments can last days, weeks, months, if not years.

TRADR builds on the research and experience of the NIFTi project [21]. In July 2012 NIFTi assisted in structure damage assessment in Emilia Romagna, after it was hit by over 250 seismic events in May–June 2012, causing widespread damage to an area rich in cultural heritage (Fig. 1). Together with the Vigili del Fuoco, the Italian national rescue organisation responsible for disaster response, NIFTi fielded a human-robot team with a mobile command post, two unmanned ground vehicles (UGVs), and two quadcopter unmanned aerial vehicles (UAVs). The crucial insight from this deployment was the need for integrated persistent situation awareness [22]. Multiple robots need to be sent into the area, together (synchronous operation) or one after another (asynchronous operations). Different kinds of robots play complementary roles in this process. They need to build integrated persistent situation awareness gradually over multiple sorties, to allow the team to coordinate its efforts (team-level), and learn to best execute its tasks (task-level).

Fig. 1
figure 1

NIFTi deployment in Emilia Romagna. Top: Structural damage on Duomo in Mirandola. Bottom (left-to-right): UGV, UAV and mobile command post

Full size image

TRADR addresses the ensuing challenge of making the experience of a human-robot disaster response team persistent over multiple sorties during a prolonged mission. We employ proven-in-practice user-centric design methodology (Fig. 2, left), involving tight cooperation with end users and tight integration of technology. The TRADR use cases involve response to a medium to large scale industrial accident by teams consisting of human rescuers and several ground and airborne robots (Fig. 2, right). The team collaborates to explore the environment and gather measurements and physical samples. TRADR’s goal is to enable the team to gradually develop its understanding of the disaster area over multiple synchronous and asynchronous sorties (persistent environment models), to improve team members’ understanding of how to work in the area (persistent single- and multi-robot action models), and to improve team-work (persistent human-robot teaming). TRADR missions will ultimately stretch over several days in increasingly dynamic environments.

Fig. 2
figure 2

Left: TRADR one-year-round development cycle. Right: TRADR UGV and UAV

Full size image

Project Partners The TRADR consortium consists of 12 partners,Footnote 1 including 3 research institutes: DFKI (coordinator), Fraunhofer, TNO; 5 universities: ETH, KTH, CTU, ROMA and TUD; one industry partner: Ascending Technologies; and 3 end-user organizations, representatives of the fire-brigades from Germany (Stadt Dortmund Institut für Feuerwehr und Rettungstechnologie), Italy (Vigili del Fuoco directed by the Ministero Dell’interno) and the Netherlands (Gezamenlijke Brandweer). 8 of the partners have already collaborated very successfully in the NIFTi project.

2 The TRADR Concept

In this section we present the research challenges addressed in TRADR in more detail and contrast the TRADR approach with related work.

Fig. 3
figure 3

Data fusion for grounding robot and maps. Figure from [23]

Full size image

2.1 Persistent Environment Models

Low-level situation awareness of the TRADR system requires sensory data from all involved robots registered in space and time, to keep creating and updating robot centric representations, and ground them into the world coordinate frame. The obtained representations are furnished to other parts of the TRADR system, which maintain higher level situation awareness. Persistent multi-robot environment models are grounded in two different aspects: environment representation and adaptive action.

Regarding environment representation, 3D mapping has so far essentially been studied for a single robot starting from an empty map. In TRADR we need to develop new data structures similar to octrees [42] and multi-resolution surfel maps [37] but with the added capabilities to integrate different sensor modalities from different robots, to scale to arbitrary environment sizes, and to cope with dynamic obstacles [34]. In order to achieve robust grounding we fuse all available modalities (Fig. 3).

Regarding adaptive action, impressive demonstrations of aggressive manoeuvres have shown the capabilities of UAVs but always in a closed environment with high-precision external tracking systems [25, 27]. To replicate these results in field experiments, it is necessary to improve the performance of current state estimation techniques relying on vision or laser sensors to complement IMU measurements [1, 41]. While for UAVs the difficulty lies often more in control since they are unstable systems, UGVs research is more focused on path planning. A plethora of algorithms allow robots on flat ground to find optimal paths using robot constraints [20, 35] but few approaches investigate moving in a rough terrain by using flippers [8, 31] and these are not yet ready for large-scale or dynamic environments. To this end, we develop algorithms to recognize different terrains in front of the robot and changing the morphology by adjusting the flippers (Fig. 5) for smooth traversal (Fig. 4).

Fig. 4
figure 4

Perception of obstacles, functional recognition. Overview of the Digital Elevation Maps (DEM) for given type of obstacles. Figure from [44]

Full size image
Fig. 5
figure 5

Robot modes Robot morphology, Figure from [44]

Full size image

2.2 Persistent Models for Acting

Building persistent models for action in TRADR basically corresponds to the human-robot team learning on the job. The models for acting will obviously rely heavily on the world models described above, but also learn from experiences generated in human-robot interaction on different autonomy levels.

Consider the following example. A UGV is sent to explore a given part of a building and retrieve some samples. It starts off in fully autonomous mode and successfully passes some difficult terrain and obstacles. Then it comes to an even more difficult area that is judged to be beyond its current capabilities. It stops and requests human support on a lower autonomy level. The human then guides the UGV across the terrain in an intelligent teleoperation mode. The choices made by the human during the traversal are stored and made accessible to the system. The path chosen by the human will be a preferred option in the next autonomous traversal attempt. Similarly, when a door needs to be opened or a sample of a possibly toxic liquid needs to be collected, the autonomous mode can request help by a human and then learn from that experience.

To achieve the above, we build upon state of the art approaches such as the intelligent teleoperation described in [30], the Click and Grab functionality of [2], the augmented virtual reality interface of [3], and the flipper position control of [31]. But the ambition of learning action models on the job on a team level goes beyond those approaches. Also the ambition of developing these persistent models will influence the design of the algorithms, leading to new results across all autonomy levels.

2.3 Persistent Models for Multi-Robot Collaboration

Multi-robot collaboration presupposes intention to collaborate, awareness of roles, partial knowledge, distinct beliefs, desires, capabilities and goals [4, 5, 11, 15, 29, 39]. Although significant research results have been achieved in the last thirty years, the concept of persistent collaboration is new in TRADR, as it requires persistence to be verified through sorties where an enormous amount of data is collected by the robot team. The challenge is to model how the information content of the data collected is preserved, and it is lifted to knowledge, while changing the team, changing the ways of communication and changing the experience gathered. Persistence asks for strong communication structures at different layers for role assignment, for distributed task inference and for sharing the team members current state. Persistence also demands consistent continuous information sharing which is especially hard in damaged environments and has never been experienced before.

We aim to develop a statistical-logical model for flexible collaborative planning. This model exploits the powerful language of the Flexible Temporal Situation Calculus (FTSC) [13], extended with constraints specifying dependencies between robot’ abilities and their spatial distribution, also accommodating statistical inference [33]. The model includes a knowledge and memory structure which is used, through sorties, to manage information sharing, common plan generation and dynamic role allocation. Both role and task allocation is based on a cost assigned to resources, robot groups capabilities, tasks and contexts [16, 24]. A learning schema, based on a Bayesian approach to tensor factorization is proposed to build a relation between group composition and costs [43]. Group reconfiguration exploits the stimulus-response framework, proposed in [17], modeling the human inspired mechanism of task switching in robot cognitive control. Finally, an extension of the ACL communication language is proposed for modeling the information flow between robots, in order to support collaboration [14]. This language is also used for knowledge retrieval and updating, via OWL [40].

2.4 Persistent Models for Human-Robot Teaming

As robots become more sophisticated a tendency has arisen within HRI to perceive them as teammates rather than tools [19, 32]; also in the context of disaster response robotics the importance of robots capable of operating as a (social) team-member has been acknowledged and addressed [12, 28]. Even though in NIFTi multiple robots were employed, they did not necessarily partake on the team-level; each robot was controlled by an individual operator taking orders from the human commander. This is similar in a number of other projects, where teams of heterogeneous robots are employed in a collaborative fashion, but it is human operators who provide the linkage between the robots and the human rescue workers, e.g., [7, 9]. A stronger notion of human-robot collaboration is developed in the alpine rescue project SHERPA [26], employing a metaphor of the human as “busy genius” who collaborates with a group of robots with different capabilities (the “SHERPA animals”) towards a common goal. TRADR will also go beyond an approach in which robots are mere tools, instead aiming at robots with an adaptive level of autonomy (e.g. semi-autonomous navigation, data gathering etc.) as members of flexible teams improving their collaboration over time. To realize this, TRADR is developing a framework for coordination of human-robot teaming, which is built on agent-based technology [18]. This framework manages the different roles, objectives, responsibilities and expectation for members of the team (which consists of both robots and humans and which may change over different sorties) and allows for conflict resolution and dynamical task-allocation depending on capabilities, task-load and chances of success.

2.5 Persistent Models for Distributed Joint Situation Awareness

Situation Awareness (SA) is paramount for a team to work effectively in disaster response missions [36]. To achieve robust SA on a team-level in TRADR, we are designing a Tactical Display System (TDS) that builds on the experiences gathered with the system developed to support distributed joint SA in NIFTi (Fig. 6, left) and existing end-users systems (e.g. the system employed by the GB fire-brigade, Fig. 6, right). The TDS will provide trustworthy and relevant tactical information about the physical environment and give access to a hierarchical representation of experiences to support tactical decision making (e.g., task allocation, (re)planning and coordination). It will be designed to support guided (a)synchronous information exchange between distributed or co-located actors through multi-modal interaction (graphical UI and spoken dialogue). This guidance needs to be personalized and context-tailored. A survey [38] found that in many cases adaptivity towards the user is realized through a customizable interface that does not significantly affect the behavior or interaction patterns of the systems. Following in NIFTi footsteps, TRADR aims to push adaptivity beyond simple widget placement, concretely adapting the system’s behavior to different use contexts.

Fig. 6
figure 6

Left: screen capture from the NIFTi TREX system showing a base map of the disaster area and various icons depicting location of rescue workers, robots, warnings, notes etc. Right: screen capture from the GB fire-brigade system showing various tactical information

Full size image

2.6 User-Centric Design and Development

TRADR adopts a scenario-based roadmap to guide iterative development of the persistent models described in the previous subsections, to drive continuous integration of the development results into a technical system, and to allow evaluation of the integrated system with end-users in yearly cycles (Fig. 2).

The roadmap defines a large-scale industrial disaster scenario. This is a kind of disaster where persistence is key to a successful mission. We need multiple robots to investigate the disaster from different angles (literally), and we need to use them over a number of sorties to gradually build up and maintain situation assessment, e.g., through observation and sample gathering. Within the industrial accident scenario, the roadmap then defines yearly use cases which deal with situation assessment under increasingly more complex circumstances, as described in Tab. 1. In Fig. 7 various use case setups at the TRADR end user training facilities are illustrated.

Fig. 7
figure 7

TRADR use cases set up at training sites in Germany, Italy, and the Netherlands

Full size image
Table 1 TRADR roadmap: year-by-year use cases within the industrial disaster scenario
Full size table

End users are closely involved in TRADR: each year of the development cycle in Fig. 2 starts by a deep domain analysis with end-users, followed by the development and integration of the components. The development cycle is rounded off by evaluating the developed components on system-level and performing end-user evaluations of the integrated system.

Integration takes place in a continuous process. An (as far as possible) automated procedure combines periodically the latest component versions, performs a static analysis of the code, and executes run-time tests. Reports of successes and failures are reported to the responsible developers, who can take the necessary actions. The components are mainly based on the ROS framework; however, since in TRADR more than one mobile robot is involved in the mission, we must set up a multi-master mechanism, which is necessary for the cooperation of multiple ROS-based systems.

2.7 Related European Projects

Several other European projects address the deployment of (teams of) UGVs and UAVs in various disaster response scenarios. ICARUS [9] and DARIUS [7] target the development of robotic tools that can assist during disaster response operations, focusing on autonomy. SHERPA [26] is focused on the development of ground and aerial robots to support human-robot team response in an alpine scenario. None of these projects addresses the persistence issues. In TIRAMISU [6], a toolbox is developed for removal of anti-personnel mines, submunitions, and Unexploded Ordnance (UXO). It includes a component called TIRAMISU Repository Service, which provides a centralized data-sharing platform that contains the locations of detected landmines and UXOs. The TRADR concept of persistent situation awareness goes beyond this in various respects as we described above. On the other hand, the EU project STRANDS [10], aims at modeling the spatio-temporal dynamics in human indoor 3D environments in order for a single robot to adapt to and exploit long-term experience in months-long autonomous operation. In contrast, TRADR deals with multiple sorties into an unstructured outdoor environment carried out by a human-robot team.

3 Conclusions

We presented an overview of the TRADR aims and approach. TRADR advances the use of the user-centric methodology established in the NIFTi project, and builds on the experience and insights obtained through the deployment of the NIFTi system, that there is a need for persistent, integrated situation awareness gathered over multiple sorties during a mission, and that different kinds of robots each play complementary roles in this process. To this end TRADR develops the capacity for persistent environment models, persistent multi-robot action models and persistent human-robot teaming.