Keywords

1 Introduction

With service robotics getting more and more important, robot demand has extended from factory automation towards mobile robot systems. Thus, the topic of mobile robotics has become important, and a lot of research has been performed. However, in some cases a single mobile robot is not sufficient to execute a task or deal with all problems [1]. For example, Knepper et al. [2] introduced the IkeaBot which is a coordinated furniture assembly system with multiple KUKA youBots. Based on that work, a flexible assembly system with cooperative robots was suggested in [3]. When multiple robots work together, cooperative mobile manipulation becomes important and poses new challenges—especially to the software structure.

Already in the 1990s, Dudek et al. [4] and Cao et al. [5] described a classification for cooperative mobile robotics. Dudek et al. [4] defined a taxonomy for multi-agent mobile robotics, where such a system can be e.g. differentiated by the number of agents, the communication range, topology and bandwidth as well as the reconfigurability and the composition of homogeneous or heterogeneous agents. Similarly, Cao et al. [5] defined research axes such as the differentiation, the communication structure or different types of modeling other agents. Moreover, they made a distinction between centralized and decentralized control in the software for mobile robots. Later, Farinelli et al. [6] added a classification based on coordination, where they compared cooperation, knowledge, coordination and organization of agents in multi-agent systems. This classification defines aware multi-robot systems where each robot knows that it is cooperating with other robots.

Concentrating on aware systems, this work analyzes which software structure and distribution can be used. However, there is not only one software structure and distribution that is possible in cooperative mobile robotics. From our point of view, there are different levels and aspects in the software architecture of a multi-robot system that can be distributed, ranging from the application level to low-level real-time device control. Typical early software architectures such as 3T [7] include reactive behavior as well as planning and execution, but are limited to single robot systems. The decision to distribute the software solution on one of the mentioned levels may affect the other levels and the complexity of the solution. Every possible solution, i.e. the chosen distributed software architecture, has its advantages but also its shortcomings that must be considered. For example, in current robotics systems component-based software frameworks which facilitate transparent distribution (e.g. ROS) are commonly used (cf. [8]). However, although communication between distributed components is easy using these frameworks, the decision about distribution as well as the assignment of responsibilities to certain components affects the overall capabilities of the solution (e.g. support for hard real-time).

Fig. 1
figure 1

Case study of two cooperating youBots: a Simulated scenario. b Real world scenario

Full size image

In this work, we introduce a taxonomy for distributed software architectures in cooperative mobile robotics. However, the concepts are not specific to mobile robots, but also apply to other cooperating robots such as teams of industrial robots in automation systems. Hence, we are interested in finding a generalized representation and description of distributed robotics systems that can be used to classify and compare software architectures of distributed robots. Additionally, we give advantages and disadvantages of applying distribution on different levels. It is important to be able to compare distributed robotics systems as the chosen software architecture often influences or sometimes even determines the complexity of the solution.

For experimental results, a case study is used where two KUKA youBots [9] physically interact with each other to transfer a work piece from one robot to the other. This scenario is inspected in different levels of difficulty. In simulation (cf. Fig. 1a), both robots can be coordinated in real-time, exact position information is available and all control inputs and trajectories are exactly followed. Initially, workpiece transfer happens while the first robot is standing still, and then while both robots are moving. As some of the assumptions made for simulation are not valid for real robots, a second scenario with real youBots is analyzed (cf. Fig. 1b). There, a youBot platform (left, without arm) is initially carrying a workpiece, which is then picked up by the second youBot (right) while both youBots are moving. The youBots and the workpiece are tracked externally using a Vicon optical tracking system, so precise position information is available.

In Sect. 2, the different levels for structuring and distribution of software for (mobile) robots is introduced. Subsequently, the identified levels (i.e. the real-time, system, and application level) are discussed in Sects. 3–5. Implications on the world model of robotics software are addressed in Sect. 6. To show the general validity of the suggested taxonomy, the possibilities of structuring and distribution on each level are explained using three different robotic frameworks in Sect. 7. Experimental results with different possible solutions of the case study are presented in Sect. 8. Finally, Sect. 9 concludes this work and gives an outlook.

2 Different Levels of Structuring Robotics Software

When designing a software architecture for a distributed robot scenario, we propose to group the software components into different layers as illustrated in Fig. 2. Each of the hardware devices present in the robot solution is represented and controlled by a device driver which is defined as the component that communicates with the hardware device through the vendor-specific interface. Additionally, the device driver is responsible for exchanging data with the surrounding software components. It has to derive control inputs and forward them to the device, as well as receive feedback from the device and make it available to other software components.

Each device driver can belong to a real-time context where data transfer and coordination between components occur with given timing guarantees. Depending on the implementation, the real-time context can contain only one device or span over multiple devices. Within a real-time context, reactions to events or the processing of sensor data can be guaranteed to happen before a given time limit. This allows to handle safety-critical situations that require timing guarantees (e.g. to stop the robot if an obstacle occurs), or to execute precise behaviors (such as actions that happen at a given point on a trajectory).

Fig. 2
figure 2

Software structure for distributed robots

Full size image

Above the real-time level, one or multiple real-time contexts belong to a system where all knowledge is shared between the components. Hence, all components within one system are allowed to access each other’s data, as well as to communicate with and send commands to each other. This allows components to directly include other components’ data into planning or computation, however no real-time guarantees are given unless the communication is handled within one real-time context.

To perform a desired task, systems can be controlled from applications that coordinate the work flow. Within an application, data is read and commands are sent to the controlled systems, so that the corresponding devices execute the task. However, if data from one system is required for an action in another system, it is the responsibility of the application or the deployment to facilitate the data transfer, as there is no concept of implicit shared data between systems. The overall behavior of cooperating robots results from the interplay of all applications that coordinate the robots.

Each application performs its work based on its world model, i.e. the knowledge about the controlled devices and systems, as well as about the environment including other (cooperating) devices. This includes geometric information such as positions and orientations of the relevant objects, as well as also physical data (such as mass and friction), shape data (such as 3D models for visualization or collision checks), semantics and ontologies. Information from the world model can be stored and used in applications, systems or real-time contexts, and can also be shared between different applications and systems. Structurally, the world model data can be differentiated into dynamic and static knowledge, with static knowledge (e.g. maps, shapes and ontologies) being valid everywhere, while dynamic knowledge (such as positions and sensor data) may be known in only one system or be different in different systems.

Depending on the requirements and technical limitations of the robot solution, the size and distribution of real-time contexts, systems and applications and thus the structure of the software can vary. The following sections discuss different design decisions concerning this structure based on the examples of the case study and using the three popular frameworks OROCOS, ROS and the Robotics API. OROCOS as a component framework mainly targets control systems with real-time guarantees [10]. The main focus of ROS is to be a component framework with transparent distribution, which over time has collected a large amount of algorithms as reusable components [11]. The Robotics API focuses on high-level robot programming using modern programming languages (such as Java) while still providing real-time guarantees [12].

3 Real-Time Level

First, the existing hardware devices and device drivers have to be grouped into one or more real-time contexts. Within a real-time context, reactions to events or the processing of sensor data can be guaranteed to happen before a given time limit. Having hard real-time guarantees allows to control precise behaviors or to handle safety-critical situations that need strict timing. In the mobile manipulator example of the case study, the available device drivers have to be grouped into real-time contexts, especially focusing on the two youBot platforms and one or two arms.

Fig. 3
figure 3

Variants of real-time contexts: a No real-time. b One device. c One device incl. logic. d Two devices. e All used devices, together with logic

Full size image

3.1 Software Structure on the Real-Time Level

Generally speaking, there are five different choices to structure these devices (and their device drivers) into real-time contexts. In the first case (i.e. the real-time context in Fig. 3a), the device driver software is written without real-time in mind. Here, the real-time context only spans the (possibly real-time capable) firmware or controller present in the device itself. For the youBot arm, this could mean that only the position control mode of the arm motor controllers is used. Thus, it is sufficient to give one joint configuration that the robot is expected to move to. While easy to implement, no synchronization between the joints or support for precise Cartesian space motions is possible. Moreover, no guarantees can be given regarding the interpolation quality of user-defined trajectories or the timing of reactions to events (unless supported directly by the device).

In the next case (cf. Fig. 3b), the device driver and the communication with the device is implemented in a real-time capable fashion. This requires to use a real-time operating system (RTOS) and more care when implementing the device driver, but allows to execute precise custom trajectories and handle sensor events with timing guarantees. Besides the device driver, additional real-time logic (cf. Fig. 3c) can be present that implements control, trajectory tracking or coordination of the device. For example, a real-time capable driver running at 250 Hz can be implemented for the youBot Platform on a RTOS such as VxWorks or Xenomai. It provides the motor controllers with smooth control set points for velocity or torque control, which allows precise user-defined trajectories or custom feed-forward or feedback control laws. However, as a drawback only information that is provided by the device itself can be included in the control law. For example, only the wheel position can be controlled exactly, but the position of the entire robot in Cartesian space is inaccurate (due to wheel slip and other factors limiting odometry precision), and the platform motion cannot be synchronized with the arm motion.

Increasing the real-time context, multiple devices can be combined up to all devices that are physically connected to the controlling computer (cf. Fig. 3d). Both the youBot arm and the platform—connected to the onboard computer via EtherCAT—can be controlled from a real-time capable software on a RTOS. In this way, coordinated motions between platform and arm are possible by combining the five joints of the arm and the three degrees of freedom provided by the omni-directional platform. This allows, for example, to execute Cartesian space motions of the end-effector relative to a point in Cartesian space known to the youBot (such as the position where the youBot started assuming that odometry exactly provides the current position relative to this origin based on wheel rotations). Additionally, one device can react to events that occur at other devices or are detected by other sensors. However, in order to be able to specify these reactions, either this part of the real-time logic has to be changed for a specific application, or a flexible specification language is required in the real-time logic [13]. To enable easy cooperation between multiple robots, the devices of all robots could be combined into one real-time context (cf. Fig. 3e). However, if the corresponding devices are connected to different PCs, real-time distribution becomes important to establish this kind of real-time context.

Fig. 4
figure 4

Examples for distributing real-time contexts: a No distribution. b Distribution over two computers

Full size image

3.2 Distribution of Real-Time Contexts

In the simplest cases, all devices structured into one real-time context are connected to the same computer (cf. Fig. 4a), which requires no distribution. Regarding our case study where both youBot platforms are not connected to one computer using a single (wired) EtherCAT bus, the real-time context has to be distributed in order to achieve this structure (cf. Fig. 4b). However, to distribute a real-time context, special real-time capable communication is required. For stationary robots such as manipulators or automation systems, as well as for complex mobile robots where different devices are connected to the different on-board computers (such as the PR2 or DLR’s Justin), this is possible through Ethernet or a field bus like EtherCAT. In the automation domain, standard equipment such as PLCs are used, while in robot research software frameworks such as aRDx [14] or OROCOS [10] are preferred. But between mobile robots, using a wired connection usually is no option, and standards for general purpose real-time capable wireless connections are not yet common, so providing a single real-time context is not yet widely usable. In summary, while distributing a real-time context over multiple computers can improve the scalability of the solution (w.r.t. processing power or device connectivity), the need for deterministic communication implies special requirements (such as field bus hardware or dedicated networks) that make the solution more complex or expensive.

4 System Level

Proceeding to the system level, one or more real-time contexts can be grouped into one system. Within a system, all components are allowed to access each other’s data, and to communicate with and send commands to each other. This allows components to directly include other components’ data into planning or computation, though no real-time guarantees are given (unless handled within one real-time context).

4.1 Software Structure on the System Level

Working with the two youBots, three different structures of systems are possible: As shown in Fig. 5a, each youBot can work in its own system optionally together with further logic used in the system. Then, no implicit data transfer occurs between both youBots. The second option is to group both real-time contexts for the two youBots into one system (cf. Fig. 5b), in addition with further computation logic. Then, the system introduces communication between both real-time contexts that however does not provide real-time guarantees. Finally, one real-time context spanning both youBots can be used in a system (cf. Fig. 5c), which allows real-time cooperation of the youBots. However, this may require a distribution of the real-time context.

Fig. 5
figure 5

Variants of system structure: a Two separate systems. b Two real-time contexts. c One real-time context

Full size image

Using a big system spanning all robots (cf. Fig. 5b, c) has the advantage of simplifying application programming or deployment: All the data that any component might need is made available everywhere in the system. Hence, no manual data transfer is required. This especially covers the world model. Within one system, a consistent world model is possible, because the best knowledge about the world is available to every component. Moreover, every change to the common world model is available to every component immediately.

However, there can be various reasons to use multiple systems: The sheer amount of data present in a big (multi) robot system can be a technical reason. Scalability can be limited by the management overhead induced by the data transfer between a great amount of components, and the addressing or mapping of data to components can become problematic. Furthermore, network bandwidth or reliability can be a limiting factor. In particular, this can be a problem when multiple robots are used that cooperate in varying teams. While for constant teams the corresponding robots could be joined into one system, varying teams quickly increase the required system size as all robots that might work together in a team at any time have to be within the same system.

But also more political reasons can opt for the separation into multiple systems, if cooperating robots belong to different people or parties. In this situation, not everyone might want to provide access to all of the robot’s data, or allow everyone else to control the robot. Then, matters of trust or access control become important that do not fit into the share-everything theme of a system. However, these reasons do not occur between the different devices within one robot (such as the arm and the platform of one youBot), so grouping them into two different systems does not really make sense and has been omitted in the figure.

Fig. 6
figure 6

Examples for distributing systems: a No distribution. b Separate logic. c Distributed devices

Full size image

4.2 Distribution of Systems

Looking at the distribution aspect of a system, some constraints are given by the real-time contexts: As the real-time contexts have to communicate with the corresponding hardware devices, the assignment to a certain computer is usually given. Depending on the devices and their connectivity, this can lead to a solution without distribution (cf. Fig. 6a), if all devices are connected to the same computer. In this case however, further logic components can be moved to a different computer as shown in Fig. 6b, based on performance deliberations. When the devices are connected to different computers and are not handled within a distributed real-time context, distribution of the system is required. Here, each real-time context’s assignment is given, while the further logic components can be assigned to the different computers based on further requirements (cf. Fig. 6c). For communication between the different components and real-time contexts, no timing guarantees are required. When data is to be exchanged between components on the same computer, communication can be performed locally, while otherwise network connections can be used to transfer the corresponding data and make necessary information available. Therefore, standard communication methods including wireless ones such as WiFi are applicable, however reliability or bandwidth can be limiting factors.

5 Application Level

To perform a requested task, one or multiple systems have to execute actions that are controlled and coordinated by one or multiple applications. There are various ways to specify applications: Mainly sequential workflows can well be expressed as a programming language control flow. For reactive behavior, model-based approaches such as statecharts (e.g. [12, 15, 16]) or Petri nets (e.g. [17]) can be more appropriate. Solutions offered by different robotics frameworks are discussed in Sect. 7.

Fig. 7
figure 7

Variants of application structure: a One app. b Two apps. c Three apps. d Two systems, like (a). e Two systems, like (b). f Two system, like (c)

Full size image

5.1 Software Structure on the Application Level

Figure 7 gives the different possibilities to structure the application(s) for controlling two youBots. One way is to control all robots from one application as shown in Fig. 7a. This defines all the interaction present in the solution in one place and thus makes it easier to understand. However, if varying teams are needed in a certain scenario, the corresponding application has to coordinate all robots at the same time. This can become confusing if the concurrent execution of multiple subtasks is encoded in one control flow or sequential state machine. Thus, separating concerns into subtasks, one for each team, should be considered within the application.

Another way is to use multiple applications, e.g. one for each controlled robot (cf. Fig. 7b). In this way, team behavior can be implemented by only locally and independently describing the behavior of each robot. However, the applications have to coordinate, either through explicit communication or through observation of the environment or other robots. As a third way, a further application coordinating both applications is a possible solution (cf. Fig. 7c), which however leads to a coordination application similar to the one in Fig. 7a. Using separate applications can also be required for political reasons, as described in Sect. 4.

In a multi-application cooperation scheme, however, the resulting behavior is not easily understandable by looking at one place, but only by examining the interaction of all different applications involved. In multi-agent robot systems (e.g. [18, 19]) every agent is controlled by one application. The overall behavior is given by the agents’ interaction. As shown by Hoffmann et al. [20], this can lead to self-organizing properties making the overall solution more robust. It is even possible that the application for each robot only implements low-level behaviors, and the resulting behavior completely emerges from the interaction [21].

Another structuring approach looks at the relation between applications and systems. It is possible to control (different devices of) one system from multiple applications (cf. Fig. 7b, c), one system from one application (cf. Fig. 7a, e) or multiple systems from one application (cf. Fig. 7d). The software framework should allow an application that was intended for use with multiple systems (e.g. Fig. 7d) to also work when all corresponding devices exist in one system, while the distribution aspect on the system and real-time level is handled by deployment (e.g. through configuration). This may however not be possible the other way round, if the application relies on having one system (or even real-time context).

5.2 Distribution of Applications

When looking at the application level, different possible variants of distribution can occur. An application can run on another computer than the controlled system(s). In this case, network communication is required to transfer data between the system(s) and the application. However, this kind of distribution is equivalent to distribution on the system level, when seeing the application as a component in the system. When multiple applications are used, they can be executed on different computers. Then, coordination between the applications has to be implemented in a way supporting remote communication. Finally, a single application can be distributed onto multiple computers. Therefore, programming language concepts such as remote procedure calls or service-oriented architectures can be used, which can however be used in a standard fashion and do not require any special, robotics-related treatment.

6 World Model

If a robot is expected to cooperate with another robot, it needs the relevant data about that other robot. In the easiest case, both robots belong to the same system, thus the required information is already accessible. Then we refer to both robots as controlled robots within this system. Once robots from multiple systems are to be coordinated from one application, the world model becomes more complex. The information about robots in other systems can originate either from communication or observation. If the other system provides (maybe read-only) access to the information, then we term the other robot a remote robot. However, if the information is only available through observation, we define the other robot as an observed robot. Thus, the world model has to keep (a descending amount of) information about controlled, remote and observed robots.

Fig. 8
figure 8

Different options to represent the world model of multiple systems: a Separate world models. b Global world model. c Mixed world model

Full size image

When working with two youBots, each of which belongs to its own system, some information (e.g. the robot positions) is available in different systems with varying precision, and has to be organized. One way is to keep one world model per system. Figure 8a shows an example of two resulting world models. The figure concentrates on geometric information, with coordinate systems (frames) indicating positions of objects in Cartesian space, while arrows represent positions or transformations between these frames. Solid arrows indicate controllable positions, while dashed arrows indicate positions retrieved through measurement or communication. Orange arrows (i.e. L1–L3) belong to the system of the left youBot, while blue arrows (i.e. R1–R3) belong to the right youBot system. The frames could be augmented by additional information such as shape data, which is omitted here for clarity. In this case, the system of the left youBot knows information about the left youBot’s start position together with the distance traveled, as well as current position information about the right youBot. On the right youBot system, the opposite situation occurs.

Using separate models has the advantage that multiple systems can just use different instances of the same world model, which allows to re-use models designed for single systems to a large extent. However, for cooperation a lot of information has to be duplicated, such as robot models of robots that occur as controlled robot in one and as a remote robot in another system. Additionally, the different world models have to be kept consistent. For example, a workpiece that is grasped and moved in one system also has to appear as grasped and moved in the other system.

The second way is to keep one global world model with all the objects and relations, and to provide access to the different transformations or sensor values for each system (cf. Fig. 8b). This has the advantage that the world model is always consistent (as far as topology and static information is concerned, however different systems can still disagree about object transformations), and structural changes performed in one system are automatically present in other systems. However, this scheme lacks flexibility when dealing with observed robots: While a mobile robot can keep track of its movement since the start through odometry measurements, an observer has no way to achieve this information from outside. Thus, the frame graph contains two transformations for controlled and remote robots (cf. R1 and R2 in Fig. 8a), while for observed robots only one transformation is available (cf. R3 in Fig. 8a).

To solve this problem, we propose a mixed world model scheme (cf. Fig. 8c). In a mixed world model, the static data is shared between all systems, while dynamic data can be different for each system. For example, information about physical objects (such as youBot geometry) as well as static connections (such as the position of the youBot arm relative to the youBot platform) are shared. Dynamic connections (such as the position of the youBot relative to the origin, or the fact that the youBot is positioned relative to the world origin or youBot origin) can be different for each system. Still, in both systems it should be possible to compute the transformation of the left youBot to the right youBot, using the data and topology present in each system (and to use it for planning and execution). This combines the advantages of a shared world model with the flexibility to include limited observations, while still allowing the application to address one youBot in a uniform way.

7 Support in Software Frameworks

When looking at the different software frameworks, a different focus becomes obvious. ROS itself puts no emphasis on real-time guarantees, so typically a real-time context only spans one device (cf. Fig. 3a–c), where a single device is encapsulated into a ROS node, providing an interface to execute the required local commands. Sometimes multiple devices (such as a youBot arm and platform) are combined into one node, however this leads to higher coupling. On the system level, all nodes that run using the same ROS master are seen as a system. In this situation, all these nodes can subscribe to any data published by other nodes, and post messages or actions, thus providing the transparent data exchange required for a system.

To implement applications in ROS, Python scripts can be used for sequential workflows, while SMACH state machines introduced by Bohren and Cousins [22] allow reactive behavior. However, communication with multiple ROS masters from one application is not natively supported. To share data between different systems (i.e. ROS masters), concepts like multimaster or foreign_relay [23] can be used to forward topics between multiple masters. The forwarded topics have to be configured during deployment. Additionally, working with multiple masters is one of the new use cases motivating ROS 2.Footnote 1 Within one ROS system, the world model is managed through data periodically published by different nodes. It includes transformation data as provided by the tf service, as well as robot models and data (e.g. sensor values) from other nodes.

In OROCOS usually most devices are combined into one real-time context (cf. Fig. 3b–e), because the framework targets real-time capable component systems with device drivers implemented in C++ on a RTOS. When using multiple robots, wired connections are used to ensure one real-time context, sometimes even for mobile systems [24]. Looking at the system level, OROCOS as a framework does not contain features for non-real-time communication, however often ROS is used to combine multiple real-time contexts into one common system [25] that allows for non-real-time communication and data sharing. Consequently, OROCOS does not provide direct support for access to multiple systems. Control flow can be expressed in LUA scripts, while complex coordination is possible using rFSM statecharts as suggested by Klotzbücher and Bruyninckx [15]. World models are usually implemented as components in an application dependent manner within the real-time context, which can include geometry, semantics and history [26].

Using the Robotics API, the underlying Robot Control Core [27] is implemented in C++ for Xenomai and includes real-time capable drivers for devices connected to the corresponding computer. Additionally, the Realtime Primitives Interface [27] allows for the flexible definition of real-time logic to execute user-defined tasks. In this way, all devices physically connected to the computer can form a real-time context (cf. Fig. 3c–e). The system term here refers to the concept of a RoboticsRuntime, which represents one real-time context and makes the data available to applications in a non-real-time way as well as to other devices in the same context for real-time reactions. Control flow can be expressed directly in Java applications, as well as through statecharts, e.g. by using Java libraries for SCXML. It is easily possible to use multiple systems in one application (as in the Factory 2020 case study [12]) and to share limited amount of data between different systems or applications—either through common data sources (such as a Vicon system connected to both youBots through WiFi) or through explicit direct transfer. The frame graph [28] contains semantic information such as the relation between frames (e.g. if they are static, temporary or have a transformation that can change during execution). With further information about physical objects (e.g. their physical properties or shape), it serves as a world model in an application and can be used for planning as well as task execution.

8 Experimental Results

The simulation experiments described in the case study have been performed using the Robotics API as a framework and its corresponding simulation and visualization engine. A single application as shown in Fig. 9a was used to control one system (cf. Fig. 5c) containing a single real-time context (cf. Fig. 3e). In this real-time context, both youBot arms and platforms were simulated, as well as the youBot grippers. The application was programmed in an object-oriented fashion, referring to the robots and work pieces as software objects. Moreover, the application used a single world model and expressed all interaction in the control flow of a Java method. Its initial version, where the first robot was not moving during transfer, was easily extended into the second version where both platforms were moving. This extension mainly consisted of adding commands to move the first platform, and to make the second platform synchronize its motion to the position and movement of the first platform. This showed the simplicity of programming and synchronizing robots in the idealized simulation case where every device belongs to the same real-time context and can be synchronized, and where exact positioning is available.

Fig. 9
figure 9

Software structure for the example application: a Simulation application. b Real application

Full size image

Transferring the results from simulation to reality, various changes had to be done. The tests were conducted on two KUKA youBots using a Vicon optical tracking system for external localization. A straightforward approach would be to combine both youBots into one real-time context. Then, the same distribution scheme as in simulation could be re-used, as well as large parts of the implementation. However, lacking real-time communication over wireless networks (cf. Sect. 3.2), this was not easily possible. On the other hand, while for stationary manipulators adding a common real-time context greatly simplifies and improves the precision of physical cooperation, in the mobile case the gains are less clear. This is because precise cooperation does not only require exact timing synchronization, but also exact spatial synchronization. For stationary manipulators, this can be achieved by appropriate calibration procedures. For mobile systems, this is in general more problematic due to sensor inaccuracies. External positioning systems—e.g. the Vicon system used here—can mitigate this problem. However, wireless real-time communication becomes a problem again when it comes to transmitting the position information to the youBots.

Thus, we decided to choose an alternative distribution approach as shown in Fig. 9b. On each internal computer an instance of the Robot Control Core was running, which formed the real-time context (cf. Fig. 3d) as well as the system for the corresponding youBot. On the system level, this structure corresponds to Fig. 5a. Vicon tracking data for both youBots and the workpiece was streamed to both youBot systems through a WiFi connection from an external PC running the Vicon software. Looking at the application level, each youBot was controlled from a separate application (cf. Fig. 7e). The motion of the platform carrying the workpiece was controlled through teleoperation. The other youBot was controlled by a Java method similar to the one in the simulation case.

However, both applications used separate world models (cf. Fig. 8a). The workpiece and the other youBot platform were not modeled as Java objects, but only the Vicon position data was used to synchronize the motion and find the grasp position. The youBot arm used joint impedance control to mitigate position inaccuracies. Still, the experiment succeeded and the work piece could be transferred. Instead of separate world models, a single application with a mixed world model could have been used, which would be closer to the (single) world model in the simulation case. This way, changes to the world model topology (e.g. the information that the object has been grasped) would have automatically been transferred to the second youBot’s system and static position data would be known to both youBots.

9 Conclusion and Outlook

In this paper, we introduced different levels for structuring the software for distributed robot applications: real-time, system, application level. Moreover, implications on the world model have been discussed. The structure on the different levels can be used and combined independently, motivated by technical as well as political constraints. The different options for structuring and distribution have been explained based on a case study of cooperating mobile manipulators and various robot frameworks, and evaluated both in simulation and in real world setup with two KUKA youBots. In the example applications, different ways to distribute the software on different levels have been introduced, and the advantages and drawbacks for the given scenario have been shown.

Overall, it became clear that there is not a single optimal way of structuring and distributing the software. The levels presented in this work will hopefully be a starting point that can help developers in designing and discussing their software architecture. Based on non-functional requirements to the developed solution (e.g. reactiveness, synchronization quality, data privacy, trust), the choice of the appropriate distribution scheme and framework(s) for implementation should become easier. As next steps, we plan to implement the mentioned other ways of distribution and to evaluate the gains for the given scenario. This especially includes the use of a mixed world model, as well as ways to share a world model between multiple applications or to synchronize relevant structural changes.