Keywords

1 Introduction

The use of multirobot systems in civilian missions requires high variability due to the diversity of domains [4, 21]. Moreover, robotic systems are defined through a craftsmanship instead of established engineering processes. Programming missions for robots requires high knowledge of robotic programming and robot mechatronics. While domain users are experts in their domains (e.g., emergency, commercial and agriculture) they are not trained to program missions for multirobot execution in their domains using the low-level APIs provided by robot vendors. Not much has been done to enable domain experts to easily use robots to execute missions in the respective domains.

To address this problem, Di Ruscio et al. introduced FLYAQ [3, 7]. FLYAQ is a platform designed to enable non-expert domain users to program missions for a team of multicopters. The platform has been then generalized to different types of robots in [4, 6]. The platform is extensible, so that domain-specific robots and missions can be defined. Unfortunately, this platform can only define missions at design time. This is unrealistic since most missions will be faced by unforeseeable and emergent situations during mission execution, and, consequently, robots should be resilient to these unforeseeable and emergent situations. For example, one robot may malfunction calling for re-planning so that another robot can take the roles this robot was executing. This need for run-time adaptation is clearly described in the Robotics Multi-Annual Roadmap 2020 [21]. In this context, the document describes the degree in which models can be used in robotics in three steps ([21] Sect. 5.2). Step 1 assumes that models are used to define missions by people at design time. Step 2 requires robots to use models at run-time to interact and explain what they are doing. Step 3 means that robots adapt and improve models to redefine what they are doing based on artificial intelligence.

The FLYAQ platform uses models according to step 1. In this line of research, we intend to improve FLYAQ to support self-adaptive robots at the mission level, thus achieving step 3. This means that robots can change their behaviour to successfully carry out missions under unforeseen circumstances. We achieve this by introducing a declarative language for describing mission goals and constraints. In this research we exclusively focus on the high-level strategic, domain-specific, collaborative aspects of self-adaptation. To this end, we specify mission objectives in a declarative way, as properties, using a language we call the Mission Specification Language (MSL). We present a technique that allows the generation of such a MSL for a specific FLYAQ extension (e.g., emergency, commercial, agriculture). As MSL is declarative, it does not specify how the mission is planned for a team of robots, but instead specifies what goals must be achieved and what constraints cannot be violated. This way, missions become fully specified only at run-time and they can be re-planned at run-time.

Paper structure: Sect. 2 discusses the background of this research. Section 3 introduces the property specification language. Section 4 evaluates the approach by showing an implementation of the property specification language. Section 5 discusses related work. Section 6 concludes the paper with opportunities for future works.

2 Background

In this section, we briefly explain domain-specific modelling, and the FLYAQ platform, on which we build our research.

Domain Specific Modelling. In Domain-Specific Modelling (DSM) [14], a methodology in model-driven software engineering, the general goal is to provide means for domain users to model systems in their problem domain. Model-driven techniques such as metamodelling and model transformation enable the creation of Domain-Specific Modelling Languages (DSMLs). These DSMLs can be used by domain experts, to specify, for example, missions for a team of robots. Current DSM techniques allow domain users to model at the domain level and simulate, optimise, and transform the model to other formalisms, synthesise code, generate documentation, etc.

Fig. 1.
figure 1

(adapted from [6]).

The family of FLYAQ DSMLs

Full size image
Fig. 2.
figure 2

(from [7]).

A screenshot of the FLYAQ tool

Full size image

FLYAQ Platform. The FLYAQ platform [3, 4, 7] employs domain-specific modelling to take care of the various domains involved in mission definition and specification. The approach proposes a family of DSMLs for the specification of missions of multirobot systems (MMRSs), as shown in Fig. 1:

  • Monitoring Mission Language (MML): this DSML consists of the context layer and mission layer. This DSML is meant to be used by domain users, to model missions. Missions are represented in the mission layer as sequences of tasks on a map, as shown in Fig. 2. The context layer provides additional constrains over the mission area, such as obstacles and no-fly zones;

  • Robot Language (RL): using this DSML, types of robots or individual robots can be defined by a robot engineer, mapping out their capabilities and characteristics;

  • Behaviour Language (BL): this language allows the definition of sequential atomic movements and actions of each robot that are used to instruct the individual robots. The BL serves as the low-level language, to which high-level missions defined in MML can be transformed automatically using the MML2BL transformation. This transformation takes care of low-level planning, such as path finding, covering areas, etc. while achieving the high-level goals. Code can be easily generated from the generated BL models, and then it can be uploaded to the individual robots.

Mission goals, robot characteristics, and actions should be customised to the application domains. Therefore, extensions can be defined on MML, RL and BL, as shown in Fig. 1. In case of MML, extensions may define a task to “scan an area by taking pictures”. Example extensions to RL may include domain-specific notions like “number of propellers”, “launch type” (horizontal or vertical takeoff), “maximum altitude”, etc. BL may be extended with movements like “take off” and “land”, and a “go to strategy” (move first over the horizontal or vertical axis, or move diagonally?), “take a picture”, “start recording a video”, etc.

For example, it is possible to define extensions in FLYAQ to allow flying robots to take pictures of areas. Using this extension, one can specify missions to e.g., survey an area where a public event is being held. Another example is in the domain of agriculture. One multicopter is able to detect pests by taking pictures and using image recognition techniques. If a pest is detected, another multicopter that is able to spray insecticide must spray the infected plants. It should only spray plants that are infected. We use these examples throughout the paper.

Despite its extension mechanism, FLYAQ does not support (a) advanced temporal constraints (other than order, fork or join) over various tasks or robots in MML, e.g., a certain task can only start if another robot is surveying the task area (for safety reasons), or video recording can only start after clearance (for privacy reasons); and (b) run-time adaptation of a mission due to some information at run-time, e.g., taking pictures of areas where high temperature was detected by another robot, or reacting to a loss of signal of a robot. The research presented in this paper addresses these shortcomings.

3 Mission Specification Language

Our approach extends the FLYAQ platform as shown in Fig. 3. The mission layer of MML is annotated, and as a consequence a Mission Specification Language (MSL) can be generated automatically from MML and a Property Template to better match the platform extensions of MML. MSL extends MML with language constructs to define temporal properties for robot missions. Our approach ensures that, when an extension is defined as done in FLYAQ, no additional effort is required to generate MSL.

Fig. 3.
figure 3

Overview of the approach as an extension of the FLYAQ platform.

Full size image

3.1 Mission Specification Language

The mission specification language (MSL) is intended to specify properties of a mission that allows users to define temporal mission constraints in a highly declarative way. This complements MML, where areas are selected, and specific tasks, obstacles and no-fly zones are plotted on the map. MSL replaces the order, fork and join of MML, supporting more expressive constraints. We use a number of temporal patterns, taken from Dwyer et al. [8] and Autili et al. [1], as a basis for the Property Template from which MSL is generated. According to this work, properties consist of a temporal pattern in a scope, over some propositions P, Q, R and S (i.e., occurrences of something, e.g., spraying, entering an area, etc.). Temporal patterns can be absence (something should never occur), universality (something should always occur), existence (something should eventually occur), bounded existence (something should occur at most n times), precedence (an occurrence of P must be preceded by an occurrence of Q), or response (an occurrence of P must be followed by an occurrence of Q). Scopes can be globally, after the occurrence of R, before the occurrence of S, between occurrences of R and S, or after an occurrence of R until an occurrence of S (after until).

The declarative constraint specification shields the user from the actual planning. For example, if pests are detected, the corresponding areas are sprayed. This is an example of a response pattern with global scope. The user may use a precedence pattern to say that a pest needs to be detected at a location before this point is sprayed. This constraint can be met in a number of equally valid ways. A first option would be that one robot first detects all locations, then returns to the base where its data is downloaded and locations of infected plants are uploaded to a second robot, who goes out to spray the infected plants. A second option would be that two robots perform the task in parallel: one robot sends coordinates of detected pests to the other robot, which only sprays infected points. The second robot may follow a preplanned path, or may plan its path at run-time, according to the received coordinates. Collisions may occur, or may be avoided by flying at different altitudes. A third option would be that multiple robots detect pests, and multiple robots spray. If robots can adapt their mission at run-time, this may involve advanced scheduling, employing run-time monitors [5]. This shows that a declarative language can be supported by very simple to very advanced algorithms. The goal of MSL is that the domain user is shielded from such advanced planning algorithms.

To further illustrate MSL, we give some more examples of properties.

  • Between entering and exiting an area, a robot can never exceed a given altitude. According to Dwyer et al. [8], this is an absence pattern with between scope. Note that this between scope may be more intuitively expressed as “during” or “while”.

  • Between receiving a “stop” message and a “start” message, pictures cannot be taken.

  • A robot can only start its activity if another robot is in a given position to monitor this activity.

3.2 Run-Time Adaptation of Multirobot Missions

In its current state, the MML platform generates robot missions at design time. This means that robot missions cannot be adapted at run-time. We intend to support the run-time recalculation of BL models (i.e., robot commands) from a declarative mission description; this is needed in case information at run-time prompts the robots to change the mission. Our approach is applicable to various implementation techniques: for example, the mission recalculation may be achieved by the robot or by the ground station, and may be specified off-line or at run-time, or a mix of these.

Fig. 4.
figure 4

The annotated class diagram of the MML mission layer (context layer remains unchanged and is not shown).

Full size image

In order to allow run-time information in a mission specification in MSL, we altered the existing MML mission layer from [4], as shown in Fig. 4. We have changed the metamodel in several ways:

  • We have extracted a Shape class (and Polygon, Point, Line subclasses) from the original PolygonTask, LineTask, and PointTask. In particular, the new Polygon class serves now as superclass of Area in the context layer of MML in [6]. This new Shape class will allow users to specify new shapes on the map that may trigger rules like: do not record within a specific area.

  • The meaning of Task has been extended. At mission specification time, a task may be addressed by multiple robots. After mission generation, tasks are split up into multiple concrete tasks, each for one robot.

  • TaskDependency has been removed from MML and its functionality will be subsumed by the specification language.

  • We added run-time language constructs (annotated with rt), so that specifications can be defined in terms of the current state of the mission in terms of tasks and position. We added the following run-time information in terms of tasks:

    • currentTask: the task a robot is currently working on;

    • coveredTasks: the concrete tasks that are planned for a robot;

    • todoTasks: the concrete tasks that a robot still needs to perform;

    • finishedTasks: the concrete tasks that a robot has done;

    • performingAction: the action (defined in the task) a robot is currently performing. It may be none if e.g., the robot is moving and the action is instantaneous (e.g., taking a picture).

    We added the following run-time information in terms of position:

    • currentPosition: the current position of a robot;

    • coveredPoints: the points of a concrete task that are defined by the cover function;

    • todoPoints: the points of a concrete task that still need to be visited;

    • finishedPoints: the points of a concrete task that have been visited;

    • in: the shapes the robot is currently in.

As is usual in FLYAQ, extensions can be defined for specific application domains, as shown in Fig. 3. Note that for brevity, we do not show the MML context layer and RL (which can be extended in its own right).

3.3 Generation of the Property Specification Language

As shown in Fig. 3, a domain-specific MSL can be generated from the annotated MML (as shown in Fig. 3), with defined extensions (e.g., to enable detection of pests in an area, and spraying certain plants). This means that extensions have to be defined only once, and can be used for specifying missions in the original MML as well as in MSL. The metamodel of MSL, which results from the language generation process without an extension, is shown in Fig. 5. It consists of three parts:

  • Mission layer: the upper part (unshaded) represents our variant to the original MML mission layer, which allows the user to define missions at design-time like in the original MML. For example, “pictures should be taken in an area, with a distance of x from each other”. Additionally, shapes can be defined, that can be used in MSL properties. In case of an MML extension, extensions will also appear in this part.

  • Temporal pattern layer: the middle part (shaded) represents the temporal patterns, which allow the user to define temporal constraints based on the patterns by Dwyer et al. [8]. For example, “after R happens, P must be followed by Q”.

  • Proposition layer: the bottom part (unshaded) represents the language fragment to define propositions P, Q, R, and S of temporal patterns. More specifically, it allows the user to specify a condition on the state of a mission (i.e., a structural pattern). For example, “a robot is in a specific area”, or “a task is completed”. In case of an MML extension, pattern versions of extensions will also appear in this part.

With MSL, a mission can be specified by plotting an area on the map, and defining a DetectPest and Spray task in this area, using the MSL mission layer, which is extended with language concepts from agriculture. With the MSL temporal pattern and MSL proposition layer, a property can be specified that states that detecting a pest at a location must result in spraying that location.

Fig. 5.
figure 5

The generated metamodel of MSL without extensions.

Full size image

The MSL metamodel of Fig. 5 is generated fully automatically from an annotated (and possibly extended) MML metamodel (Fig. 4) and the generic property template (shaded part of Fig. 5). Why do we need to automatically generate an MSL metamodel? Please note that, if there was no generative approach, each of the domain-specific language constructs (e.g., spraying a pest, maximum altitude of a robot, etc.) would have had to be modelled a second time in MSL. With our approach, the extension mechanisms of FLYAQ can be reused as described in [4], and a domain-specific MSL is generated without any additional effort.

We use techniques from the ProMoBox framework [17, 18] to achieve this. First, the MSL mission layer is generated by taking the annotated MML metamodel and removing all run-time language constructs, which are annotated with rt, thus creating the unshaded upper part of Fig. 5. Next, the property template is merged into this model by adding an association called specification from Mission to Specification. Finally, the annotated MML metamodel is taken for a second time, and the run-time language constructs are kept (by removing the annotation). This time, the metamodel is converted into a structural pattern language by using the RAMification process [16]: relaxing all lower multiplicities, making all abstract classes concrete, and changing all attribute types to Condition, as can be seen in the proposition layer of Fig. 5. This RAMified metamodel is merged into MSL by generating inheritance links from all top-level classes to PropertyElement.

3.4 Transforming MSL to BL

Transforming MSL to BL (see MSL2BL in Fig. 3) can be done according to several stategies. Given the tight relation between MSL and MML, the transformation algorithms of MML2BL [7] (i.e., path finding, covering areas) can be reused. Moreover, various implementation strategies as mentioned in Sect. 3.1 can be covered by MSL2BL: mission recalculation may be achieved by a mix of the robot or the ground station, off-line or at run-time. These strategies may requiring enhancement of BL to e.g., explicitly support data communication or monitoring. As this paper focuses on the definition and generation of MSL, this is left as future work.

4 Evaluation: Implementation of MSL as Textual DSL

In this section, we evaluate the MSL by introducing an implementation as a textual language in Xtext [9], and show how missions can be expressed in this language.

4.1 A Concrete Syntax for MSL

According to what described in [1], temporal properties (the shaded part of Fig. 5) might be profitably described using a structured English grammar. For instance, we can devise a textual syntax for the MSL proposition layer (the bottom part of Fig. 5), where each of the associations can form a subsentence with the two attached instances. For example, “a Robot currently on a GeoCoordinate” denotes the presence of an instance of Robot and an instance of GeoCoordinate, with a currentPosition link in between. More intricate, “a Robot r currently on a GeoCoordinate with latitude lower than 100” denotes additional conditions on the robot, etc. A structured English grammar to represent a subsentence for one association is defined as follows (id, Label, Value, Attribute, Class, Association are terminals):

figure a

The above grammar is combined with the grammar for temporal properties presented in [1] so that temporal properties can be described in standard LTL or CTL. This might enable the use of model checking approaches, like UPPAALFootnote 1. With this grammar, temporal patterns involving multiple links can be expressed with AndPatterns. MML extensions can be used by instantiating classes defined by the extension. This is illustrated below in the examples.

Fig. 6.
figure 6

Screenshot of the MSL in Xtext.

Full size image

Our current implementation in Xtext includes variable name resolution, parse error visualisation, auto-completion and syntax highlightingFootnote 2. A screenshot of the MSL editor is shown in Fig. 6. Since both the FLYAQ platform and MSL are implemented on top of the Ecore platform, they can be easily merged at the EMF layer [19].

4.2 Examples of MSL

This section presents examples of temporal properties defined in MSL, while illustrating the relation between the grammar presented above and the MSL metamodel presented in Fig. 5. For these examples, we define a MML extension in the agricultural domain as shown in Fig. 7, with:

  • DetectPest: scanning for a pest in an area and in case of detection, send some coordinates;

  • Spray: spraying pesticides at a point;

  • ReceiveCoordinates: receiving coordinates where a pest has been detected, with the “at” association referring to the received coordinates.

Note that, after generation of MSL, these additional language constructs will occur twice in MSL, namely in the MSL mission layer and in the MSL proposition layer.

Fig. 7.
figure 7

The MML extension.

Full size image
Fig. 8.
figure 8

Three examples of temporal specifications as instances of MSL.

Full size image

The example of Fig. 8 (top) shows the abstract syntax of the MSL property “a robot only sprays at a location if it has received these coordinates to spray at that location” as an object diagram. The Specification consists of a Precedence pattern. The left AtomicPattern states the condition Q, saying that a ReceiveCoordinates task is executed at a coordinate p. Note how the “at” association is used. The right AtomicPattern P describes a robot r, spraying at aforementioned point p. Note that the coveredPoints link is superfluous, because if the robot is currently performing an action of a task, it must be inside the task area. In structured English grammar, the temporal specification is as follows (leaving out the superfluous quantification and coveredPoints link): “Globally, if a SprayRobot r performing a Spray and r on a Coordinate p , then it must have been the case that a ReceiveCoordinates at p beforehand”. Note how “at” is automatically resolved to an instance of the “at” association.

The example of Fig. 8 (bottom left) represents “in a certain area, a robot can never exceed a given altitude”. Note that the Area class (now a subclass of Polygon) is part of the MML context layer and is not shown in Fig. 5. In structured English grammar, the temporal specification is as follows: “Globally, it is never the case that a Robot r in an Area with name as “lowflyzone” and also r on a GeoCoordinate with altitude more than 20”.

The example of Fig. 8 (bottom right) represents “a robot can only perform a certain task if another robot is at a certain position”. In structured English grammar, the temporal specification is as follows: “Globally, it is always the case that a Robot performing a Task implies a Robot on a RelativeCoordinate with x as 100 and y as 200 and z as 10”. Note that, since spatial constraints are at the very core of FLYAQ, it is interesting to introduce syntactic sugar for a robot being at a coordinate, e.g., by allowing syntax like “a robot on (100, 200, 10)”.

5 Related Work

In this section, we present related works on run-time adaptation of multirobot missions. There are several works that focus on robotics and self-adaptation, like [10, 11, 20]. For the sake of space in this section we focus on related works in run-time adaptation of (robot) missions, with a focus on MDE approaches. While most of the mission specification tools (e.g., [23]) and the FLYAQ platform [3, 6] provide for specification of multirobot missions at design time, there is need to have specification and recalculation of missions at run-time for missions executed under uncertain environments.

In an effort to leverage run-time adaptation for UAV based systems, the work in [2] uses an ensemble concept to aggregate teams collaborating in a mission at run-time. This platform focuses on the aggregation of agents but on not the high-level expressiveness of the mission properties. Using run-time models for automatic reorganization of multirobot system, the work in [24] focuses more on techniques for task distribution based on the goals and organisation of the teams, but not how goals are expressed so that adaptation at run-time is made easy. In [12] robots adapt models at run time, but configurations are made by an expert programmer, not domain experts declaratively. The work in [15] focuses on the behavioural model and how it auto-validates at run-time, yet we employ a generative approach. The work in [22] focuses on design-time to run-time explication of models, however this work does not really deal with adaptation triggered by run-time uncertainties. The work in [13] proposes an approach that uses models at design-time and run-time for collaboration. The proposed approach is specific to a particular domain without a clear path to adapt it for working in other domains.

6 Conclusion and Future Work

In this paper, we extended the FLYAQ platform with MSL, a highly declarative language that allows users to describe robot missions with temporal properties as constraints. The declarative nature of MSL allows run-time adaptation of these missions in case of unforeseen circumstances. We showed how MSL can be automatically tailored with domain-specific extensions by a generative approach. Additionally we presented a structured English grammar for MSL.

Future work will mainly focus on the mapping from MSL to BL (a language for describing individual robot movements and actions), allowing run-time adaptation and exploring different execution strategies. We intend to model communication between robots and/or the ground station explicitly in BL to achieve this. Furthermore, we are planning to incorporate real-time constraints in missions. Moreover, since our approach for enabling run-time adaptation of missions is model-driven and relies on code generation, we intend to analyse the feasibility of generating code in real-time.