Keywords

1 Introduction

Current developments in automation, planning and artificial intelligence allow future UAV systems to perform increasingly complex tasks. At the moment these systems heavily depend on a connection to the command center on the ground, which delegates the corresponding tasks and makes further decisions if necessary. Communication delays or disturbances, e.g. by hostile jamming, to this command center can eliminate these advantages [9]. This problem is tackled by the concept of manned-unmanned teaming (MUM-T). In MUM-T, manned and unmanned mobile assets (air, land, sea, space) interoperate to pursue a common mission objective. The unmanned platforms as well as their mission payloads are commanded by the manned asset. The required mission planning and management capabilities for a single operator are a highly relevant field of research [1, 3]. The high work demands, arising from the multi-platform mission management and tasks execution, besides the usual pilot tasks, necessitates a certain degree of automation. Although highly automatic planners are feasible to solve such multi-vehicle planning problems in real time, they increase the risk for automation-induced errors such as the loss of situational awareness, complacency, or opacity [19]. The Institute of Flight Systems (IFS) at the Universität der Bundeswehr München addresses these problems by developing adaptive assistance, mission management and guidance systems. In previous research we studied the team-based guidance of three UCAVs (unmanned combat aerial vehicle) from aboard a single-seat fighter aircraft [4]. Even though all missions were successfully completed, the intentionally chosen high degree of automation temporarily led to mental under load of the pilots and was lacking in adaptability to balance the operator’s activity and work demands in the sense of degrading situational awareness and complacency over the course of the mission. The experimental subjects further expressed the desire to be able to assign dedicated tasks to the UCAVs during mission execution, especially in less demanding situations [4]. In the helicopter domain, we investigated multi-UAV guidance on a task-based level [16]. During the mission execution the crew was supported by an adaptive assistance system, that recognized the currently performed tasks of the crew, determined the workload and proactively avoided phases of excessive stress [2]. The evaluation of the concept with German helicopter pilots showed the advantages of the concept such as reduced workload and increased performance [14,15,16]. On this background we developed interaction concepts [7, 8] and implemented a mixed-initiative mission planner for the guidance of multiple UCAVs from aboard a manned fighter cockpit [5, 6]. This work experimentally evaluates the intervention possibilities of this system with German Air Force pilots and is structured as follows: First the different actors, their roles and relations are described with the help of a work system analysis. Based on this, we present the human machine interface for the task delegation and the intervention possibilities in this process. The different interventions are then experimentally examined and evaluated in realistic mission situations.

2 Approach

The initial step in the development of a MUM-T system for the cockpit-based cooperative UCAV guidance is a top-down analysis of the individual system participants and their relationships. The Work System notation, described in [17], provides a semantical and graphical language for such a top-level system design with strong focus on human-automation work share and is used in the following. Within the Work System there exist two roles, defined as follows:

  • Worker: The worker knows, understands and pursues the Work Objective by own initiative. There has to be a human taking the role of the worker, in any case.

  • Tools: The tools receive the orders from the worker and will only execute them when commanded. Usually conventional automation (e.g. FMS, auto pilot) takes the role of a tool.

Fig. 1.
figure 1

Work system of the mixed-initiative planner, as part of the Assistance System (Color figure online)

Full size image

Three different types of actors, i.e. humans, artificial cognitive agents, and conventional automation, are now integrated into the Work System. These actors are assigned in accordance with their capabilities and given requirements, to the Worker or Tool role. The entities of the Work System can stand in a hierarchical (green connector) or heterarchical (blue connector) relationship to each other. Figure 1 depicts the current work system for the cooperative UCAV guidance in the fighter domain at the IFS. The system contains cognitive agents installed aboard the UCAVs and the manned fighter aircraft. The cognitive agents aboard the UCAVs are responsible for the execution of assigned tasks in the role of a tool. In case of errors or situational changes, these agents are capable to pursue their tasks independently as long as the plan is not affected, however they cannot pursue the Work Objective independently. The cognitive agent (Assistant System) aboard the manned fighter supports the pilot in mission planning and execution and therefore adopts the role of a worker in this system. The pilot (i.e. the Human Worker) stands in a hierarchical as well as a heterarchical relationship to the assistance system. The hierarchical relationship enables the pilot to delegate tasks to the UCAVs through the assistance system [7]. A mixed initiative mission planner in the assistance system integrates the delegated task into the mission plan of the UCAVs, considering resources, constraints and timings [6], and then delegates the task to the UCAVs through the hierarchical relationship between the AS and the UCAV Agent. The heterarchical relationship between the pilot and the assistance system enables the system to support in the task assignment processes, the resolution of planning conflicts and the identification and improvement of sub-optimal plans. The intervention concept is based on the basic requirements for assistance according to Onken and Schulte [13], which are defined as follows:

  1. 1.

    Draw the attention of the assisted human operator(s) with priority on the objectively most urgent task or subtask.

  2. 2.

    If the person is overtaxed, transfer the task situation into a manageable one for him.

  3. 3.

    Only take-over tasks that the human is principally not capable to accomplish, or which are of a too high risk or likely a cause of too high costs.

These two distinct modes of cognitive automation (i.e. the hierarchical and heterarchical relationship) [17] allow the mission planning to be initiated and executed by both parties. Regardless of who initiated the planning, the following steps have always to be performed:

  1. 1.

    Selection of the desired task

  2. 2.

    Delegating the task

  3. 3.

    Integrating the task into the mission plan of the UCAV

According to the assistance levels presented above, steps 1–3 of this process can be partially or completely taken over by the assistance system. These different types of the cockpit-based UCAV guidance (with/without assistance) and the corresponding human machine interface are presented below. Afterwards, the appropriateness of the different interventions is evaluated in different mission scenarios with German Air Force pilots.

3 Human Machine Interface

This chapter describes the Human Machine Interface (HMI) for the cockpit-based cooperative UCAV guidance with different levels of intervention. First, the hierarchical relation of the HMI, the task creation and delegation, is presented. Then the adaptation of the HMI to the heterarchical relation, i.e. the different intervention levels, of the assistance system are presented.

Fig. 2.
figure 2

Human machine interface

Full size image

3.1 Delegation

The hierarchical relationship between the pilot and the assistance system enables the task delegation to the UCAVs. This interaction takes place via the multi-functional display of our experimental fighter cockpit, shown in Fig. 2. The pilot first selects a target on the tactical map. According to the selected target [10] the pilot can then create a task using a radial context menu at the target location, see Fig. 2a. After the task creation the pilot can adapt the task parameters, (Fig. 2b) and delegate it to the team members with the delegation interface, shown in Fig. 2c. The task creation and delegation process is described in detail in [8]. The position of the own fighter aircraft (grey symbol) and the team members are indicated in Fig. 2d&e. The red circles, Fig. 2f, mark enemy radar or missile defense positions which shall be suppressed or, if possible, circumvented for safe mission execution.

3.2 Assistance

The heterarchical relationship between the assistance system and the pilot allows the system to support the pilot in the task creation and planning process, compare Fig. 3. These intervention levels are analogous to the to the basic requirements, as follows:

Fig. 3.
figure 3

Different types of intervention, hint simplification, take-over

Full size image
  1. 1.

    Hint: Pop up dialog box (PUD) at the target with description of the missing task, the pilot has to create the task himself and assign it to a team member via the delegation interface as described before, see Fig. 3.1.

  2. 2.

    Simplification: PUD at the target with description of the missing task, as well as the most suitable team member for the task, is shown to the pilot. Additionally, the position for the new task is visualized in yellow on the time line. For the delegation of the task and the replanning, the pilot can accept the proposal directly in the PUD, Fig. 3.2.

  3. 3.

    Take-over: PUD on the target with description of the automatically delegated task and the corresponding team member. The pilot can revise this decision by clicking decline, restoring the old plan, Fig. 3.3.

4 Experimental Setup

The evaluation of the different interventions is carried out in a human-in-the-loop experiment. In this context, the influence of incorrect interventions (propose engagement of SAM position that can be circumvented) on the mental workload (MWL), and system acceptance are examined. Another focus is set on the experimental determination of the adequacy of the intervention level, i.e. to high/low intervention, in the missions. Therefore, we first develop hypotheses and define missions and situations based on these hypotheses. These missions are then integrated into the simulator, shown in Fig. 4, and carried out by German Air Force pilots. Retrospectively the different situations in the missions are replayed and evaluated with the help of questionnaires.

Fig. 4.
figure 4

The MUM-T fighter simulator at the IFS.

Full size image

4.1 Hypotheses

The following aspects are to be examined with regard to the mental workload and appropriateness of the intervention:

  1. H1:

    False intervention increases the mental workload (i.e. propose/take-over engagement of SAM position that can be circumvented).

  2. H2:

    Simplification/take-over intervention reduces the MWL if the target, a UCAV or the fighter is threatened.

  3. H3:

    Intervention is undesired if there is enough time to solve the problem.

  4. H4:

    The automatic delegation of tasks is preferred when the own fighter is threatened.

Fig. 5.
figure 5

Example mission (A) with the different mission phases: Endangering of high value target (B), UCAV (C), Fighter (D) and Route (E).

Full size image

4.2 Missions

The hypotheses are examined with three missions containing the desired types of situations. For briefing technical reasons and to avoid automation surprises, a fixed intervention level was defined for each mission, i.e. all interventions in a mission were on the same level. Each mission contained the reconnaissance, engagement and battle hit assessment of a high value target and the reconnaissance of two secondary targets. The missions were pre-planned and the pilot was responsible for the target engagement and target verification. The other tasks were performed by the UCAVs. Throughout the mission area pop up threats, i.e. enemy surface to air missile sites (SAM), had to be expected. The rules of engagement in these missions stated that those threats should only be engaged if an aircraft or high-value target is endangered. An exemplary mission with the individual mission phases is shown in Fig. 5A. The letters on the left side indicate the positions of the different threats in the mission scenario. In Fig. 5B-E the situations occurring during the mission execution are shown in detail. The first situation in this mission (B) is the threat to a target, in the sense that no aircraft is within range of the SAM site and there is sufficient time to complete the task, i.e. delegate a HARM task to suppress the enemy SAM site. The second situation (C) describes the pop-up of a threat with one or more UCAVs in range, represented with a red circle. In this case, an immediate reaction of the pilot is necessary. An escalation of this situation is shown in Fig. 5D, with an additional threat to the manned fighter aircraft. The last situation (E) shows a threat to the route. In accordance with the rules of engagement, the threat has to be circumvented. Each of the missions covered these four different situations and to eliminate possible spill-over effects from previous pop-up SAMs, the occurrence of two threats in the missions were at least 60 s apart.

4.3 Data Acquisition

After the mission execution, the four situations, shown in Fig. 5, were replayed and the adequacy of the intervention was determined. Additionally, the impact of the intervention on the pilots’ mental workload for the situation was assessed. For this purpose, the mental workload with and without assistance was determined using NASA-TLX questionnaires. As performance measure we evaluated the interaction time between the pop up and the elimination of the threat.

5 Results

The missions were carried out with eight active German Air Force pilots. First, the effect of the intervention on the mental workload is discussed, followed by an evaluation of the performance and questionnaires.

5.1 Mental Workload

The effects of the different intervention stages on the mental workload without an acute threat are shown in Fig. 6. The interventions in the case of an endangerment of the route, Fig. 6 left side, showed a slight workload increase for all types of interventions. In case of the hint and simplification, the pilots had to reject the dialogue message. For the take-over intervention the pilot had to contradict the faulty intervention, i.e. the engagement of the SAM site, otherwise the rules of engagement would have been violated and therefore the mental workload slightly increases. However this increase was too small to support the hypothesis H1. The workload results for an active threat to an UCAV or the manned fighter are shown in Fig. 7. When the UCAV and the fighter are threatened, the hints lead to a higher workload. This can be explained by the fact that in addition to the delegation of the engagement of the threat the hint had to be rejected. For the simplification and take-over interventions little or no effect on the mental workload can be observed and therefore the hypothesis H2 has to be rejected.

Fig. 6.
figure 6

Nasa TLX scores of the different intervention levels for the threatening of the route and a target.

Full size image
Fig. 7.
figure 7

Nasa TLX scores of the different intervention levels for the threatening of the UCAV and fighter.

Full size image

5.2 Performance

The evaluation of the interaction times is divided into the different intervention levels (i.e. hint, simplification, take-over). For the hint, only the dialog had to be removed for the route, thus showing a very short reaction time. In case a target, UCAV, or the fighter was threatened, a HARM task had to be created and delegated to a UCAV. There is a huge time difference if a target or a UCAV is threatened in contrast to a manned fighter. This can be explained by the fact that in the case of a threat to the own fighter aircraft, evasive maneuvers were immediately carried out and then the delegation of the task was addressed. For the intervention with the simplification it showed up that the proposals for the route, the UCAV and the fighter were processed equally fast. The elimination of the threat to the target was somewhat faster, which could be interpreted as a higher situational awareness of the area. The evaluation of the take-over intervention only shows a valid time for the route, as the wrong decision of the system had to be counteracted. For the target, the UCAV, and the fighter, the time when the dialogue was closed is shown here, but the threat is eliminated from the system intervention.

Fig. 8.
figure 8

Interaction time of the individual intervention stages for the elimination of a threat to the route, target, UAV or fighter.

Full size image
Fig. 9.
figure 9

Evaluation of the questionnaires after the mission.

Full size image

5.3 Questionnaires

The evaluation of the questionnaires is shown in Fig. 9 and the results are referenced as follows (Question number, Intervention, Situation), the intervention and situation are abbreviated with the initial letter. The hint was evaluated rather negatively in the situations, since the pilots did not feel supported (Fig. 9.1H). Hypothesis H3 is supported by the fact that the majority of pilots did not wish more intervention if a route was threatened (Fig. 9.4HR). On the other hand, if the targets are threatened, the pilots desired more intervention although there was enough time to solve the problem (Fig. 9.2HT & 4HT). In case of a threat to the UCAV and the fighter (Fig. 9.2HU & 2HF), the pilots did not have enough time to solve the problem with the hint, which also correlates with the interaction times (Fig. 8). In case of a threat to the target, the UCAV, and the fighter, the pilots also desired more support by the system (Fig. 9.4H) than a simple hint. One problem here was that in addition to the dialog message, a sound notification is triggered when a threat occurred. The pilots remarked that such an intervention would be useful if they had no situation awareness for the task. In contrast to the hints, the simplification of the task was evaluated very positively. However, some test persons expressed the desire for more support of the system, especially in case of a threat to the fighter (Fig. 9.1SF & 4SF). For the automatic take-over of tasks it was shown that the faulty interventions when threatening a route (Fig. 9.1TR & 3HR) were evaluated too positively by the pilots, because here a false system decision was made. A reason for the positive evaluation could be the simple correction of the incorrect intervention by the decline button (compare Fig. 3). In the case of a threat to the manned fighter, the intervention level was found to be more appropriate than the simplification (Fig. 9.4SF vs. Fig. 9.4TF). This supports the hypothesis H4, however significance cannot be shown.

6 Conclusion and Future Work

In this work we experimentally evaluated the effects of different intervention levels in MUM-T mission. Starting with a work system analysis we derived two modes of automation, delegation and assistance, required in such MUM-T missions and presented their realization in the HMI. The assistance functions of this system were then systematically evaluated with German Air Force pilots in four different threat situations (route, target, UCAV and fighter). For each of the three intervention levels (hint, simplification and take-over) the impact of the intervention on mental workload, performance and system satisfaction was evaluated. The results showed that simplification and take-over intervention are desirable in time-critical situations. However, the automatic take-over of tasks in non mission critical situations (threat of route, target and UCAV) was criticized. This corresponds with the third basic requirement of [13] to only take-over tasks that the human is principally not capable to accomplish. A similar effect was observed for the hint intervention, here the intervention was unnecessary, because in case of a threat to the route or the target, the pilot directly realised the situational change and in case of a threat to the UCAVs or manned fighter aircraft, a higher intervention was desired. In order to achieve an adaptive intervention, more extensive analyses of the current situation, the pilot’s situational awareness [18] and his current mental workload [11] must be incorporated into the decision-making process of the assistance system [12].