Keywords

1 Introduction

The R&D project CASIMUS (Cognitive Automated Sensor Integrated Unmanned Mission System) deals with the challenge to support the crew of a two-seated military transport helicopter with sensor equipped unmanned aerial vehicles (UAVs). The MUM-T—approach ensures that no further chains of command are involved as these UAVs are directly deployed during missions by the helicopter crew. They shall enable the crew to reconnoiter the intended routes of flight and to survey certain regions of interest such as operation areas or potential landing zones. To achieve such nearby and time-critical reconnaissance, the UAVs and their payload sensor systems have to be scheduled, controlled and sensor output needs to be monitored by the helicopter crew. This is expected to influence the crews’ workload through the extension of the crews’ task spectrum and a higher overall mission complexity. To counter this potential workload increase, an adaptive, cognitive assistant system [1, 2] provides situation-adapted support by continuous crew supervision and aims to balance crew workload. For UAV platform guidance, a high-level task-based-guidance paradigm has been proposed [3]. Similarly, a corresponding automation layer will be proposed in this paper for the domain of mission sensor deployment, where human centered adaptive automation is expected to reduce and simplify reconnaissance tasks, regarding technical limits of the automated system functions. To select a suitable level of automation, an agent system suggests the automation level by reweighting the current user needs and the available technical capabilities. The sensor assistance system developed in this context is prototyped and tested in simulated MUM-T transport helicopter missions at the Institute of Flight Systems (UniBwM).

2 Problem Scope

2.1 Automation as a Solution

The nature of the UAV reconnaissance system used in CASIMUS, containing multiple distributed sensor carrying platforms directly guided from an airborne manned helicopter, differs in many ways from conventional UAV systems. In conventional systems, the platforms are controlled and monitored from a ground control station (GCS), where a dedicated payload operator monitors and controls the sensor system, while the UAV pilot controls and operates the vehicle itself. In the MUM-Teams considered, one of the helicopters crew members has to take on these tasks, most likely the pilot-not-flying (PNF). A task spectrum similar to the GCS’ payload operator ones would be added to the PNF’s regular tasks. The need to control multiple UAVs and monitor their sensor output by only one operator further aggravates the situation, which may eventually lead to an excessive amount of workload. Moreover, the resulting operative workstation environment differs substantially from those of conventional UAV systems.

A higher degree of process automation would be the classical way to deal with these problems resulting from intensive user involvement, continuous monotonous activity or high human workload. To efficiently incorporate the new sensor operation activities in the regular task range of a PNF, machine perception capabilities integrated into the sensor systems should to be on his hand as supporting, automated high-level functionality.

2.2 Automation as a Problem

In today’s cockpit environments, human crews are used to work with highly automated systems, characterized by reliable and deterministic behavior. Still, complex automation can lead to out-of-the-loop—effects of the crew [4], misinterpretation of automation states (“Opacity-effect” [5, 6]) or relying without question in automation systems (“Over-Reliance” [7]). Wrong belief about the automation systems state and capability is known to be a reason for human failure. Distrust in automation will occur when automation systems don’t work in a reliable way. The operators’ degree of “trust in automation” is a criterion whether to use automation support at all or perform the task in person [8].

When it now comes to offer automated functions in the domain of mission sensor deployment and data assessment, the problem may be further aggravated due to domain specific technical circumstances. Software based exploitation of data acquired by mission sensors bears the vulnerability to fail or deliver non-reliable results under changing circumstances (e.g. false positive or true negative detections). Mainly, environmental changes (daytime, weather, nature of ground, illumination conditions and signal-to-noise-ratio) are affecting the correctness of delivered output. Specifically computer vision algorithms have system inherent uncertainties that yield to technical result imperfection, especially under a wide operational range. Knowledge about critical conditions for successful usage is often hidden or reserved to expert users.

Furthermore, mission sensor systems carried on UAVs have performance limits, based on physical measurement principle and sensor resolution. Again, expert user knowledge is necessary to cope with these features. To take this into account, the probability of failure [9] during operation must be considered when automatic sensor data processing is employed.

2.3 General Approach

From the above one can summarize that sensor data evaluation can be performed alternatively by machine based automation systems or by the human operator. Machine-based evaluation can process data with high bandwidth, works fatigue-proof, but can’t ensure perfect detection performance. Man-based evaluation is exhausting, causes high human workload, only on limited bandwidth, but brings in very high cognitive human capabilities.

Main requirement for a technical system which tackles this antagonism is to achieve a balance between tolerable human workload and necessary reconnaissance performance. Thus, in our context, the proposed assistant system has to deal with the occurrence of human workload as well as reliability of complex technical systems’ automation. To achieve this goal, variable, situation-dependent levels of automation are proposed to be used in this domain to represent adaptive machine capabilities combined with different stages of human involvement.

3 Related Work

This chapter aims to relate the proposed approach to background paradigms and previous work.

In [10], three basic requirements for assistant system behavior were proposed to be applied by assistant systems for crew support. Here the assistant system acts on a stepped de-escalation scheme. It ranges from guiding the crews’ attention to the most urgent task over transferring a complex task situation in a manageable one up to allocating specific automated means for the execution of tasks that are not accomplishable by the crew [1]. Applying these design requirements to a sensor assistant system fosters the idea of situation-dependent crew support and machine-process execution. In this context the paradigm of a variable degree in automation is referred to as adaptive automation or variable automation. Different level arrangements result in stepped involvement of the human operator [11, 12]. Adaptive automation systems select a suitable level of automation or choose a situation-adequate mode depending on the current context [13]. Changes in automation degree can be initiated by both, the human operator and the technical system. The system is specified as adaptable (if the user invokes changes) or adaptive (change initiative by automation itself) [14]. As pointed out above, offering variable levels of automation shall be used in the proposed sensor assistant system to counteract work-load intensive situations.

The guidance of multiple UAVs out of a helicopter cockpit with support of an cognitive assistant system was successfully demonstrated in simulated missions [2] in a past R&D project. However there, idealized sensor deployment was assumed, which did not require the crew to interact with the UAVs sensor system. The assistant system proposed in this paper now should also support the crew when operating more state-of-the-art payload systems.

Also in [2] an error-free, nearly fully automated target recognition (ATR) system approximated sensor data processing. In CASIMUS now data processing will be aspired which includes typical artefacts and imperfections, in order to investigate on the reliability issue discussed above. Therefore, a “Sensor- and Perception-Management System” framework [15, 16] (SPMS) shall be integrated which hosts mission sensor data processing algorithms. This framework administers environment—and context-adapted machine perceptive capabilities by selection and application of appropriate image processing algorithms. High level capabilities realized this way can be applied for typical airborne surveillance tasks such as aerial mapping as well as object detection and tracking. In addition, each algorithm is accompanied by a “trustworthiness” figure which assesses its reliability in the given circumstances.

4 Sensor Assistant System

Figure 1 gives an overview of the system blueprint for offering sensor deployment support, implemented along the requirements set out above. The figure shows three divisions:

Fig. 1
figure 1

Sensor assistant system overview

Full size image

  • Automation Functions: a repository of installed machine-performable automation functionality in the domain of Computer Vision and Gimbal Control

  • Management: A decision mechanism to determine the appropriate automation degree depending on different input conditions. Results are forwarded to the operator or, under specific conditions executed automatically.

  • Knowledge Models: models of human working processes and their costs of human resources allocated occurring during performance for different automation levels (“Sensor Taskmodel”) like illustrated in [17]; model of automation levels (“LOA”)

The sensor assistant system receives several inputs. Two are of main interest, the workload measure of the human operator (provided by an external system [18]) and the computer vision algorithms’ trustworthiness (provided by the “Sensor- and Perception-Management” System [15, 16]). These inputs are model-based values; the workload is a taskmodel-based construct, and the trustworthiness is delivered by a reliability estimation process. The sensor assistant systems’ goal is to allocate the best fitting automation function setup for varying task, workload and automation trustworthiness parameters. To react on this input signals, an agent system was designed, which is able to propose changes in the automation level. These level suggestions can be verified by superordinate authorities, before the automation allocation for those levels takes place.

4.1 Sensor Automation Functions

In this chapter the available automation functions are explained in more detail.

“Computer Vision.” Automated functions in this group are intended to relieve the human operator from perceptive-cognitive activity (e.g. interpreting acquired sensor data), which demands high user involvement, specifically when it comes to monitoring parallel data sources from teaming UAVs. Therefore, a spectrum of different task-related algorithms from the computer vision domain has been implemented and integrated. The implementation of the algorithms follows the modular scheme as laid out in [16]. Further, variations of sensor data access and data visualization functions are available (e.g. mosaicking of aerial images).

“Gimbal Controller.” The mission sensors onboard the UAVs considered are typically attached to a cardanic mount (gimbal) to enable dynamic sensor gaze control during flight. The degrees of freedom of the two axis demand for continuous user steering when special ground tracks or ground patterns are required to be scanned. An automated gimbal control application was developed to decrease the humans’ sensorimotor activity when operating the gimbal. The automation functions offer several control modes that are useful for the existing task spectrum.

4.2 Levels of Automation

As described in Sect. 2.3, the two aspects human workload and technical system reliability should be balanced. For this purpose, different levels of automation from the repertoire of automated functions described above had to be laid out. According to the scheme of “Levels of Automation” [11, 12], several levels with different depth of user involvement are suggested. This ensures, that the distribution of task load can be varied between man and machine purposefully.

Motivated by the aspect of human workload reduction during task performance, the level design proposed foresees in one direction an increasing automated support which potentially results in workload reduction. However high-levels of automation through computer vision bear the potential of high machine unreliability. This aspect indicates to prefer lower level of automation and therefore more robust data processing. This leads to the assumption that moving towards lower level of automation increases trustworthiness but also increases workload and vice versa. This interrelation leads to the effect, that human workload can be reduced as long as systems’ reliability does not undercut a certain value. In the opposite direction, the fallback in lower levels, induced primarily by low automation trustworthiness, involves the human tighter, but ensures that the automation delivers more trustable results.

Automation Levels for Sensor Data Evaluation. Figure 2 shows the arrangements and composition of image processing and information presentation functionalities as levels. These levels can be mapped on the left side scale indicating the abstraction level and presentation properties of resulting surveillance data.

Fig. 2
figure 2

Levels of automation in data evaluation

Full size image

With upwards level steps, data will be transformed from signal (image) data to abstract information, similar as pointed out in [19]. This efficiently can reduce the bandwidth of data to be analyzed by the operator and supports their spatial correlation to deduce mission relevant results. Further, the representation form of data can be varied from an “image stack” (chronology of images/Full Motion Video) to a spatial map representation by either image stitching (level 2) or spatial arranged object information on a tactical map (level 3). The red/blue schematic in above figure thereby symbolizes the expected “workload” division between human and machine system.

In terms of trustworthiness, moving on levels upwards introduces machine imperfections step by step. Applying computer vision methods to detect objects on level 1 may be prone to high false-positive and true negative detections. Additionally when moving to level 3, false classifications can contribute to a further reduction in trustworthiness.

In contrast, the selective level adaption in downwards direction is aimed to involve the operator when automated functions appear to fail, thus introducing a more intense cognitive fusion performance on human side [9], where preprocessed data and information provided by maximally exploited automation capabilities are delivered to the crew for confirmation and/or further information extraction.

Automation Levels for Gimbal Controlling. The domain of payload steering also offers variable automation, Fig. 3 shows the automation functions’ usage graded in four levels.

Fig. 3
figure 3

Levels of automation in payload gimbal automation

Full size image

The automated sensor control functions offer the capability to scan ground patterns of routes and areas automatically (level 1). In higher levels, system initiated zoom adaptations update the sensors’ ground sample distance (GSD) automatically (level 2). The highest automation level (level 3) issues navigational requests to the platform’s FMS (flight management system), requesting platform repositioning. These levels should reduce the human’s sensorimotor activity, control process complexity, system process supervision times and intervention demands.

5 Machine Decision Making for Selecting the Automation Level

5.1 Working Method

When using the suggested automation levels in a (simulated) surveillance UAV scenario, the critical parameters that initiate the automation level adaption need to be observed as base for the decision process. Figure 4 shows the functional core that is aimed to determine the necessary change of the automation degree based on the two main input variables. First of all, the humans’ workload is considered as the main indicator for the decision process to act. In addition, also the automation functions trustworthiness has to be taken into regard, as motivated above, to find the best available solution (automation level) from the predefined set to satisfy the workload reduction request.

Fig. 4
figure 4

“Automation level determinator” based on an agent system

Full size image

The core of the automation management consists of an artificial agent system, which utilizes of a Markov Decision Process (MDP). With regard to modelling, this method was chosen because the problem comprises full control ability about the state transitions (in contrast to Markov Chain or Hidden Markov models) and, in this first prototype, completely observable states were assumed (in contrast to Partially Observable MDP). Such agent system, according to [20], can be classified as: an artificial, controlled, reactive, individualistic, conservative, nonreasoning no perception and no memory agent.

First, a manageable state scenario had to be found, on which the MDP operates. The MDP consists of set of such internal states S, a set of actions A to transfer between the states when they are applied, a transition model T (which defines the transition probabilities between the states) and a reward model R (defines rewards for applying actions in the states).

By this a state space model was build (Fig. 5). The five states S represent the impact of automation usage on the HC-Crew and the consequence of imperfect automation. The available agent actions A describe five different modifications of the automation degree. The agent states and actions were defined as:

Fig. 5
figure 5

State space transition model

Full size image

  • S = {S1, S2, S3, S4, S5} = {“no Support”, “untrusted automation”, “over-support”, “optimal automation”, “under-support”}

  • A = {A1, A2, A3, A4, A5} = {“start Automation”, “stop Automation”, “increase level”, “decrease level”, “keep current state”}

Figure 5 shows the state space transition model with the feasible transitions (emitting specified agent actions) between the states. The states regard human workload (blue marked in Fig. 5) and automation trustworthiness (orange marked in Fig. 5) aspects. The actions are affiliated with specified rewards, directing the desired agent decision behavior.

The solution of this MDP—problem produces a policy that yields to the highest reward for action application. This policy is a list of optimal agent actions that are applied in the different states.

On stimulation, the agent’s inner system state is updated by an observation. The state interpreter in Fig. 4 determines a current state. On this state S, a specified action A is applied, and a subsequent state S’ will be reached. The agents’ objective is now to reach the inner target state S4 “optimal support” by applying selected actions, which is aimed to transform the environment observed by the agent system (human workload) to an optimized range (minimized workload). The following chapter shows the automation management system described above working in a reduced toy problem setup.

5.2 Proof of Concept in a Toy Problem Setup

During mission performance, the human workload is assumed to vary over time, mainly depending on the performance of tasks for mission fulfilment. When using automated sensor systems in mission environments, the productivity and efficiency of machine recce capabilities may change (expressed by the automation trustworthiness), caused by varying environment conditions, viewing distances, viewing angles. Hence, the best fitting automation level has to be determined online whenever such changes occur.

To demonstrate the sensor assistant systems’ capability to handle such situations, a set of input variables was created, comprising a trend of measures of human workload and automation trustworthiness (Fig. 6). In this trend, a fixed threshold (“minAutomationTrustworthiness” in Fig. 6) of 75 % is assumed to classify automation as suitable to support.

Fig. 6
figure 6

Timeline of schematic input signals, use cases, a value threshold and activity peaks

Full size image

In the toy problem setup, the received input triggered automation level adaption activities. From the trend in Fig. 6, four typical use cases (1–4) were taken that were to be handled and solved by the agent at discrete time steps. The agents’ actions were applied on these use cases and the automation level adaption took place.

The assistant systems’ outputs are presented in Fig. 7. Depicted in magenta are the agents’ decisions and the consequential changes between the automation levels.

Fig. 7
figure 7

Timeline representation of agent actions and resulting automation levels

Full size image

In this setup, a high base workload of the HC-Crew was assumed, so the sensor automation runs already in level 3 of 4 from the beginning.

As shown in Figs. 6 and 7, the agent automatically reacts to the varying input parameters mentioned above. High human workload initiates automation degree increase, as long as minimum trustworthiness margin is not undercut (use case 1). Too low automation reliability decreases the automation level (use case 2) or prevents from changing to an automation level that potentially induces higher automation failure rates (use case 3). At last, the assistant system notices the return of sufficient automation trustworthiness and initiates a level increase if necessary (use case 4).

With the application of the agent decisions on the use cases illustrated above it can be demonstrated that the agent systems’ models and actions are suitable to handle the different situations occurring in this problem field.

6 Realization Process and Future Work

After this first functionality proof of the decision making core, the transfer of the concept in an executable implementation will follow. Therefore, the system will be interfaced to make use of the combined task-, resource—and interaction model (“Sensor Taskmodel” in Fig. 1) to regard task-specific differences.

For functional demonstration of the holistic HC-assistant system applying sensor automation with usage of the proposed sensor assistant system, a closed-loop-operation will be realized. Finally, the evaluation of the concept and the effects on operator performance will follow in human-in-the-loop experiments with transport helicopter crews.