Abstract
This article describes and validates a concept of a workload-adaptive associate system for military helicopter crews. We use adaptive automation to support helicopter pilots during Manned-Unmanned Teaming missions, where the crew of a manned helicopter operates several unmanned aerial vehicles from the cockpit. We introduce a cognitive agent, which behaves like an additional, artificial crew member. It dynamically adjusts its level of assistance by choosing different workload-adapted strategies of assistive intervention depending on free mental resources of the crew. To evaluate the prototype, we conducted an extensive pilot-in-the-loop campaign and analyze situations of “near misses”, where the associate system corrects human erroneous behavior.
You have full access to this open access chapter, Download conference paper PDF
Similar content being viewed by others
Keywords
- Adaptive automation
- Associate system
- Cockpit automation
- Workload-adaptive
- Human factors
- Human-agent teaming
1 Introduction
At the Institute of Flight Systems, the concept of Manned-Unmanned Teaming (MUM-T) is a well-established approach to improve future military aviation. Guiding multiple reconnaissance Unmanned Aerial Vehicle (UAV) from the cockpit yields many advantages including information gain, flexibility and safety. But the additional tasks of operating the UAV and process the information of their sensors increase the workload of the cockpit crew. If the demand exceeds the available mental resources (i.e. workload peak), e.g. in time-critical multi-tasking situations, it results in performance decrements and human errors [1]. In the aviation domain, automation has always been a solution to reduce workload. But shifting more tasks from the human to (conventional) automation brings other problems: Loss of situation awareness, workload peaks due to high demanding cognitive tasks, which remain with the human, and “automation induced workload peaks” due to clumsy automation [2].
To counteract this, Onken and Schulte introduce two modes of cognitive automation [3, 4]. This automation is able to handle and support cognitive tasks, to avoid exceeding mental resources on the one hand and too little or wrong human involvement on the other hand. One mode is, to establish a cognitive agent (, see Fig. 1) onboard of each UAV which facilitates a task-based (instead of parameter-based) UAV guidance [5]. The agent is able to understand high-level commands, formulated as tasks, and reports the results back on the same task-based level. Several studies show, that these cognitive agents enable the crew of a manned helicopter to guide multiple UAV from the cockpit while performing complex mission scenarios [6].
As second mode, we introduce a cognitive agent (, see Fig. 1) as associate system to support the crew adapted to their mental state. The goal is, to prevent workload peaks and human errors while keeping the crew “in the loop” to prevent out-of-the-loop problems [7], complacency and automation bias [8]. Therefore, the associate system takes the role of an artificial, restrained-behaving co-pilot. Onken and Schulte describe a guideline for this restrained behavior as a set of rules [3]. The basic rule is, to let the crew do their job as long as possible without the associate system intervening. However,
-
1.
if the attention of the human is not on the objectively most important task, guide the human’s attention to that task;
-
2.
if, nevertheless, the human is overtaxed, transform the task situation into one, the human can handle again. And only
-
3.
if the human is in principle not able to perform the task and the cost of a failure would be too high, adopt the task as a last resort.
These escalating behavior rules link the type and amount of assistance to the attention and mental state of the crew. Therefore, the adaptive associate system needs a context-rich representation of mental workload [9]. It is not sufficient to know if the workload is high, but in addition it is necessary to know the reason, i.e. the causal task situation, which leads to high workload. Only the knowledge about the task situation enables the agent to support the crew task-based, also for cognitive tasks.
2 Task-Based Operationalization of the Mental State as Precursor of Adaptive Assistance
As basis for the workload-adaptive associate system we introduce a context-rich definition of mental workload, as described in [9]. It includes
-
the currently pursued work objective and the resulting tasks, which are necessary to achieve this objective, i.e. plan;
-
the set of tasks, which the operator currently executes, i.e. activity;
-
the demand on mental resources, which is necessary to execute the activity and
-
behavior patterns, which the operator typically shows during the task execution and variations from these patterns [10].
2.1 Task Model
We operationalize this definition of mental workload with tasks. Therefore, we developed a hierarchical task model which contains the domain knowledge of our application, i.e. all tasks which can occur during a MUM-T helicopter mission [11]. This machine-readable task model enables the associate system
-
to have sufficient knowledge of the domain, similar to the knowledge of the human crew. This includes the demand on mental resources, which is necessary to execute a specific task, constraints for tasks and relations between tasks;
-
to communicate with the crew in a natural manner, since humans communicate very efficient by using tasks as expression for very complex situations;
-
to communicate in the same efficient way between different modules of the associate system;
-
to know different variants of task sharing between the crew and the automation. According to [1], working memory load can be reduced by automating tasks. That is, increasing the level of automation, means to decrease the involvement and therefore the taskload of the human operator. The Levels of Automation (LoA) in our task model define, how the associate system can reduce the crews taskload by automating a specific abstract task higher (like the mission planning task “PilotPlanMission” in Fig. 2).
2.2 Plan
Based on the tasks of the task model, the plan is a dynamically generated sequence of mission tasks (, see example mission plan in Fig. 3), which fulfills the mission objective (, see Fig. 3). In our application, the plan depends on the objective of the MUM-T helicopter mission and constraints like terrain, air spaces and others. To plan and schedule this sequence, we use a mixed-initiative approach, where the pilot is in charge to plan the mission for reasons of plan situation awareness, transparency and trust. The mixed-initiative mission planner (MIP) supports the planning process by intervening on own initiative in case of threat avoiding, missing tasks or optimization [12]. The MIP knows different levels of automation (the three levels depicted in Fig. 2), which the associate system can use to simplify planning related problems.
2.3 Activity
Beside the plan, the activity is a key element in our definition of the crew’s mental state. Therefore, our activity determination observes the crew to identify the current activity on-line [13]. It uses different measurement sensors – gaze tracking and manual interaction tracking as well as speech recognition – and the knowledge about observable evidences from the task model. It combines all simultaneously observed evidences by using an evidential reasoning approach derived from Dempster-Shafer theory to assign each evidence to a task from the task model [13, 14]. The activity is the set of elementary tasks (, see Fig. 2), which the crew executes at a given moment. We assume, that if the percentage, which supports the execution of a specific task (blue belief value in Fig. 4) is greater than 0.5, the task is part of the activity. Therefore, in the example in Fig. 4, CommunicateIntern and FlyTransitFriendManual is the current activity of the pilot.
2.4 Demand on Mental Resources
To be able to estimate the demand on mental resources, which is necessary to execute the entire activity, the task model stores the demand on mental resources for each single task as demand vector [15]. After identifying the activity, our resource assessment combines all related demand vectors by using the conflict matrix of Wickens’ Multi-Resource Theory to estimate an overall workload value [11, 15, 16] (Fig. 5). Due to the eight separately stored components of the demand vector for the different resources (visual-spatial, visual-verbal, auditory-spatial, auditory-verbal, cognitive-spatial, cognitive-verbal, response-manual and response-vocal), our method provides information, which resource of the pilot leads to a workload peak. And, as mentioned before, from the activity determination, the associate system knows which task situation leads to that workload peak.
3 Concept and Implementation of Adaptive Assistance
The context-rich representation of mental workload is the basis for adaptive interventions, which support the crew adapted to their mental state for specific task situations.
Figure 6 depicts the functional architecture of the associate system. The process of inferring adaptive interventions consists of three phases. Before the associate system is able to identify the trigger for assistance – neglected task, (predicted) workload peaks and critical events – it preprocesses the mental state (Phase ① in Fig. 6). Thereby, it uses the plan to project the mental state into the future, to
-
know which elementary tasks are necessary to fulfill the plan and
-
identify situations of high workload in the future, which may occur during the plan execution.
In the second step, the associate system identifies trigger for adaptive assistance (“critical states”) and plans the intervention (Phase ② in Fig. 6). To implement the adaptive assistance, the associate system uses the human-dialog interface, which is part of the human-machine interface. In addition, the associate system uses other (cognitive) automation (e.g. the mixed-initiative planner or the adaptive crew sensor interaction) to simplify task situations or adopt tasks via the automation dialog interface (Phase ③ in Fig. 6). For detailed information about the process of identifying trigger, the decision process and stages of intervention see [17].
3.1 Phase 1: Identifying Trigger for Adaptive Assistance
For predicting future mental states and identifying resource conflicting task situations as well as neglected tasks, the associate system refines the plan – which is part of the “input” mental workload (see grey box “Mental Workload” in Fig. 6) – to a detailed plan on elementary task level (which is the lowest level of the hierarchical task model). It uses the task relationships from the task model to identify all elementary tasks, which belong to a specific mission task. Figure 7 shows the mission task EnterHOA, which is the task of entering the helicopter mission area, and its subtasks after the refinement process.
Because the activity determination expresses the activity on the same level of elementary tasks, the associate system is able to match the activity with the detailed plan and check the completed tasks (green task-boxes in Fig. 7). All tasks which are planned, but not executed timely are neglected task and trigger for adaptive assistance (red task-boxes in Fig. 7).
In a second step, the associate system schedules all elementary tasks. Therefore, it uses the execution time and task constraints from the task model. This results in a task timeline, as Fig. 8 shows. The associate system simulates the execution of all future task situations and estimate the needed demand on mental resources (red graph curve in Fig. 8). A situation in which the crew has to perform many tasks in parallel and therefore their demand on mental resources exceeds a threshold is a workload peak (red marked period in Fig. 8) and trigger for adaptive assistance.
The third trigger for adaptive assistance are critical events, like changes of the tactical situation, failures of the helicopter systems or threatened UAV. The associate system cannot predict such events. However, if they occur, it supports the crew by guiding the attention, offering a possible solution, or directly implementing a solution. The following section explains which stage of intervention (which is the “amount of help”) is appropriate.
3.2 Phase 2: Deriving Adaptive Interventions
The process of deriving adaptive interventions aims to find the appropriate level of assistance for a given problem by taking the workload, attention and criticality of the problem into account. Basis for this is the set of rules, mentioned in the introduction. To implement the restrained behavior, the associate system follows the decision process in Fig. 9: First, it traces back each trigger (grey boxes on the left side) to its causal task(s). After comparing this task with the current activity, the associate system infers if the crew works on the solution (Question A in Fig. 9). If the crew is currently not solving the problem, the associate system simulates, whether the crew can handle the current task situation including all other tasks, which are necessary to solve the problem (Question B in Fig. 9). Therefore, it combines the demand vectors of the hypothetical activity using the same method as for the resource assessment, described in Sect. 2.4. If the crew can handle the entire task situation, the associate system guides the attention to the problem (Stage 1 on the right side in Fig. 9), but if not, more assistance is necessary. If the crew is already overtaxed, or any additional task would overtax them, the associate system simplifies the task situation by adapting the human-machine interface (Stage 2a in Fig. 9) or changing the level of automation (Stage 2b in Fig. 9). Only if the problem poses an extreme or high risk (Question C in Fig. 9) and the crew cannot handle it, the associate system is allowed to adopt the problem-solving task(s) (Stage 3 in Fig. 9). We implemented this decision process with the cognitive framework Soar [18]. For detailed information about the decision process and its implementation see [17].
3.3 Phase 3: Stages of Adaptive Interventions
After identifying the appropriate stage of intervention, the associate system implements the different stages as follows. For guiding the attention to a problem, it uses the human-dialog interface. Therefore, it overlays a dialogs box on the multi-function displays (MFD) of the cockpit (see left image in Fig. 10) and highlights all related objects on the MFD. In addition, it can simplify the task situation by adding a “short-cut button” to the dialog box, which implements the proposed solution with one button click (second image from left in Fig. 10). This “short-cut button” is also available via the helicopter control stick. Another possibility to simplify the task situation is to change the level of automation, e.g. to increase the level of automation of the mixed-initiative planner to simplify a planning related problem. For critical situations, the associate system can adopt task(s) like a forgotten landing check (right lower image in Fig. 10), if the human is not capable to handle it. The MFD displays the information on a “history list of adopted tasks” (right upper image in Fig. 10). In addition, a speech synthesizer announces the adopted task (e.g. Set System Configuration Landing) as a human team mate would do in this situation.
4 Evaluation
To evaluate the associate system, we conducted a human-in-the-loop experimental campaign with trained military helicopter pilots in our helicopter mission simulator at the Institute of Flight Systems (see Fig. 11). The purpose of the experiment was, to investigate the interventions of the adaptive associate system during realistic and very complex MUM-T mission scenarios. Therefore, in Sect. 4.2 we analyze two (of many occurred) situations of “near misses”, where the associate system corrected human erroneous behavior during the missions. In addition, in Sect. 4.3 we present and analyze the overall system rating given by the pilots.
4.1 Experimental Design
Our aim at the Institute of Flight Systems is, to evaluate new concepts by testing them as highly integrated systems in very realistic and immersive scenarios. Therefore, the results typically have a very good external validity. But the high complexity of such systems requires domain experts as test subjects (i.e. experienced military helicopter pilots) and moreover an extensive training. We trained our pilots for about two days before conducting experimental trials for another three days. The study population comprises an overall number of seven participants (Age: M 50.4 SD 9.2, Flight Hours: M 3933 SD 1807) grouped into four crews. Every crew consists of a pilot in command (PIC), who leads the mission and guides three UAV for reconnaissance purposes, and a pilot flying (PF), who is responsible to fly the helicopter, communicate and manage the systems of the helicopter. One participant was part of two crews (one time as PIC and another time as PF). Since the associate system supports both crewmember in very different ways, this participant could rate the associate system two times.
The crews performed six different transport helicopter missions (Mission duration in minutes: M 46 SD 10), with mission elements like troops transport, Medical Evacuation (MedEvac) and Combat Search and Rescue (CSAR). The missions are very complex, since all contain many events like suddenly occurring enemies and mission goal changes. These compacted, challenging missions provoke human errors. The intention behind that: In normal workload conditions, highly trained pilots perform good and human errors are very rare. However, to evaluate the associate system, the deliberate provocation of errors is necessary. Therefore, our missions are designed to represent the most stressful parts of helicopter missions. The pilots confirmed that the missions are very stressful, but nevertheless they rated the scenarios as realistic (Scenario is realistic: M 5.6 SD 0.7 on a 7-point Likert scale).
We did not vary and compare configurations of the system, but analyze, when, why and how the associate system intervenes to help the crew and how the crew rates these interventions.
4.2 Adaptive Interventions
One intervention affects the pilot in command (PIC), who is responsible for planning the mission and operating the UAV to ensure a reconnoitered flight route for the helicopter. In this situation, the PIC forgets to assign the route reconnaissance task for the next flight leg of the helicopter route to a UAV (highlighted in magenta in Figs. 12 and 13). In addition, he is involved in an ongoing route reconnaissance task of another UAV (see green frame in Fig. 12, which shows the tasks of the detailed plan for the next minutes). The two parallel tasks PlanMission and ReconRoute, starting at the red line in Fig. 12, lead to a predicted workload peak for the immediate future (red marked area, “now” is at the left border of the red marked area). To relieve the task situation for the PIC, the associate system instructs the mixed-initiative mission planner to increase the level of automation (remember the planning automation levels in Fig. 2) and to propose the next relevant tasks on its own initiative. The planner proposes the forgotten route reconnaissance task (see dialog box in Fig. 13). Due to the urgency and criticality of this task, the associate system automatically accepts the planner proposal. The pilot rated this workload-adaptive intervention as very helpful and appropriate for this risky and time-critical situation.
One situation in the field of system management affects the pilot flying. Because of a mode confusion during the After-Takeoff checklist, the pilot switches on the landing lights erroneously. Within a possibly threatened area, this is an avoidable safety risk. Therefore, the associate system guides the attention to this wrong configured system state (left blue dashed line in Fig. 14). Because the activity determination enables the agent to recognize, that the pilot flying is doing a low-level flight and has therefore less free resources for doing an additional manual task (see estimated overall workload, represented by the green solid graph in Fig. 14), it decides to simplify the task situation by offering the “short-cut” button. Thereby, the pilot can accept the help with a button on his cyclic stick. The pilot accepts the help via this button (brown dashed line, see Fig. 14) and the associate system switches the lights off (right blue dashed line). As soon as the associate system solves the problem (green dashed line), it notifies the pilot by announcing the related task from the task model (i.e. SetLandingLightsOff). The speech announcement is also visible as task block “ListenAssistanceSpeech” in Fig. 14. The pilot flying rated this intervention as very supporting and helpful. In addition, he stated that the explaining text of the intervention (see Fig. 15) makes it easy to understand the problem and how this problem could arise.
These situations illustrate, that the associate system adaptively supports the crew by taking the (projected) mental state and the criticality into account, and that the pilots feel appropriately supported. Beside the investigation of situations of near misses, the pilots assess the overall performance and behavior of the associate system.
4.3 Results
Most of the pilots rate the supporting interventions of the associate system as expedient and helpful (see Fig. 16). The pilots state, that the interventions are justified and the system reacts correctly in dangerous situations. One strength of the associate system is, that it keeps the overview of the whole situation, if the human is focusing on one task and guide the attention to the most urgent task, e.g. new threats, if necessary. Another major benefit is that the interventions save time by simplifying the task situation, which is very valuable during complex missions. These statements, regarding the attention guiding and task simplification, support the rating, that the stage of intervention is mostly situation-adapted and appropriate (see Fig. 16).
Although the system knows the current activity, there are rare cases, where one of the pilots and the associate system start working on the same task simultaneously. This is confusing and the pilot needs to spend time to understand this unnecessary intervention. Therefore, the associate system tries to prevent such cases, but due the delay between action planning and action implementation – which both, human and associate system, have – it was not possible to preclude these cases completely. A better situation awareness of the upcoming tasks and the human-agent task sharing, i.e. by pre-announcing of the task sharing, could solve this issue. [19] proposes a possible solution for this kind of pre-announcement.
Most of the pilots appraise the behavior of the associate system as transparent and comprehensible (see Fig. 17). Due to the fact that the associate system intervenes workload-adaptive and there is no pre-announcement of interventions, the predictability is lower. The pilots think, that more training may increase the predictability, but workload-adaptive systems are low predictable at all. As mentioned above, pre-announcing the next tasks and possible interventions increases the situation awareness of the crew [19].
The pilots state, that the associate system is neither hasty nor restrained, but in this case the neutral position is a positive rating, because they state, that they interpret “restrained” as negative like “too restrained”.
5 Conclusion
We presented a concept of a workload-adaptive associate system, which supports the crew of a manned helicopter during military MUM-T helicopter missions. The process of inferring workload-adaptive interventions relies on a context-rich operationalization of mental workload based on tasks. With this definition of mental workload, the associate system is able to project future task situations to identify all pilot tasks which are necessary to reach the mission goal. In addition, it predicts situations of high workload in the future and eases them proactively by using higher levels of automation. After identifying trigger for supporting interventions, the associate system supports the crew by guiding the attention, offering a possible solution or directly implementing a solution. Thereby, it behaves restrainedly and helps only if necessary and as less as possible to keep the human in the loop and to prevent typical pitfalls of highly automated systems, like out-of-the-loop problems, complacency and automation bias.
Our pilot-in-the-loop experiments show, that the concept of supporting cognitive tasks workload-adaptively recovers human errors like neglected tasks and relieves time critical task situations before they occur. The pilots rate the interventions of the associate system as helpful and expedient. However, future improvements of the system should address the transparency e.g. by pre-announcing the human-agent task sharing.
We implemented this concept in the domain of military aviation. But the concept is not limited to this domain. It is transferable to other domains, where a human operator collaborates with highly automated systems, e.g. civil aviation, highly automated driving or power plant management.
References
Young, M.S., Brookhuis, K.A., Wickens, C.D., Hancock, P.A.: State of science: mental workload in ergonomics. Ergonomics 58, 1–17 (2015)
Wiener, E.L.: Human Factors of Advanced Technology (Glass Cockpit) Transport Aircraft. NASA CR 177528, Ames Research Center, Moffett Field (1989)
Onken, R., Schulte, A.: System-Ergonomic Design of Cognitive Automation: Dual-Mode Cognitive Design of Vehicle Guidance and Control Work Systems. SCI, vol. 235. Springer, Heidelberg (2010). https://doi.org/10.1007/978-3-642-03135-9
Schulte, A., Donath, D., Lange, D.S.: Design patterns for human-cognitive agent teaming. In: Harris, D. (ed.) EPCE 2016. LNCS (LNAI), vol. 9736, pp. 231–243. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-40030-3_24
Rudnick, G., Schulte, A.: Implementation of a responsive human automation interaction concept for task-based-guidance systems. In: Harris, D. (ed.) EPCE 2017. LNCS (LNAI), vol. 10275, pp. 394–405. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-58472-0_30
Uhrmann, J., Schulte, A.: Concept, design and evaluation of cognitive task-based UAV guidance. Int. J. Adv. Intell. Syst. 5, 145–158 (2012)
Endsley, M.R., Kiris, E.O.: The out-of-the-loop performance problem and level of control in automation. Hum. Factors 37, 381–394 (1995)
Parasuraman, R., Manzey, D.H.: Complacency and bias in human use of automation: an attentional integration. Hum. Factors 52, 381–410 (2010)
Schulte, A., Donath, D., Honecker, F.: Human-system interaction analysis for military pilot activity and mental workload determination. In: Proceedings of 2015 IEEE International Conference on Systems, Man, and Cybernetics, SMC 2015, pp. 1375–1380 (2015)
Donath, D., Schulte, A.: Behavior based task and high workload determination of pilots guiding multiple UAVs. Procedia Manuf. 3, 990–997 (2015)
Honecker, F., Brand, Y., Schulte, A.: A task-centered approach for workload-adaptive pilot associate systems. In: Proceedings of the 32nd Conference of the European Association for Aviation Psychology – Thinking High AND Low: Cognition and Decision Making in Aviation, Cascais (2016)
Schmitt, F., Roth, G., Schulte, A.: Design and evaluation of a mixed-initiative planner for multi-vehicle missions. In: Harris, D. (ed.) EPCE 2017. LNCS (LNAI), vol. 10276, pp. 375–392. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-58475-1_28
Honecker, F., Schulte, A.: Automated online determination of pilot activity under uncertainty by using evidential reasoning. In: Harris, D. (ed.) EPCE 2017. LNCS (LNAI), vol. 10276, pp. 231–250. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-58475-1_18
Shafer, G.: A Mathematical Theory of Evidence. Princeton University Press, Princeton (1976)
Wickens, C.D.: Multiple resources and performance prediction. Theor. Issues Ergon. Sci. 3, 159–177 (2002)
Maiwald, F., Schulte, A.: Enhancing military helicopter pilot assistant system through resource adaptive dialogue management. In: Vidulich, M.A., Tsang, P.S., Flach, J.M. (eds.) Advances in Aviation Psychology. Ashgate Studies in Human Factors and Flight Operations (2014)
Brand, Y., Schulte, A.: Model-based prediction of workload for adaptive associate systems. In: Proceedings of 2017 IEEE International Conference on Systems, Man, and Cybernetics, SMC 2017, pp. 1722–1727 (2017)
John, E.: Laird: The Soar Cognitive Architecture. The MIT Press, Cambridge (2012)
Brand, Y., Ebersoldt, M., Barber, D., Chen, J.Y.C., Schulte, A.: Design and experimental validation of transparent behavior for a workload-adaptive cognitive agent. In: Karwowski, W., Ahram, T. (eds.) IHSI 2018. AISC, vol. 722, pp. 173–179. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-73888-8_28
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2018 Springer International Publishing AG, part of Springer Nature
About this paper
Cite this paper
Brand, Y., Schulte, A. (2018). Design and Evaluation of a Workload-Adaptive Associate System for Cockpit Crews. In: Harris, D. (eds) Engineering Psychology and Cognitive Ergonomics. EPCE 2018. Lecture Notes in Computer Science(), vol 10906. Springer, Cham. https://doi.org/10.1007/978-3-319-91122-9_1
Download citation
DOI: https://doi.org/10.1007/978-3-319-91122-9_1
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-91121-2
Online ISBN: 978-3-319-91122-9
eBook Packages: Computer ScienceComputer Science (R0)