Keywords

These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

1 Introduction

Accomplishing a dangerous and complicated mission, while minimizing risks and costs, is a primary goal in many military situations. Because Unmanned Aerial Vehicles (UAVs) can provide information and/or munitions deployment without exposing a human to harm, they naturally support such a goal and are becoming ubiquitous. For example, many military intelligence, search, and reconnaissance (ISR) missions can benefit from using Unmanned Aerial Vehicles (UAVs) and Unmanned Ground Vehicles (UGVs). Civilian applications can also benefit from using UAVs including wilderness search (Goodrich et al. 2008), disaster damage assessment (Murphy et al. 2008), and infrastructure inspection.

With reduced radar signatures, increased endurance, and the removal of humans from immediate threat, unmanned (also known as uninhabited) aerial vehicles have become indispensable assets to militarized forces around the world, as proven by the extensive use of the Shadow and Predator in recent conflicts. Despite the absence of a crew onboard in any of these UAVs, human operators are still needed for supervisory control. Although less mature, civilian uses of UAVs also require supervisory control for complex operations like border patrol, agriculture monitoring, and disaster response.

UAVs require human guidance to varying degrees and often through several operators, which is what essentially defines a UAS (Unmanned Aerial System). For example, the Predator and Shadow each require a crew of two to be fully operational. However, it is often desirable to design systems such that the current many-to-one ratio of operators to vehicles can be inverted.

In order to develop UAV technologies to effectively support single operator, multiple UAV control, the focus should be on the creation of an organization of human operators, software agents, and UAVs such that mission effectiveness is maximized at a minimum cost. The remainder of this chapter examines the role of autonomy and user interface design on UAS design, with an emphasis on not only placing UAVs at the right locations at the right time but also on managing mission- critical payloads of those UAVs. The chapter begins by identifying key attributes of the supervisory control problem for UASs. Since military UAVs represent state of the art, the first half of this chapter use military systems to provide context.

2 Supervisory Control of Multiple UASs

The move from platform-centric warfare to network-centric warfare (NCW) represents a shift in the role of humans both in mission planning and actual operation. As has already been evidenced in the development of fly-by-wire, highly automated aircraft and missile systems (such as Tomahawk and Patriot), military operators are less in direct manual control of systems but more involved in the higher levels of planning and decision making and remote operations.

This shift in control from lower-level skill-based behaviors to higher-level knowledge-based behaviors is known as human supervisory control (HSC). HSC is the process by which a human operator intermittently interacts with a computer, receiving feedback from and providing commands to a controlled process or task environment, which is connected to that computer (Fig. 99.1) (Sheridan 1992). All UAVs in the U.S. Department of Defense inventory operate at some level of supervisory control as depicted in Fig. 99.1.

Fig. 99.1
figure 1

Human supervisory control of a UAV

Full size image

Human supervisory control in UAV operation is hierarchical, as represented in Fig. 99.2. The innermost loop of Fig. 99.2 represents the basic guidance and motion control, which is the most critical loop that must obey physical laws of nature such as aerodynamic constraints for UAVs. In this loop, operator actions are focused only on the short term and local control (keeping the aircraft in stable flight), and generally human control in this loop requires skill-based behaviors that rely on automaticity (Rasmussen 1983).

Fig. 99.2
figure 2

Notional supervisory control loops for unmanned aerial vehicles

Full size image

The second loop in Fig. 99.2, the navigation loop, represents the actions that some agent, whether human or computer-driven, must execute to meet mission constraints such as routes to waypoints, time on targets, and avoidance of threat areas and no-fly zones. The outermost loop represents the highest levels of control, that of mission and payload management. In this loop, sensors must be monitored and decisions made based on the incoming information to meet overall mission requirements. In this loop, decisions require knowledge-based reasoning that includes judgment, experience, and abstract reasoning that in general cannot be performed by automation.

Finally, the system health and status monitoring loop in Fig. 99.2 represents the continual supervision that must occur, either by a human or automation or both, to ensure that all systems are operating within normal limits. The control loop line is dashed as it represents a highly intermittent loop in terms of the human, i.e., if the human is engaged in another task, with the highest priority given to the innermost loop, health and status monitoring becomes a distant, secondary task.

From the human-in-the-loop perspective, if the inner loops fail, then the higher (outer) loops will also fail. The dependency of higher loop control on the successful control of the lower loops drives human limitations in control of a single and especially multiple UAVs. If humans must interact in the guidance and motion control loop (hand fly a UAV), the cost is high because this effort requires significant cognitive resources. What little spare mental capacity is available must be divided between the navigation and mission management control loops. Violations of the priority scheme represented in Fig. 99.2 have led to serious problems exemplified by numerous Predator crashes. When operators become cognitively saturated or do not correctly allocate their cognitive resources to the appropriate control loops in the correct priorities, they violate the control loop constraints, potentially causing catastrophic failure.

While Fig. 99.2 demonstrates supervisory control at the single vehicle level, Fig. 99.3 represents a notional system architecture that will be required for single operator control of multiple UASs. In order to achieve this futuristic system, operators will need to interact with an overall mission and payload manager while relegating routine navigation and motion control tasks to automation. The challenge in achieving effective management of multiple UAVs in the future is not only to determine if automation can be used to reduce workload but how and to what degree in each of the control loops in Figs. 99.2 and 99.3, as well as what kinds of decision support will be needed by operators given the high-workload environment.

Fig. 99.3
figure 3

Notional supervisory control loops for single operator, multiple UAV control

Full size image

One important consideration in developing the architecture depicted in Fig. 99.3 is the degree of centralization/decentralization of both the UAVs and the human. For example, in current single operator, multiple UAV operational paradigms, the operator independently controls all the vehicles in a serial fashion, and the vehicles do not work collaboratively to solve problems. Such an approach necessarily limits the number of vehicles the single operator can control since effectively the incoming tasks per vehicle represent a queuing network with a single server (the human) who has limited capacity (Cummings et al. 2007).

On the other end of the control spectrum are decentralized networks of UAVS with distributed autonomy across the vehicles such that the UAVs can locally reason independent of the network, while sharing information between the vehicles and a human supervisor at opportunistic times. Such control paradigms are task- based for the operator as opposed to vehicle-based for the centralized case (Clare and Cummings 2011). These architectures with significant advanced and increased autonomy are what will enable significantly more UAVs to be controlled by one person, often referenced as swarm control.

3 Single Operator Management of Multiple UASs

Increasing the autonomy across the three control loops discussed previously (Figs. 99.2 and 99.3) is the critical architecture component for allowing one or a small team of operators to effectively control multiple UAVs. By increasing UAS autonomy, operator workload will theoretically be reduced as it could reduce the number of tasks for the operator, and it should reduce the level of interaction even at the highest levels of control in Figs. 99.2 and 99.3. For example, those UAVs that are flown in an autopilot mode relieve the operator from the manual flying tasks that require significant cognitive resources. This frees the operator to perform other critical tasks like mission planning and imagery analysis.

Higher levels of automation across the control loops depicted in Figs. 99.2 and 99.3 will be critical in achieving the single operator, multiple UAV control vision; but how, when, where, and what level of automation should be introduced are still difficult problems. For example, delegating navigation is necessary for organizations where a human must manage multiple UAVs. When the human must sequentially manage payloads, such delegation becomes even more critical.

While workload mitigation can occur through increasing automation, it can inadvertently cause higher workload as well as loss of situational awareness, complacency, and skill degradation (Parasuraman et al. 2000). For example, some UAV researchers have found that intermediate levels of management by consent (automation as an assistant to the operator) are preferable to manual or more fully automated control (Ruff et al. 2002). However, management by consent means that the number of tasks could be high since operators must always be in the loop, potentially saturating operators, especially in the multiple UAV domain. Moreover, as has been shown in multiple UAV control research, operator performance can dramatically decrease under management by consent given increasing workload and various decision aids (Cummings and Guerlain 2007; Cummings and Mitchell 2008).

Given that an increasing number of tasks will have to be automated to achieve single operator control of multiple UAVs, particularly for swarm management, the question then becomes what to allocate to automation. Previous research has demonstrated that in the scheduling and execution of high-level tasks, of multiple UAVs, management by exception can improve operator performance (Cummings and Mitchell 2008). Management by exception occurs when automation decides to take an action based on some set of predetermined criteria and only gives operators a chance to veto the automation’s decision.

While this control scheme can be effective in time-critical, high-risk domains like shutting down a near-critical nuclear reactor, in intentional, highly uncertain domains like command and control, it can be dangerous. Under this control scheme, operators are more likely to exhibit automation bias, a decision bias that occurs when operators become overreliant on the automation and do not check to ensure automated recommendations are correct (Mosier and Skitka 1996).

Automation bias was operationally seen in the 2004 war in Iraq when the U.S. Army’s Patriot missile system, operating in a management-by-exception mode, engaged in fratricide, shooting down a British Tornado and an American F/A-18, killing three. The system was designed to operate under management by exception, and operators were given approximately 15 s to veto a computer solution. Unfortunately, the displays were confusing and often incorrect, and operators admittedly lacked training in the highly complex system (32nd Army 2003). Given the laboratory evidence that given an unreliable system, humans are still likely to approve computer-generated recommendations (Cummings 2004), it is not surprising that under the added stress of combat, Patriot operators did not veto the computer’s solution. Automation bias is a significant concern for command and control systems, so it will be critical to ensure that when higher levels of automation are used, especially at the management-by-exception level, this effect is minimized.

3.1 A Meta-analysis of Previous Multiple UAV Studies

There have been numerous research studies published that have examined various aspects of multiple UAV control. A meta-analysis was performed across those studies that focused either explicitly on operator capacity or human supervisory control aspects of multiple vehicle control in order to determine any significant trends or lessons learned, particularly in regard to levels of automation and the control loops discussed above.

3.1.1 Previous Studies

One solution investigated by Dixon et al., to reduce UAV operator workload in the control of one or more small- to medium-sized UAVs, such as the Shadow, consisted of adding auditory and automation aids to support the potential single operator (Dixon et al. 2004). They showed that a single operator could theoretically fully control a single UAV both in terms of navigation and payload if appropriate offloading strategies were provided. For example, aural alerts improved performance in the tasks related to the alerts, but not others.

Conversely, it was also shown that adding automation benefited both tasks related to automation (e.g., navigation, path planning, or target recognition) and non-related tasks. However, their results demonstrate that human operators may be limited in their ability to control multiple vehicles that need navigation and payload assistance, especially with unreliable automation. These results are concordant with the single-channel theory, stating that humans alone cannot perform high-speed tasks concurrently (Welford 1952; Broadbent 1958). However, Dixon et al. propose that reliable automation could allow a single operator to fully control two UAVs in a centralized setting.

Reliability and the related component of trust is a significant issue in the control of multiple uninhabited vehicles. Ruff et al. (2002) found that if system reliability decreased in the control of multiple UAVs, trust declined with increasing numbers of vehicles but improved when the human was actively involved in planning and executing decisions. These results are similar to those found by Dixon et al. (2004) in that systems that cause distrust reduce operator capacity.

Ruff et al. (2002, 2004) determined that higher levels of automation actually degraded performance when operators attempted to control up to four UAVs. Results showed that in centralized settings, management by consent (in which a human must approve an automated solution before execution) was superior to management by exception (where the automation gives the operator a period of time to reject the solution). Management by consent appeared to provide the best situation awareness ratings, the best performance scores, and the most trust for controlling up to four UAVs.

Dunlap (2006) also subscribe to management by consent in their development of a distributed architecture to control multiple unmanned combat aerial vehicles (UCAVs). In this system, a UCAV plan is proposed by the automation, and the operator can either accept or reject the plan or submit an alternative. This recommendation can include both target assignments and routing. While they tested four, six, and eight UCAVs with increasing levels of environmental complexity, their final design limited the UCAV loadout at four. In one experiment, they noted that automation bias was a prevalent problem, stating the operators “had become attenuated to automatically accepting the usually correct proposals from the UCAVs,” which resulted in an increased kill rate for no-targets under the higher levels of automation.

In terms of actually predicting how many UAVs a single operator can control, there are only a few studies that examine this question, and they all focus on the centralized architecture. Cummings and Guerlain (2007) showed that operators could experimentally control up to 12 Tactical Tomahawk Land Attack Missiles (TLAM) given significant missile autonomy. Operators only had to interact in the mission management loop, and all other loops were highly automated. In a UCAV setting, Cummings and Mitchell (2008) demonstrated that the number of UCAVs that a single operator can control is not just a function of the level of decision support automation but also the operational tempo and demands. Operators under low workload performed well regardless of the level of decision support, but under high workload, performance degraded. When considering operational and workload demands for a suppression of enemy air defenses mission, operator capacity was estimated at five UCAVs.

In a demonstration of the capabilities of a single operator attempting to control multiple Wide Area Search Munitions (WASMs), given high levels of autonomy across all control loops in Fig. 99.3 with only higher-order goal tasking for mission management, Lewis et al. (2006) posit that an operator can effectively control up to eight WASMs. The assumption is that the automation embedded in the vehicle’s coordinates, without human intervention, specific tasks such as target detection and choice of the most appropriate member to execute the mission, which are capabilities that are not yet operational. The WASM study is similar to the Tactical Tomahawk study in that all flight-control and navigation functions are allocated to the automation alone, and the human intervenes for very high-level goal management.

Thus, there have been a cross section of studies that have examined operator performance and capacity in the control of multiple UAVs; however, it is not clear how many meaningful comparisons can be made across the different domains primarily because of two parameters: (1) what constitutes control and (2) what level of automation was used to aid the operators. In order to more directly compare these studies, the following section will discuss the scale on which comparisons can be made.

3.1.2 Level of Automation Trends

In this meta-analysis, the maximum number of UAVs that an operator effectively controlled in each study was extracted. It should be noted that in all of these reported studies, the control occurred in simulated test beds of medium fidelity and that all of these studies represent centralized control architectures. Approximate levels of automation (LOAs) across the control loop(s) from Fig. 99.3 were subjectively identified. While numerous levels and scales of automation and autonomy have been proposed (Parasuraman et al. 2000; Endsley and Kaber 1999; Wickens et al. 1998; Endsley 1995), the ten-level scale originally proposed by Sheridan and Verplank (1978) (SV-LOA) was chosen, as this is a commonly referenced taxonomy. Some of the categories in Table 99.1 were combined to reflect functional similarities. For example, levels 7–10 were combined since the human can take no action. Recognizing that different stages of information processing can be supported by automation (Parasuraman et al. 2000), the decision and action selection stage is represented in the assessment.

Table 99.1 Levels of automation
Full size table

In Table 99.2, which presents a summary of these findings, the numbers of UAVs potentially controllable by a single human operator are referenced along with estimated levels of automation for each of the three control loops (MC – motion control inner loop, N – navigation, MM – mission management outer loop). It should be emphasized that the LOAs selected were approximate since they were both subject to interpretation and assigned post hoc from studies not originally intended to answer the research question. In addition, in many simulations, the LOA was not fixed so the range of LOAs was identified in these cases. In this comparison, an air traffic control (ATC) study was also included since it embodies many of the same principles of human supervisory control that are relevant to the control of multiple UAVs (Hilburn et al. 1997). Since air traffic controllers’ primary focus is safe navigation of aircraft, there is no associated mission management control loop.

Table 99.2 Multiple UAV study comparison
Full size table

Table 99.2 reveals interesting trends. Without explicitly discussing it in their respective studies, all researchers automated the inner motion control loop as depicted in Figs. 99.2 and 99.3. Thus, some form of autopilot was needed to relieve operator workload and free cognitive resources for higher loop control. To achieve the goal of one person controlling many UAVs, operators should only monitor the piloting/maneuvering of the vehicle, not do it themselves. However, this is a cultural problem more than it is a technological problem, as this technology is available today in all UAVs, but resisted in some communities, i.e., some organizations still insist that a human “fly” the vehicle instead of commanding it using various flight profiles.

Figure 99.4a demonstrates a general increasing trend in the number of vehicles an operator can control as a function of increasing automation in the navigation control loop of Fig. 99.2. Thus, given increasing navigation support and a fully autonomous flight-control system, operators can handle more UAVs when they do not have to attend to local and even global navigation concerns. The highest operator capacity was seen in the Tomahawk missile and WASM domains because, as they are oneway UAVs traveling at high speeds, there is little time for human intervention.

Fig. 99.4
figure 4

Max number of UAVs vs. LOAs for the (a) navigation loop and (b) mission management loop

Full size image

When examining the mission and payload management control loop, Fig. 99.4b demonstrates the ability of operators to control more vehicles as they are provided with increasing automated decision support. It is interesting to note that given some automated navigation assistance and management-by-consent automation in the mission management loop, there is a convergence of operator capacity at 4–5 vehicles per operator. The next remarkable increase in operator capacity in centralized control settings (8–12 vehicles) is not seen until management by exception is introduced in the mission management or navigation loops. These increased levels of automation will be critical for increased operator capacity since, as previously discussed, if operators are required to attend to local navigation functions, they simply do not have the cognitive resources to successfully attend to all of the tasks in the mission and payload management loop.

As mentioned previously, to truly allow one operator the ability to control significantly more vehicles than the studies represented in Table 99.2, the control architecture must shift from centralized, vehicle-based to decentralized, task-based. The decentralized approach leaves the human just in the knowledge-based mission management loop and offloading the lower two loops in Figs. 99.2 and 99.3 to the automation. Thus, the limiting factor is not the number of vehicles an operator is controlling, but rather the number of tasks generated from each vehicle. The higher the autonomy and reasoning across a network of UAVs, the fewer tasks the team of vehicles generates.

In recent experiments, it was demonstrated that with a task-based, decentralized control system, doubling operator task load led to an operator workload increase of only ~50 %. Also in these studies, operator workload never approached unmanageable thresholds despite the doubling of task load. Thus, a task-based, decentralized control system may be robust to high task load situations, in effect allowing an operator to control theoretically limitless numbers of UAVs as long as they are generating, as a group, a manageable set of tasks (Clare and Cummings 2011).

4 The Knowledge-Based Loop: Managing Payloads

It can be difficult to operate a remote UAS when an operator is required to spend most of his or her time in the mission and payload management outer loop in Fig. 99.2. Such tasks are typically knowledge-based and require judgment and reasoning, not easily codified in an algorithm. One such task that causes high workload in today’s UAV operations is the surveillance task such as watching video feed. The perceptual problem associated with this task is the “limited angular view associated with many remote vision platforms creates a sense of trying to understand the environment through what remote observers often call a ‘soda straw’” (Woods et al. 2004).

Conventional, workstation-based UAS interfaces do not integrate multiple sources of information into a coherent representation. Many interfaces include a map, UAS platform operations, and flight-control parameters in separate locations on the screen. Most systems deployed today show the footprint of the camera, but this is often the extent of the integration. For operators using conventional displays, the cognitive workload associated with just getting enough awareness to control flight and navigation (i.e., the inner loops in Fig. 99.2) can be high enough that it is difficult to interpret imagery, plan strategies, and do other higher-level tasks that are relevant for the mission. This problem is frequently solved by adding more humans to the team who are responsible for mission-level issues (Burke and Murphy 2004) although, as noted in Sect. 99.3.1.1, offloading tasks to automation can theoretically enable an operator to manage both navigation and payload management.

A better approach to this problem is to represent real sensor information in an ecological way (Vicente et al. 1995). In the robotic domain, one important and useful ecological technique is the use of mixed-reality interfaces. Such interfaces combine real data with virtual elements and range from augmented reality to augmented vir- tuality interfaces (Milgram and Kishino 1994). An example of such interfaces, taken from Cooper (2007), is shown in Fig. 99.5. In this interface, satellite imagery and terrain maps are fused to create a virtual representation of the world. The UAV is projected into this virtual representation, and waypoints, compass, and other flight information are displayed within this virtual representation. Perhaps most importantly, video information is projected into the virtual world, giving context and stability for information detected in the camera. Most systems currently deployed project a sensor footprint (though perhaps without video) in this way, and most also provide separate video feeds on larger, higher-resolution displays to support video analysis.

Because a mixed-reality display is not constrained to a single perspective (e.g., pilot’s perspective and payload operator’s perspective), there are a number of different ways for information to be presented in a mixed-reality display. This chapter now briefly describes the chase perspective and the map-based perspective. Each technique is appropriate for certain types of tasks and modes of interaction between the human and a UAV.

4.1 Reactive Navigation and the Chase Perspective

The chase perspective is illustrated in Fig. 99.5. This perspective presents sensor information in a way that supports locomotion and is a typical representation used in racing games because it allows the direct perception of the relationship between the vehicle and the afforded directions of vehicle movement. An additional example of the chase perspective is shown in Fig. 99.6 (Quigley et al. 2004). In this display, a virtual UAV is included in the display to represent the pose of the UAV relative to the ground. This virtual UAV is overlayed on the video image received from the UAV and allows the operator to directly perceive the attitude of the aircraft with respect to the ground. (In the figure, two virtual UAVs are shown; one indicates the actual pose of the UAV as received from telemetry, and the other indicates the commanded pose of the UAV.) The chase perspective shown in Fig. 99.6 is taken from an interface that runs on a 7-in. or smaller display.

Fig. 99.5
figure 5

A mixed-reality display fusing control, navigation, and sensor information

Full size image
Fig. 99.6
figure 6

Integrating a chase perspective with flight control

Full size image

Note that the chase perspective for the UAV is earth-centered rather than pilot- centered; in the human factors literature, these differences of perspective are known as outside in and inside out, respectively. When the operator is on the ground, banking right is not accompanied by a pilot-perceived change in the earth’s horizon nor is it accompanied by other vestibular cues. Since the operator is on the ground, the chase perspective adopts a ground-based perspective wherein a bank command is depicted by having the virtual UAV dip its wing in the commanded direction, thus matching the model of the aircraft to the mental model of the remote operator.

It is possible to take this ground-centered perspective a step further. Although the technology discussed in the next few paragraphs is a bit outdated, it is still relevant since it can be directly applied to controlling UAVs using tablet-based displays when those tablets are equipped with inertial sensing, as in “flying by iPad,” especially given the increasing use of smart phone and tablet devices in control applications for UAVs (Jackson et al. 2011; Cooper and Goodrich 2005). Since a fixed-mount camera, typical of cameras mounted on micro aerial vehicles, rotates when the UAV banks, the operator must switch from a ground perspective to the UAV perspective in order to interpret video. This switch might be a cause of cognitive workload because the ground-based operator must interpret rotations in the video caused by a banking UAV.

An interface that makes both video and bank angle have a ground-centered reference frame is shown in Fig. 99.6. This interface is built for a control device called the PhyCon (for Physical Icon) (Quigley et al. 2004). Rather than using a handheld computer to issue commands to the UAV, a physical model of the UAV is used. When the operator banks the model, the actual UAV also banks. The PhyCon is almost certainly not viable for practical systems, but it does reinforce the idea that the inertial sensors from modern smart phones and tablets can be used for human control of UAVs. Although it is somewhat difficult to see in the figure, the pose of the aircraft is projected onto the video from a ground-centered reference frame using the chase perspective. This is a type of mixed-reality interface (Milgram and Kishino 1994). (In practice, there are actually two virtual depictions of the UAV. The first virtual UAV is depicted using the actual telemetry from the UAV. The second virtual UAV is depicted using the commanded pose from the PhyCon. Having both of these projected into the augmented reality display allows the operator to see that the actual UAV is responding appropriately to the commanded pose.)

The video feed is digitized and displayed on a computer (in this case, a laptop which presents the video through an eyeglass-mounted display). Prior to presenting the imagery, the telemetry from the UAV is used to rotate the image so that the horizon stays approximately level. This is depicted in Fig. 99.7. Rotating the image so that the horizon stays level means that both the video and the UAV attitude are depicted in a ground-relative reference frame.

Fig. 99.7
figure 7

North-up perspective

Full size image

An important lesson from Cooper (2007) is that chase perspectives are best suited to so-called reactive searchers. In a reactive search, an operator is not systematically navigating through a search pattern or a series of waypoints. Instead, the operator is using real-time video information to directly control the UAV so as to track a moving target, follow a trail or terrain feature, or inspect a pipeline or convoy. Chase-based perspectives are useful in such domains because they create a tight coupling between flight dynamics and video information. Figure 99.6 enables this tight connection by changing the orientation of the UAV in response to video information, and Fig. 99.5 enables this tight connection by allowing the operator to control a controllable, floating waypoint (labeled the “carrot”) that the operator can reactively control.

4.2 Systematic Navigation and the Map-Based Perspective

The chase perspective primarily supports reactive control. When it is necessary to operate at the level of navigation or planning, it is often useful to have a map of some sort. For example, a common UAV mission is to search an area for a missing person or a target. Issuing commands for these searches and depicting the progress of these searches may be easier for the operator if a map-centered reference frame is used (Plumlee and Ware 2003), and in a direct comparison of a chase perspective and map-based perspective it was found that the map-based perspective was more useful for systematic search (Cooper 2007). The task is to present map information in a useful way and then to integrate the video into the map using this map-centered perspective.

Figure 99.7 depicts a mixed-reality display that integrates a virtual map with video from a robot (Cooper 2007). The video is depicted in this virtual world in such a way that video, map, and UAV pose information are simultaneously visible. There are a number of desirable features of such an interface, including (a) the ability to determine what has been searched and what needs to be searched, (b) the ability to perceive how the robot is oriented with respect to landmarks in the world, most importantly, a fixed north-up orientation, and (c) the ability to augment map information with icons or other semantic labels.

The ability to determine what has been searched and what needs to be searched has been shown to be very useful in wilderness search and rescue (Morse et al. 2010) as has the ability to keep a stabilized camera view in a north-up orientation (Morse et al. 2008). Map-centered displays, with either a north-up or linked orientation (Plumlee and Ware 2003), can be augmented with mission-specific symbology.

It is important to note that map-based interfaces have been used to construct augmented reality displays in aviation. These displays, which may be either heads- up or head-down and which may be retrofitted to older aircraft (Prinzel et al. 2004), are referred to as synthetic vision displays (Schnell et al. 2004). Several human factors studies have been conducted, many showing that there is an increase in navigation-related situation awareness with negligible loss in aviation-related situation awareness (Alexander and Wickens 2005), presumably because a greater field of view and subsequent sense of realism can be obtained with such displays.

5 Multi-operator, Multi-UAV Supervisory Control and Payload Management

The previous sections explored how autonomy and the user interface affected the state of the art in human supervisory control of single and multiple UASs. In this section, emerging trends are summarized that will likely impact the next generation of UASs.

The first emerging area follows immediately from the increases in complexity that occur when a human is required to manage multiple UASs. Decision support systems (DSSs) will be needed (a) to help humans manage attention between different vehicles, sensors, and mission parameters and (b) to help humans allocate and schedule resources so that assets are used efficiently. In single operator, multiple UAV control as depicted in Fig. 99.3, DSSs will be needed to assist primarily in the mission payload management task, as well as possibly lower-level tasking such as navigation tasks.

Given that the large majority of military UAVs, as well as likely future civilian UAV applications, are surveillance missions, an important question is whether a single operator can manage multiple video feeds that will occur in such multi- UAV surveillance missions. Such missions are only possible with high degrees of automation in the lower-level control loops in Fig. 99.3, as well as also in the mission management loop. Such automation could take the form of automated target detection, which is currently a very active area of basic research but with no major breakthroughs as of yet for airborne systems.

Another form of automated assistance in such settings could take the form of scheduling tools that prompt operators when to switch their attention across different UAVs under their control. Such DSSs rely on developing operator models that also consider the operational context of the vehicles in order to make recommendations to operators for not only when to switch but to which UAV or task that needs attention. While such attention management is key to efficient single operator–multiple UAV control, developing operator models that are robust in the inherently stochastic world of command and control is difficult (i.e., Bertuccelli et al. (2010)), so this is also an open area of current research.

The second emerging area is one of organization, particularly organizational issues that arise when multiple humans share management responsibilities for multiple UASs. The philosophy behind work in this area is that future operational models of human-UAS teams will have N humans manage M vehicles where N is much less than M. One key organizational issue is whether specific humans are assigned to fixed UASs or whether the humans can interact with any UAS according to mission demands (Mekdeci and Cummings 2009; Lewis et al. 2010). A second key organizational issue is how humans can provide mutual support to each other so that resulting behavior exploits the advantages of highly effective teams without invoking the negative behaviors (such a social loafing) that can arise when responsibility is distributed across an organization (Whetten et al. 2010).

The third emerging theme is the need to form a common operational picture when multiple sensor payloads are distributed across a wide geographical area (Sirak 2007; Feitshans et al. 2008; Velagapudi et al. 2008). This theme builds on current operational models where specific humans are responsible for controlling unmanned air and ground platforms but where other humans can benefit from information being returned from those platforms. This is particularly important in military applications where micro-UAVs might be used by a small squad of soldiers and large UAVs might be used as a strategic asset. Although different groups of humans might be responsible for controlling operational, tactical, and strategic UAVs, each group of humans might benefit in the role of so-called information consumers (Goodrich and Schultz 2007) of a common operational picture.

The fourth emerging theme is an extension of the third theme and seeks to automatically allocate UAV resources by providing information to and requesting information about a particular mission. Although this has been explored in other contexts, it is easily illustrated in wilderness search and rescue where the path of a sensor footprint may be automatically planned as the result of (a) search experts providing information about the likely influences on missing person movement, (b) incident commanders providing information about the relative difficulty of air and ground resources detecting a missing person, and (c) the UAV technical search specialist providing information about where the UAV should begin and end a search but allowing the UAV to optimally search the area in between (Lin et al. 2010).