When dealing with crisis management situations, humans are typically confronted with severe threats, uncertainty, time pressure, dynamic situations, and ad hoc teams. These circumstances put a great strain on the human information processing system. Researchers have found that when evaluating crisis management exercises or real crises, humans are found to be lacking in communication and coordination (e.g., Helsloot 2005; Quarantelli 1999). Crisis managers frequently fail to pass on relevant information to others and they also frequently fail to coordinate with others (Hale et al. 2005). These failures may be a result of the time pressure and uncertainty involved, as well as the fact that teams work in an ad hoc, stovepiped manner, the latter being the result of different institutions with different cultures being put together for the first time (Militello et al. 2007).

To understand and support the human in the crisis loop, we need to understand a process that flows from training to decision making and that produces tools to support each of these activities, the design of which is informed by our understanding of the process. In this paper, we will look more closely at how to support humans in the crisis loop based on a thorough understanding of the macrocognitive functions that have to be fulfilled, such as naturalistic decision making, sensemaking, coordination and communication, and planning and adaptation (Schraagen et al. 2008a). The objectives of the paper are to review work to date—along with selected illustrative examples—in order to identify possible directions for future research.

This paper is organized as follows. Section 1 discusses typical characteristics of disaster crisis management and resulting challenges put upon the crisis management organization. Section 2 goes into more detail about how the challenges may be met by performing typical macrocognitive functions. Section 3 presents a method for assessing one typical macrocognitive function, shared sensemaking. Section 4 presents a tool for supporting decision-making processes in crisis management, in particular the need to avoid tunnel vision. Section 5 discusses various methods and tools for improving the training of crisis management teams. Section 6 presents recent observations we carried out during a field trial of networked crisis management teams. As networked-centric operations is a relatively novel concept for crisis management teams, we have labeled the methods we used “future methods”. Section 7 concludes with future directions for human factors in crisis management.

1 Disaster crisis management

Given the scope of disasters, and the large number of groups and organizations involved, disaster crisis management almost by definition consists of a distributed organization. For instance, in the Enschede fireworks disaster (May 13, 2000), at the scene of the incident, the police, the fire service, and the ambulance service played a crucial role in dealing with the disaster. Given the scale of the event, these units had to operate widely apart. At one level removed from the scene of the disaster, those managing the resources and gathering information performed their jobs. At still a different location, the crisis staff coordinated all disaster response and recovery activities. Yet other organizations were involved, for instance: the Red Cross, the Disaster Identification Team, the Royal Netherlands Airforce, the Military Police and the National Reserves, and volunteer groups. Any crisis management organization, including the one involved in the Enschede fireworks disaster, can be characterized by the following characteristics:

  1. 1.

    The organization consists of a collection of temporary groups that remain together for the duration of the disaster and are dissolved afterward.

  2. 2.

    Some of the groups involved have a history of working with each other (both within and between groups), but other groups may never have worked with each other before.

  3. 3.

    The crisis management organization is faced with a complex, novel task (compared to a routine task).

  4. 4.

    The organization has a large distribution of expertise and may be both functionally and culturally diverse. Task knowledge is widely distributed across the organization, and not centralized on any one leader.

  5. 5.

    Given the large number of organizations involved, crisis management is faced with high coordination requirements among the organizations.

  6. 6.

    In order to obtain a timely, accurate, and complete overview of the situation, high communication requirements are involved.

  7. 7.

    The groups involved are geographically distributed.

The characteristics mentioned above frequently result in challenges typical for any crisis management organization (see Quarantelli 1989, 1999, for overviews):

  1. 1.

    The fact that temporary groups are involved may result in conflicts between established local agencies and outside or emergent groups (e.g., the local medical emergency team in Enschede that criticized the Disaster Identification Team).

  2. 2.

    Groups without a history of working together may have fewer shared beliefs about who knows what, resulting in longer search times for finding relevant information; groups may also have to interact with organizations with which they are unfamiliar, resulting in the use of different, for instance, informal channels of communication.

  3. 3.

    Being faced with a complex, novel task results in various problems, for instance, lack of clarity in responsibilities for new disaster-related tasks (e.g., making up lists of missing persons), the problem of dealing with volunteers, who often hinder rather than help, and the conflict between pre-disaster bureaucratic structures and distributed network structures that often emerge after a disaster (e.g., the perceived coordination failure in the Exxon Valdez response may have been due to a clash between the official, centralized, coordinators—Exxon and the Coast Guard—and emergent local stakeholders—fishermen, hatcheries, scientists, local city governments, see Topper and Carley 1999).

  4. 4.

    The distribution of expertise, together with the requirement for integrating this expertise, to solve a complex, novel, problem, imposes large coordination demands on the crisis management organization. The objective is to try to facilitate adequate linkages among the relevant groups rather than attempting to centralize authority (see research by Rulke and Galaskiewicz (2000), and Schraagen et al. (2010b), which has shown that groups of specialists perform better in decentralized network structures than in hierarchical structures).

  5. 5.

    One of the major problems in the development of proper coordination is that organizations differ in how they interpret “coordination”. There is seldom an explicit understanding of what coordination means in operational terms. Second, there are problems in coordinating organizations from the public and private sector, as these organizations differ in their interests, tasks, and goals. Third, coordination between organizations working on common but new tasks (e.g., the handling of mass casualties) is also difficult. It is partly the newness of many disaster tasks that creates strained relationships among organizations that had previously worked together in harmony.

  6. 6.

    In most disasters, there are enough communication channels available. Rather, the problems are in poor, incomplete, or inefficient information flow. Several factors contribute to these problems: volunteers may not be linked into the message system; crucial information does not reach all relevant officials (e.g., due to false assumptions that the information will be transmitted); groups have to interact with organizations with which they are unfamiliar; during an emergency time period, informal channels of communication might be used between organizations that ordinarily use formal links; communication between organizations and the public may be strained, because of uncertainty as to what the public needs in terms of information; from the public’s point of view, there frequently is uncertainty as to who should be contacted in the official organizations, resulting in an overload of inquiries and questions.

  7. 7.

    The geographical distribution of the organizations involved greatly contributes to the problems of coordination and communication discussed above.

Above, we have focused on the organizations involved in crisis management. We now briefly focus on the way individuals react in disasters. According to Quarantelli (1989), it is generally assumed that individuals—especially at the emergency time period—are likely to panic and act irrationally, will be stunned and unable to take care of themselves, act in anti-social ways, be emotionally traumatized or psychologically incapacitated, and react selfishly and in self-centered ways during and immediately after a disaster threat and impact. Based on over 500 field studies that the Disaster Research Centre alone has conducted on natural and technological disasters since 1963, these assumptions of how individuals react are almost totally incorrect in every respect. People perform far better than false speculations or widespread mythologies about human behavior under stress would indicate. In summary, the literature on disaster crisis management has shown that the source and locus of most problems in the emergency phase of disasters is not to be found in the victims, but in the organizations attempting to help them. The organizations involved struggle with problems of communication and information flow, authority and decision making, and coordination. Decision support systems should address these problems.

2 Macrocognitive functions

2.1 Naturalistic decision making

In the 1980s, a number of researchers adopted a concept of decision making that seemed quite different from the standard “option generation and comparison” framework. Lipshitz (1993) tabulated nine different models of decision making that had been cited, advocated, or proposed by this emerging community of researchers over that decade. Two of the most widely cited models were Rasmussen’s (1983, 1988) Skills/Rules/Knowledge account, along with the “decision ladder”, and Klein’s (1989) Recognition-Primed Decision (RPD) model. From its inception, the field of NDM sought to understand decision making in a way that contrasted with the established approach. The concept of decision making had often been defined in terms of a gamble: given two or more options, with certain information about the likelihood of each option to succeed, which is selected? However, the early NDM studies found that people (domain practitioners, consumers, managers, and so on) rarely made these kinds of decisions. Some have suggested the Klein et al. (1986) study of firefighters marks the beginnings of NDM. Using a structured interview method, the researchers found that fire fighters do not evaluate options. They do not conduct anything like a “utility analysis” in which a list of options is generated, and each option is evaluated. More importantly, this is a domain in which decisions could not possibly be made using utility analysis. Thus, what purchase on reality was had by “normative” models that described how rational decisions should be made? The house would burn down, or worse, people would die. In many domains, decision makers often have to cope with high-stakes decisions under time pressure where more than one plausible option does exist, but the decision maker uses their experience to immediately identify the typical reaction. If they cannot see any negative consequence of adopting that action, they proceed with it not bothering to generate additional options or to systematically compare alternatives. Thus, the metaphor of a decision as a gamble did not seem to apply very often. If the “decision as gamble” metaphor failed to describe what practitioners usually encounter and usually do, NDM would abandon the metaphor and follow the phenomena.

As the NDM framework broadened, researchers came to realize that they were interested in the cognitive functions that were carried out in natural settings. “Naturalistic Decision Making” was evolving into “Naturalistic Cognition”. The same kind of mission still applied, and the same cognitive field research and cognitive task analysis methods still applied. But it was time to recognize that the interests of the NDM community had expanded. It came to be generally understood that the designation of NDM made sense primarily in historical context—as a reminder of the initial successes in discovering how decisions are made under time pressure and uncertainty and the importance of studying decision making in real-world contexts—but no longer captured the spirit and mission of the movement.

In (2000), Klein et al. suggested the concept of macrocognition as an encompassing frame for studying the cognitive processes that emerged in complex settings. They attempted to encourage a dialog between laboratory and field researchers. Like Cacciabue and Hollnagel (1995), Klein et al. defined macrocognition as the study of complex cognitive functions including decision making, situation awareness, planning, problem detection, option generation, mental simulation, attention management, uncertainty management, and expertise. In other words, it was dawning on people that macrocognition is what NDM is really about, after all.

Macrocognition is seen as the study of cognitive phenomena found in natural settings, especially (but not limited to) cognitive work conducted in complex sociotechnical contexts. The concept of macrocognition retains the essence of NDM, but with a broader mandate. Quite a few macrocognitive functions (e.g., sensemaking, planning, and adaptation) and supporting processes (e.g., maintaining common ground, uncertainty management) have been distinguished (see Klein et al. 2003) but for the topic at hand (supporting humans in the crisis loop), some deserve special attention, particularly communication, coordination, critical thinking, adaptability, team knowledge, and shared situation awareness.

2.2 Communication

Information needed to perform a task is distributed among team members by means of communication. In effective teams, the process of information exchange between team members is clear, accurate, in a prescribed manner using proper terminology, and such that the ability to clarify and acknowledge the receipt of information is possible (Cannon-Bowers et al. 1995). Several researchers that have investigated communication in teams concluded that effective teams communicate in a similar manner (Kanki et al. 1991; McIntyre and Salas 1995; Orasanu and Salas 1993). Based on interviews and observations of more than sixty military naval teams, McIntyre and Salas (1995) concluded that effective teamwork involves information exchange that is often characterized by closed-loop communication: the sender communicates a message, followed by the receiver who confirms that message or provides feedback otherwise, in turn the sender makes sure that the content of the intended message is perceived well. Often the purpose of communication is to clarify or acknowledge the receipt of information. Effective teams communicate clearly and accurately and send and acknowledge information, instructions, or commands (Brannick et al. 1997). They also avoid excess chatter, use proper phraseology, and provide complete internal and external reports.

2.3 Coordination

Coordination refers to team members executing their activities in a timely and integrated manner. Cannon-Bowers et al. (1995) included the allocation of resources in this definition. It is the process by which team resources, activities, and responses are organized to ensure that tasks are integrated, synchronized, and completed within established temporal constraints. Several investigators make a distinction between explicit and implicit coordination (Entin and Serfaty 1999; Kleinman and Serfaty 1989). Explicit coordination is viewed as the transfer of information and resources in response to explicit verbal requests. Implicit coordination is the mechanism that supports teams to adapt in complex environments. Team members know implicitly what to do, how to compensate for their teammates’ limitations, and what information or materials they must provide for teammates (Blickensderfer et al. 1998). Team members that offer essential information and resources voluntarily to other team members engage in implicit coordination. It is asserted that effective teams coordinate implicitly, especially during periods characterized by high workload and time pressure. Under these circumstances, explicit coordination would take too much time to be effective. In order for teams to coordinate implicitly, team members need to be aware of each other’s information requirements, strengths and weaknesses, and current levels of workload.

2.4 Critical thinking

In ambiguous situations, missing information is usually inferred from the story constructed on the basis of existing information (Pennington and Hastie 1993). Although this is generally an effective strategy, there is a risk: people, even experts in naturalistic settings, tend to stick to their original interpretation of an ambiguous situation, even when information contradicting such an interpretation accumulates as the situation unfolds (framing bias or information order bias; e.g., Adelman et al. 1993, 1996). They will not switch to another interpretation of the situation (tunnel vision) and they also tend to see neutral or irrelevant information as confirming their initial explanation. People, sometimes including experts in naturalistic decision-making contexts, are inclined to focus on information that confirms their initial explanation of events (confirmation bias) and easily to discard and forget information that contradicts this (Perrin et al. 2001). Proficient decision makers sometimes consciously employ a special technique that enables them to overcome these biases, the critical thinking technique. This technique helps them to recognize when situations are unfamiliar and problematic and supports them to improve situation awareness (Cohen et al. 1996). However, when under time pressure, the technique may not be applied because it is “forgotten”. Zakay and Wooler (1984) found that training in a particular compensatory decision strategy did not improve the quality of decision making at all, and the effectiveness of the decisions was significantly lower than under no time pressure. Maule and Mackie (1990) found that, under time pressure, people resorted to a strategy of selectively ignoring certain information. Even though experts’ decision making may be unaffected by time pressure (Calderwood et al. 1988; Gobet and Simon 1996), crisis managers’ experience base is usually too limited for them to be called “experts”. In Sect. 4, we will describe a critical thinking tool to support crisis managers to apply the technique, also under time pressure.

2.5 Adaptability

Adaptability is an important teamwork behavior that refers to team members using information from the task environment to adjust team strategies by backing each other up, make mutual adjustments, and reallocating team resources (Blickensderfer et al. 1998; Kozlowski 1998; Serfaty et al. 1998). LaPorte and Consolini (1988) found that teams with records of failure-free performance are extremely adaptive to varying task demands. These teams were observed to switch between several different coordination strategies and organizational structures, with different lines of authority, communication patterns, and task responsibilities as they move between normal operations and high-tempo or emergency situations. LaPorte and Consolini (1988) concluded that effective teams have not one but several organizational structures and shift between them when needed. Furthermore, reliable teams maintain open and flexible communication lines that promote the free flow of information from lowest to highest levels as well as the other way around. Finally, in adaptive teams, team members are extremely sensitive to other members’ workload and performance in high-tempo situations. Serfaty et al. (1998) demonstrated that teams could be trained in the use of general team coordination strategies to adapt to increases in workload and stress. In particular, they found that training team leaders to provide brief (30 s) periodic situation-assessment updates every 3 min to the rest of the team greatly improved both taskwork and teamwork.

2.6 Team knowledge

Several researchers assert that performance is positively affected when team members have shared knowledge about the team and its processes (Blickensderfer et al. 1997, 1998, 2000; Cannon-Bowers et al. 1998, 1993; Klimoski and Mohammed 1994; Mathieu et al. 2000; Orasanu and Salas 1993; Rentsch and Hall 1994; Rouse et al. 1992; Stout et al. 1996, 1999). Team knowledge affects team performance in two ways (Blickensderfer et al. 2000). First, team knowledge enables team members to anticipate each other’s task-related needs. That is, based on team knowledge, team members are able to provide each other with information, resources, and assistance without being requested first and without extensive communication to coordinate these activities. This way, team members are able to perform their tasks in correspondence to those of their teammates and to provide information the moment a teammate needs it. Second, shared team knowledge provides team members with a common frame of reference that enables team members to explain and predict the tasks and the situation accordingly. This enables team members to determine strategies cooperatively. Researchers tend to agree that team knowledge supports teams to coordinate smoothly and effectively and several researchers have found evidence that suggests a positive relationship between shared team knowledge and team performance (Blickensderfer et al. 1998; Cannon-Bowers et al. 1998; Mathieu et al. 2000; Stout et al. 1999).

2.7 Shared situational awareness

The shared situation awareness (SSA) of a team refers to the degree to which the members have the same interpretation of ongoing events in the situation (Endsley 1995). SSA is important for effective decision making and team results. Without SSA, teams lack common ground on which to base their decisions, and the result may be flawed decision making due to different perceptions of the situation, the current task, the responsibilities involved, or other factors. SSA is often a problem because team members do not always understand which information is needed by other team members, because they lack the proper devices to share information, because they lack shared mental models, or because they lack the communication skills for sharing relevant information. Situation awareness is defined as “the perception of the elements in the environment within a volume of time and space, the comprehension of their meaning, and the projection of their status in the near future” (Endsley 1995, p. 36). Salas et al. (1995) concluded that team situation awareness involves two critical processes, namely the development of individual situation awareness and teamwork to develop shared situation awareness. Team members each develop their own set of situation awareness elements. Some overlap, however, must exist among team members’ situation awareness elements. Team situation awareness is dependent then on both the individual and the shared part of situation awareness. The shared part may be developed by such teamwork skills as providing periodic situation-assessment updates and checking that all team members have understood the updates.

3 Team situation awareness

In a recent review of situation awareness models for individuals and teams, Salmon et al. (2008) distinguished between team situation awareness models, shared situation awareness models, and distributed situation awareness models. In their view, team situation awareness models comprise individual team member SA, SA of other team members, and SA of the overall team. Shared SA refers to the level of overlap in common SA elements between team members. Distributed situation awareness views SA as an entity that is separate from team members and is in fact a characteristic of the system, the ensemble of humans and machines, itself (Artman and Garbis 1998). Salmon et al. (2008) argued that the team SA perspective may be sufficient for simple, small-scale collaborative scenarios, but that the distributed SA approaches are the most suited to describing and assessing SA in real-world, collaborative environments, such as in military network enabled capability scenarios. Even though we focus on measuring team SA in a complex crisis management environment, our measurement is developed from the team SA perspective.

The difference among the use of these concepts, according to Salmon et al. (2008), seems to revolve around the necessity of sharing everything with everyone. In this respect, there is a parallel with the ambiguous concept of “shared” as in “shared mental models”. “Shared” can either mean “to have in common”, as in shared beliefs, or “to distribute”, as in sharing a dessert (Cooke and Gorman 2007). However, in our opinion, the concept of team SA is not limited to the notion of “to have in common”. Team-level properties are in large part the result of team member interactions (Hackman 1987). Therefore, SA of the overall team is constituted by team member interactions (Salas et al. 1995). Individual team member SA may or may not be shared through communication processes with other team members. Hence, the communication processes are what matters in teams, not the amount of knowledge shared. Team SA (TSA), by focusing on team processes, does not assume complete overlap in knowledge and is therefore just as applicable in crisis management environments, where there is obviously no complete overlap, as in small, simple-scale scenarios.

Because achieving and maintaining TSA in crisis management environments is a continuous, complex, and dynamic team process that changes constantly, we focus on the processes of acquiring and maintaining TSA. Furthermore, the definition of a team as “a distinguishable set of two or more people who interact dynamically, interdependently, and adaptively toward a common and valued goal/object/mission, who have been assigned specific roles or functions to perform, and who have a limited life span of membership”. (Salas et al. 1992) already implies that TSA cannot be seen as a state at a certain point in time, but refers to a continuous, complex, and dynamic team process.

Most existing measurements do not measure processes but measure TSA as a product, for example, tools such as the situation awareness global assessment technique (SAGAT). Content methods and flow methods, on the other hand, can be used to gain insight in processes by observing team-member interactions. Content methods focus on an analysis of the meaning of communication. Flow methods tend to focus on the sequencing and timing of communicative interactions among team members. Although these methods are useful in laboratory settings, they are too time-consuming and expensive to be useful in field settings. They also require extensive instrumentation and the presence of expert observers and analysts. In practice, this is frequently impossible.

We have applied self-rating techniques to measure TSA in crisis management environments (Schraagen et al. 2010a). We see four distinct advantages to this approach. The first is that team processes are observable. Team members have access to, and can form judgments on, the quality of communication processes involved in the sharing of information. The second advantage is that self-rating techniques do not interfere with task performance because they are completed posttrial. Thirdly, self-rating questionnaires are very quick and easy to use because they require very little training, accompanied by very little cost. A fourth advantage is that self-ratings of TSA can be obtained from different team members (Endsley 2000), thereby offering a first step into the assessment of team SA. One aspect that needs to be taken into account when using self-rating techniques in crisis management environments is that they need to be adjusted to suit the dynamic, complex, and collaborative nature of such environments. A crisis management-specific self-rating technique thus needs to be developed.

We have applied our TSA questionnaire to a crisis management exercise. The training “ROAR” was part of a flood control training “Viking” that takes place every other year in the Netherlands. Multiple organizations were involved in this training, namely officers of first responders (police, fire department, and paramedics), liaisons of the department of defence, districts water board, and the Ministry of Waterways and Public Works. All these organizations have their own tasks and responsibilities. For effective crisis management, it is important that they have accurate and shared situational awareness of the situation and the plan. The questionnaire Shared Situational Awareness was distributed on the second day of the training (April 17, 2008). We were interested in how well the conditions, processes, and outcomes related to TSA during this training were judged by team members. We were also interested if these judgments were different for experienced and less experienced team members. The team was composed of 16 members, nine team members had little experience with being a member of an operational team (0–5 times) and seven members had more experience with being a member of an operational team (6 or more times). For each of the items of the questionnaire, the difference between these two groups was calculated. The results were discussed with an observer of the exercise and with a team member who took part in the crisis management team. In the following part, the results for conditions, processes, and outcomes will be described.

3.1 Conditions

Team members knew what the roles and tasks of the other team members were, but they did not know what information was needed by other team members. Team members used their own jargon that was not always known by other team members. Unknown terms were not always explained. The complexity of the task was judged differently by the experienced and less experienced team members. The experienced team members rated the complexity of the task as high, and the less experienced team members judged the complexity of the task as average. A possible explanation for this finding is that the less experienced team members only took the technical aspects into account and not the social aspects of the situation (for example, deciding to evacuate a town). The social aspects make the task more complex.

3.2 Processes

For some processes, an effect of experience was found. The process “predicting what might happen in the future” was rated “poor” by the less experienced team members; however, it was rated “good” by experienced team members. An explanation for this might be that more experienced team members needed less time to build a picture of the situation. They had more time to predict what might happen in the future. The less experienced team members need more time to build a picture of the situation (SA level 1 and 2—perception and comprehension) and had less time to look ahead (SA level 3—projection). “Asking critical questions to get a clearer picture of the goals and the planned actions” was also judged differently by both groups. More experienced team members judge this as more positive than the less experienced team members. After the results were discussed it was remarked that more experienced team members asked questions during the meetings, less experienced team members asked questions after the meeting, more in one-on-one situations, probably because they felt insecure to ask questions during the meeting. “Determining for whom information might be relevant and pass information to other team members” was not sufficient. This might have been caused by the structure of the building. Different organizations were located in different parts of the building and this might hamper sharing information. Another explanation is that team members did not know enough about what information was needed by the other team members (condition). Insight in each others’ expertise and knowing what information is needed by the other team members are important for shared situation awareness.

3.3 Outcomes

Team members judged the shared awareness of the situation and the reached goal as positive. The score of the shared awareness about the planned actions was lower. During this exercise, the planned actions were not really carried out. Therefore, it is difficult to judge the quality of the planned actions. The decision-making process was judged more positively by experienced team members than by the less experienced team members. Expectations that people had in advance might have caused this difference. Experienced team members may have had a frame of reference based on earlier real life experiences and exercises.

3.4 Conclusion

The most important finding was that sharing information with other team members was not sufficient. Sharing information was difficult because there was no shared information system, and the organizations were located in different parts of the building because of the structure of the building. Sharing information with other team members is easier if everybody has access to the same information and if people are located close to each other. Another problem was that team members did not have a good insight in each others information needs. This also influences sharing information. Gaining insight in the need for information can, for example, be improved by changing roles during training. More time and effort should be put into sharing information with other team members.

3.5 Limitations

We are currently validating our questionnaire with larger sample sizes. We have recently administered the questionnaire during a large-scale NATO Response Force exercise, with units distributed across three countries. Approximately, 80 respondents have filled out the questionnaire that allows us to calculate internal consistency. Another area for improvement is more rapid data collection and reporting of results to crisis managers. We are working on automating data collection and analysis, so as to be able to present results to managers during the exercise, instead of a few hours or days afterward. This should enable managers to get a quick overview of their team’s SA. As a next and logical step, we will focus on interventions that managers can employ to enhance TSA.

4 Critical thinking support

As mentioned in the previous section, sharing information to achieve team situation awareness is critical, but also one of the most difficult tasks in the early stages of large-scale accidents. Managing uncertainty in these early stages may be accomplished by fostering team collaboration, particularly collaboration to critique and refine team outcomes such as assessments and plans. This so-called Collaborative Critical Thinking (Hess et al. 2008) is the interaction between team members that manages uncertainty by revealing it, identifying its sources and devising ways to test its depths or diminish it. However, before being able to apply this cognitive skill to the group level, individuals need to be trained and supported in individual critical thinking. In novel or ambiguous situations, which large-scale accidents often create, the pattern-matching and recognition-primed strategies of experts (Klein 1998) cannot be regularly applied. In such cases, the specific cues from the incident will not trigger stored emergency plans, because of the novel situation. However, they might trigger more general strategies that can be applied to the situation (Samurçay and Rogalski 1991).

Cohen et al. (1996) introduced the concept of critical thinking to tackle the issues of tunnel vision and information bias described in Sect. 2. Heuer (1999) developed a similar concept, called Analysis of Competing Hypotheses, for the intelligence community. A piece of software, called ACH, was developed by Palo Alto Research Center (PARC), in collaboration with Heuer.Footnote 1 In ACH, all hypotheses are identified, and a list of evidence for and against each hypothesis is drawn up. The consistency or inconsistency of each item of evidence with each hypothesis is then assessed, and gaps in the evidence are identified. This is a highly analytical, non-real time process. We developed a critical thinking tool (CTT) further extending these concepts (Schraagen and van de Ven 2008). Although our CTT resembles the ACH Software, we intend it for use in real time crisis management situations, where hypotheses are generated under time pressure based on continuously arriving new pieces of evidence. The CTT helps users to keep options open during the decision-making process. This is achieved by letting the user color-code, the evidence that supports (or negates) a hypothesis (Schraagen et al. 2005). Our assumption is that by making the argument structure (i.e., the relation between evidence and hypotheses) visible in the interface of the tool, problems like tunnel vision and information bias will be reduced. We hypothesize that use of the CTT will result in a qualitatively better decision-making process, especially in situations where information is ambiguous and incomplete. Using the CTT, people try to fit a piece of information into a coherent story. The task is not to build a story based upon the incoming information, but to support (or disprove) a story (hypothesis). This way of working has the advantage of off-loading memory and enabling the user to think freely of other possibilities. In this manner, the CTT also helps to make implicit assumptions explicit. This assists people to self-critique ideas, to request critiquing assistance from another person or computer, to hand over tasks, and to review situations and learn from them.

A laboratory experiment (see Schraagen and van de Ven 2008, for details) evaluating the critical thinking tool led to two main findings. The first was that people using the tool came to better conclusions and used supporting evidence to draw those conclusions. People using the critical thinking tool were more successful in avoiding tunnel vision and information bias than people not using this tool. The drawback of using the tool, and our second main finding, was that color coding imposed a time delay, which could be a problem in a time-pressured situation such as a crisis. However, given the decreases in reading time and decision time that still occurred toward the end of the experiment, particularly in the full support group, we surmise that with practice time delays might well disappear. Furthermore, crisis management is not a one-person show, and it seems very likely that the time spent on coding evidence will prove beneficial in fostering team situation awareness and collaborative critical thinking (Hess et al. 2008).

5 Crisis management team training

The realization grows that just putting together a team of individual experts does not make an expert team (Salas et al. 1997). In recent years, it has been shown that a good approach to training teams with complex training technology is linking training goals to events in training scenarios in a controlled fashion. This is called the “event-based approach to training” (EBAT: see Fig. 1) (Hall et al. 1993; Johnston et al. 1997).

Fig. 1
figure 1

The EBAT framework (Johnston et al. 1997)

Full size image

The EBAT framework starts at the top left-hand side with the tasks to be performed by the team. The basic assumption is that training should provide opportunities for practice, enabling a team to develop critical competencies to conduct their mission, or to manage an emergency. The team and individual behavior indicating these competencies are explicitly described in the learning objectives. Based on these learning objectives, the training scenario is developed. A training scenario consists of several events that are specifically designed to trigger the team members’ behavior as described in the learning objectives. Events are critical incidents that can occur during the course of the emergency and on which the team should react. For every event, the observers know what behavior the team should demonstrate and which prototypical mistakes could be made. This facilitates a systematic observation of the team members’ behavior. Based on these measurements, the training staff is able to make a valid diagnosis of the performance and to assess to what extent the learning objectives have been achieved. During the debrief, feedback is provided to the team and, together with the team, the lessons learned are formulated. The strength of EBAT is the systematic linkage among these components. Without this, linkage is impossible to ensure that team members will have learned anything from the training.

5.1 Training team processes

Two training methods that have proven their value for training teams and their leaders are described in this section: Team Dimensional Training and Critical Thinking.

5.1.1 Team dimensional training

An important distinction that resulted from research on team training is the concept of “taskwork” and “teamwork” skills underlying team performance (Cannon-Bowers et al. 1995; Cannon-Bowers and Salas 1998). Taskwork consists of the position-specific requirements of the job, which are usually technical in nature (such as operating a certain workstation). Teamwork has more to do with processes that individuals use to coordinate their actions. Both taskwork and teamwork skills are important in any given team and team member (see Table 1).

Table 1 A distinction of skills with respect to training teams and team members (Van Berlo 1997)
Full size table

Smith-Jentsch et al. (1998) have further delineated the skills underlying teamwork, and they identified four dimensions underlying effective teamwork: information exchange, communication, supporting behavior, and initiative/leadership. These four generic teamwork skills can be regarded as key competencies of any member of an emergency management team. Learning how to work together is especially important for teams consisting of team members that frequently vary. In these cases, team members should possess adequate teamwork skills.

Team Dimensional Training (TDT) is a training methodology designed to aid instructors in training and evaluating teamwork skills (Smith-Jentsch et al. 1998; Schaafstal et al. 2001). This is accomplished through a four step training cycle: briefing a team, observing a team’s performance during a training exercise, diagnosing this performance, and debriefing the team about its performance. During the briefing phase, the four teamwork dimensions delineated by TDT—and behaviors associated with each—are presented to the team by the trainer. During the exercise itself, the observers gather positive and negative examples of behaviors that fall under each TDT dimension. During the debriefing phase, the trainer facilitates the discussion of the team’s performance, providing positive and negative examples of team behavior (Smith-Jentsch et al. 1998).

5.1.2 Critical thinking

Expert decision makers treat decision making as a problem-solving process. They use familiar elements to construct an initial interpretation of the situation. The plausibility of this interpretation is verified by explicitly challenging its critical assumptions. When faced with a complex and unfamiliar problem, experts collect and critically evaluate the available evidence, seek for consistency, and test assumptions underlying an assessment. They then try to integrate the results of the processes in a comprehensive, plausible, and consistent story explaining the actual problem situation. Experts assess the risks associated with potential courses of action by consulting their experience as well as by means of mental simulation (Zsambok and Klein 1997). The knowledge of experts and the strategies they employ when dealing with complex situations have been used to develop a new form of training in decision making and risk management: Critical Thinking (Cohen and Freeman 1997; Van den Bosch and De Beer 2007). The aim of critical thinking training is to keep trainees from assessing situations solely on isolated events. Instead, trainees are taught how they can integrate the available information into its context, which may include elements such as: the history of events leading to the current situation, the presumed goals and capacities of the enemy, potential risks associated with the environment, the opportunities of the enemy, etc. Trainees are instructed on how to identify (in)consistency and uncertainty and how to adjust or refine their story by deliberate testing and evaluation.

6 Future methods

Future methods focus on a group of teams or on the handover from one team to the other. The other theme across these methods is communication or, more specifically, information exchange between these teams. One of these methods is Network Centric Operations (NCO) (Alberts and Hayes 2007). NCO is an operational concept in which information is shared throughout the organization, vertically and horizontally. People needing information to take a decision or fulfill an assignment have access to the information, even when it is provided by a person from another unit/stovepipe. The main goal of NCO is to create a better situation awareness throughout an (ad hoc) organization. Usually there are two main limitations to improving situation awareness: time and information. We have to deal with the time limits and the fact that we do not have all the information on the situation when we have to decide. NCO is a way to improve sharing information in complex situations, where multiple teams (organizations or units) are involved. Military organizations, including some in the Netherlands, Sweden and the United States, are already experimenting with this new way of working. Since 2005 the Dutch crisis management organization is also experimenting with NCO (Van de Ven et al. 2008).

After 3 years of field studies, in 2008, a team of people started to assist Dutch safety regions (25 in total) with the implementation of NCO. This process is still on its way and will take another year. Next to the actual implementation of a software system that supports a shared view on the situation (to avoid interoperability problems a new system is introduced), new support tools are developed. A support tool to train NCO-competencies is one example, a maturity model is another. A maturity model is like a high-level map, it points out the contours of each phase, without going into details how the actual world looks.

Both tools are used to understand NCO, what does it mean and how does it work. A third tool under development is the information view. Although at this moment the information view on the incident is static, we are currently developing new ways to create a more dynamic—personal—view on the incident, while still supporting the shared view on the incident.

6.1 Discussion

While in the past focus lay on a single team, new methods focus more on the entire chain of teams. Creating a better awareness in one team is not enough if the entire incident must be managed by more that just this team. Therefore, currently main developments are on improving situation awareness in the entire chain, has every team (member) the same understanding of the situation? In the near future, when we know that everybody has the same view on the incident, time will be spent on creating awareness of the consequences and planning to fight the incident and the consequences of our actions. That is the next step in development of NCO.

7 Conclusions

This paper has provided an overview of how to support the human in the crisis loop based on a thorough understanding of team situation awareness, critical thinking support, and training critical thinking skills. Based on a characterization of crisis management organizations, we derived several problems typical for any crisis management organization, such as communication and coordination problems, but also conflicts between in-groups and out-groups, and the resulting problems of knowing who knows what, and knowing when information may be relevant for others involved in crisis management.

7.1 Team situation awareness

Team situation awareness refers (TSA) to the continuous, dynamic process of communicating relevant information about the evolving situation in order to make effective decisions. This process of communication and the resulting coordination issues are particularly challenging in crisis management situations, due to the dynamic nature of the situation, the scale of collaboration among ad hoc teams, and the fact that physical separation and stove-piped information systems frequently hinder building up TSA. We discussed several methods for assessing TSA. We have developed a self-rating questionnaire that we have tested in various field settings, one of which being a high-water crisis management exercise. In this exercise, we found that the lack of a common ICT infrastructure and the physical separation of various subgroups hindered sharing of information. Future work in this area should focus on automated communication analysis (Foltz et al. 2008), the removal of information barriers, and shared team awareness aids. Hansén (2009) has described “shared situation awareness” as a buzzword used within the Swedish Emergency Management Agency (SEMA). As a buzzword, according to Hansén, it has been successful in pointing at problems inherent in the Swedish crisis management system, such as information dissemination or lack of coordination. However, as a solution to problems, the concept has not been scrutinized critically and may have obscured the view from alternative interpretations. Hansén (2009) is correct in stating that we should not succumb to quick technological fixes for information dissemination problems, but rather should keep an open mind for organizational and institutional issues as well.

7.2 Critical thinking support

The next topic we addressed was supporting critical thinking in real time situations under time pressure with the aim of reducing confirmation bias and tunnel vision. Our research has shown that the critical thinking tool we have developed was successful in countering these biases, primarily by making competing hypotheses and the evidence supporting them visible. This could be a valuable piece of ICT support for crisis management. Extending the concept of individual critical thinking is important for crisis management situations. Although some first promising steps have been taken in the direction of collaborative critical thinking (e.g., Hess et al. 2008), future work should explicitly link the concepts of team situation awareness and collaborative critical thinking. This is particularly important in distributed organizations and temporary alliances of individuals from different formal organizations, as is often the case in crisis management organizations. Making team knowledge state accessible through collaborative technologies and social media applications is essential for “heedful interrelating” (Weick and Roberts 1993) and achieving “societal resilience” (Longstaff et al. 2009).

7.3 Training critical thinking

Besides supporting critical thinking, training critical thinking is an important venue for research as well. It has been shown empirically that the so-called “critical thinking skills” can be trained, that critical thinking training can be extended to teams and that this training can be generalized across domains (Cohen et al. 1998; Helsdingen et al. 2010). It is, however, unclear at present whether the positive effects of critical thinking training derive from the instruction in its entirety or from specific aspects of critical thinking, such as reflection or self-explanation. Future research should compare the entire critical thinking instruction with, for example, reflection prompts or self-explanation prompts only, to study the effects of these aspects separately (Helsdingen 2008).

Crisis management organizations by their very nature consist of multiple ad hoc groups, often physically dispersed, working with multiple stovepiped information systems. Working together in a coordinated fashion is a great challenge. Human factors can contribute to meeting this challenge by helping these organizations with developing and evaluating new training and support concepts, by setting up training exercises and observing these organizations during those training exercises. This should enable crisis management organizations to share information more effectively and move toward higher levels of shared awareness. In the end, crisis management organizations should be better able to do what they are supposed to do: contain crises and safe lives.