Keywords

1 Introduction

Printing a document from a computer is easy: Press print and the document is printed. But when a system failure arises, printing gets annoying, e.g. when the document is stuck in a “queue”. Often, the computer user does not know what to do next, even if the specific problem has occurred before. First, the user has to know why the problem occurred. Second, the user has to decide whether to (a) fix the problem by him/herself—“pause”, “cancel”, end queue service in system settings, restart or try all of these together or (b) search for instructions on the internet and follow them. In such a case, the system failure would be easier to fix if the system detected the reason for the fault and provided support adapted to the user’s skills. In other words, when the user has already pressed “cancel” (which is part of the fixing procedure) but does not know what to do next and stops interacting, the system would tell the user what to do next.

Such a scenario and skill loss can also occur in any situation in which a skill is rarely used. In industries like process control and air transportation it is difficult and yet important to maintain rarely used skills over a period of non-use. These rare situations in which initially learned skills are necessary are called non-routine situations [1]. Examples are the start-up of a plant, solving issues, taking corrective actions after incidents, or taking over when automation breaks down. In such cases, operators follow ‘standard operating procedures’ (SOP): They have to know which procedure is necessary, what the content of the procedure is, and how the procedure is executed [1]. A single handling error can provoke hugely harmful consequences, which can be prevented by extensive initial skill training and recurrent or refresher training after a defined time period [2, 3]. As such tasks are performed by human-computer interaction, the recall process of once-learned skills can be supported by a gaze-guiding tool that is implemented on the computer interface. A gaze-guiding interface fades in information into the control interface based on the current situation of the system and the intended procedure [4]. The present paper analyzes the potential of gaze guiding for cued skill recall in three complex task types: a fixed-sequence task, a contingent-sequence task and a parallel-sequence task.

1.1 Complex Cognitive Skills, Complex Tasks and Retention

An operator learns complex cognitive skills which consist of a combination of sub-skills. Generally speaking, a skill is a well-organized performance that requires coordinated processes of perception, cognition and action, is acquired through practice, and is performed with economy of effort [5, 6]. Complex cognitive skills are skills that are necessary e.g. in computer programming and dynamic process control tasks [7]. They can be divided into motor sub-skills and cognitive sub-skills [7]. Skills that have once been acquired can decay after periods of non-use [2] due to a decrease in retrieval strength (New Theory of Disuse [8]). Following this approach, skill retention requires the maintenance of high retrieval strength of learned skills.

Complex tasks consist of multiple elements that need to be executed [9]. According to Kluge [1], a complex task “can be decomposed into part-tasks that include sequences of steps, which need to be integrated and coordinated based on attentional processes and need to be orchestrated based on the simultaneous processing of knowledge elements into a interdependent team to meet the organizational goals” (p. 35). For the execution of complex tasks, complex cognitive skills are required, which are learned based on standard operating procedures (SOP) [10]. Such complex tasks in non-routine situations can consist of fixed-sequence tasks, contingent-sequence tasks and parallel-sequence tasks.

  • In fixed-sequence tasks, the operator first needs to ascertain what kind of task has to be executed (e.g. start-up of a plant or error management) and then needs to execute the initially learned standard operating procedures sequentially (SOPs) [1].

  • Dynamic decision making can be defined by multiple, interdependent and real-time decisions, occurring in an environment that changes independently and as a function of a sequence of actions [11]. In such an environment, decisions under certainty take place: The operator is aware of possible alternatives, consequences and the order of preferences [12]. A contingent-sequence task under certainty can consist of a fixed-sequence task in which at a special point or under a special condition, the operator has to perform the next steps based on a correct gathering of information and interpretation of the situation.

  • Parallel-sequence tasks basically consist of two sequences which have to be synchronized in time [6, 13, 14]. In these tasks e.g. the operator has to control a second task while executing a first task, and both tasks are executed based on SOPs. A conscious, directed attention allocation and time-sharing is necessary to perform the task [13, 15]. An example of such a task is when a pilot is controlling different instruments during take-off, and consequently has to divide his/her attention according to change frequency and how valuable and costly the attention is [16].

1.2 Gaze Guiding

Cued recall is the recall of elements from memory triggered by cues [17]. An example of cued recall is remembering words with the support of word categories [18]. Possible methods for applying cued recall to complex cognitive skills are job aids. Job aids are used to process, store and extend information [19, 20]. Skills that are used infrequently, consist of sequences, and contain a large amount of information can be supported by job aids [21]. A procedural job aid is designed to guide users through the procedure step by step [20, 22]. A dynamic computer-based job aid such as the present gaze-guiding tool supports the operator if s/he fails to remember the next step, and interrupts the interaction with the system. Dynamic computer-based job aids have been used for the learning of skills and can be applied through various methods, such as the “attention guidance technique” and “visual cueing”. These methods assist learners in complex learning environments in which it may be challenging to detect relevant information and cues [23]. The attention guidance technique has been particularly applied in skill acquisition [24]. Attention guidance facilitates the search for relevant information and problem solving with salient, critical information [25]. A comparable method is “visual cueing”, which has also been used for learning [26]. Visual cueing highlights e.g. tasks, graphics or animations with arrows, circles or distinctive colors [26]. Some studies have shown that visual cueing guides attention and supports fast skill acquisition of information, retention and problem solving [26–28]. However, various studies found that simply highlighting relevant areas was not sufficiently effective for improving performance and reducing errors [24, 25, 29]. As a conclusion, it seems reasonable to provide further textual information, in addition to highlighting relevant areas, and to ensure a resource-friendly processing and understanding [24]. This can be offered by a gaze-guiding tool, which is able to guide the operator’s gaze and consequently attention to the relevant areas and provide further information [30]. The optimal gaze-guiding tool balances the operator’s mental workload and does not overextend her/him with too much information [31]. Recent studies have shown that gaze-guiding methods have been used successfully as learning support [32]. For instance, the performance of surgery students was found to be positively supported by a form of gaze guiding showing the next relevant steps as compared to exploration [32].

Based on the assumption that cued recall embodied by gaze guiding harbors the potential to support the operator’s skill recall after a period of non-use, the following hypothesis is developed (and analyzed for each study):

Hypothesis: After a period of non-use and in a retention assessment, a group which is supported by gaze guiding shows a superior performance compared to a group without gaze guiding and free recall.

2 Method

2.1 Sample

Studies 1–3 were conducted from October 2014 to December 2015 with a sample of 100 participants for each study. In the following, only a subsample is analyzed to investigate gaze guiding in particular. The three studies differed only with respect to the learned task type (fixed-sequence task, contingent-sequence task, parallel-sequence task).

Study 1. From October 2014 to December 2015, 40 engineering students (15 female) participated in the study. Four participants were excluded based on the selection criteria (see below). The participants were recruited by postings on social networking sites and flyers handed out to engineering students. To ensure technical understanding which was required for the technical task, only students from faculties of engineering were recruited. Participants received 25€ for taking part. The study was approved by the local ethics committee. Participants were informed about the purpose of the study and told that they could discontinue participation at any time (in terms of informed consent). All participants were novices in learning the process control task used in the study. The recruitment was similar for all three studies.

Study 2. From April to July 2015, 40 students took part in the study (12 female). Five participants were excluded based on the selection criteria (see below).

Study 3. From October to December 2015, 40 students took part in the study (13 female).

2.2 Process Control Tasks

The complex cognitive skill in the present studies was performed in a simulated process control task: The participants had to know the content of the particular start-up procedure (SOP) and how to interact with the interface. The participants learned how to operate the microworld Waste Water Treatment Simulation (WaTrSim). The operation of WaTrSim includes the start-up of the plant, which is assumed to be a non-routine task that requires skill retention [10]. In WaTrSim, the operator’s task is to separate waste water into fresh water and gas by starting up, controlling and monitoring the plant. The operation goal is to maximize the amount of purified water and gas and to minimize the amount of waste water (only separated purified gas outcome was used for the calculations because this shows that the participants performed the last steps of the start-up procedure; water had already been produced after the operation of column K1). This goal is achieved by considering the timing of actions and following the start-up procedure. The start-up procedure differed in all three studies.

Study 1. The operation included the start-up procedure of the plant as a fixed-sequence task comprising 13 steps (Table 1). Performing the WaTrSim start-up procedure correctly and in a timely manner led to a production outcome of a minimum of 200 l of purified gas. The minimum amount of purified gas in the initial training was used as selection criterion (>=200 l). Start-up time was max. 180 s.

Table 1 Description of fixed-sequence task (Study 1), contingent-sequence task (Study 2) and parallel-sequence task (Study 3)
Full size table

Study 2. The operation included the start-up procedure of the plant as a contingent-sequence task comprising 13 steps and following five steps for each condition. The following five steps had to be executed depending on the conditions: heating W1 > 15 °C or heating W2 < 70 °C. After one of the conditions had occurred, the correct four steps had to be executed (Table 1). Performing the WaTrSim start-up procedure correctly and in a timely manner led to a production outcome of a minimum of 100 l of purified gas. The minimum amount of purified gas in initial training was used as selection criterion (>=100 l). Start-up time was max. 240 s.

Study 3. The operation included the start-up procedure of the plant as a parallel-sequence task. Two sequences had to be operated in parallel: 13 steps for sequence 1 and three steps for sequence 2. Sequence 2 had to be executed when the level of tank Bf had reached >75 % or <25 %. After one of the conditions had occurred, the correct two steps had to be executed (Table 1). Performing the WaTrSim start-up procedure (both sequences in parallel) correctly and in a timely manner led to a production outcome of a minimum of 200 l of purified gas. The minimum amount of purified gas in initial training was used as selection criterion (>=200 l). Start-up time was max. 240 s.

2.3 WaTrSim Gaze Guiding

The gaze-guiding tool used in the present study (Fig. 1) has been described, pretested [33] and evaluated as effective in a process control task [4]. The gaze-guiding tool is technically implemented with a transparent, darkened interface that leaves space for the relevant cue element, a red-orange flashing frame of the cue element, and additional textual information of the following step with a pictogram (Fig. 1). The gaze-guiding tool is faded in at a predefined time point and after the system has reached the relevant conditions for the next steps in order to ensure that the operator has time to recall and react independently without the gaze-guiding tool. The gaze-guiding tool is only faded in if the operator has not executed the relevant next step in a timely manner. After the gaze-guiding tool has been faded in, it disappears again when the operator interacts with WaTrSim. It is faded in repeatedly until the operator executes the correct adjustments of the relevant step [4, 33].

Fig. 1
figure 1

Example of the gaze-guiding tool. The interface is darkened, the valve is highlighted with a flashing red-orange frame and a yellow text box is displayed

Full size image

2.4 Research Design

Studies 1–3. The studies consisted of a 2 × 2 mixed experimental design (between- and within-subjects design). One experimental group (Gaze-Guiding group) and one Control group were analyzed at two measurement time points (week 1: initial training and week 3: retention assessment).

2.5 Variables

Independent Variables. The gaze guiding is described in 2.3 (adapted to the sequence that needs to be executed in Studies 1–3).

Dependent Variables. The production outcome was measured by the produced amount of purified gas. For the calculations, the best trial of the initial training (week 1) and the first trial of the retention assessment (week 3) were used. The minimum production outcome at the initial training depended on the task. Study 1: 200 l, Study 2: 100 l, Study 3: 200 l.

The start-up mistakes are the sum of incorrect valve adjustments and procedure mistakes, such as adjustment of the incorrect valve flow rate, and depended on the task. The start-up mistakes of the best production outcome trial of the initial training and of the first trial of the retention assessment were used for the calculations. Study 1: 0–15 mistakes, Study 2: 0–19 mistakes, Study 3: 0–21 mistakes (recalculated in percentages, 0–1).

Control Variables. Start-up time: The start-up time was measured for the best production outcome trial of the initial training (week 1) and the first trial of the retention assessment (week 3). The start-up time was limited depending on the task. Study 1: 0–180 s. Study 2: 0–240 s. Study 3: 0–240 s.

Retentivity was measured with the Wilde Intelligence Test-2, which consists of verbal, numerical and figural information [34]. First, the participants had to memorize the verbal, numerical and figural information for four minutes. After a disruption phase of 17 min, they then answered reproduction questions related to the memorized information, choosing one of six response options (scores from 0 to 21; identical for Studies 1–3).

2.6 Procedure

The procedure was similar for Studies 1–3. The participants attended twice: initial training and (after two weeks) retention assessment.

The initial training (week 1) lasted for 120 min. After completing a test concerning retentivity, participants explored the simulation twice. They were then given information and instructions about the start-up procedure. After this, they trained the start-up procedure (fixed-sequence task, contingent-sequence task or parallel-sequence task) with a manual. Following the training, the participants had to perform the start-up procedure four times without help. They were required to produce a minimum of purified gas (Study 1: 200 l, Study 2: 100 l, Study 3: 200 l).

Two weeks after the skill acquisition, the retention assessment (week 3) took place, which lasted for approximately 30 min. The participants were asked to start up the plant up to five times without help (the first trial was used to assess skill retention). The Gaze-Guiding group received the support of the gaze-guiding tool if necessary.

3 Results

The descriptive statistics are given in Table 2. In Study 1, four participants and in Study 2, five participants were excluded due to a too low production outcome at initial training. The groups did not differ in terms of control variables (p > 0.05).

Table 2 Descriptive statistics for Study 1, Study 2 and Study 3
Full size table

3.1 Hypothesis-Testing

To test the hypothesis, repeated measures ANOVAs with the two measurement time points initial training and retention assessment were conducted to measure skill retention.

Study 1: Fixed-Sequence Task (N = 36; Fig. 2 ). The analysis showed no significant difference between the Gaze-Guiding and Control group in terms of production outcome (interaction of time*group: F(1, 34) = 1.97, p = 0.169, \( n_{p}^{2} = 0.0 6 \)). The analysis of start-up mistakes showed that the Gaze-Guiding group made significantly fewer start-up mistakes than the Control group (interaction of time*group: F(1, 34) = 27.28, p < 0.001, \( n_{p}^{2} = 0.0 5 \)).

Fig. 2
figure 2

Production outcome and start-up mistakes for gaze-guiding group (GG) and control group (CG) of study 1 (fixed), study 2 (parallel) and study 3 (contingent)

Full size image

Study 2: Contingent-Sequence Task (N = 35; Fig. 2 ). The analysis showed no significant difference between the Gaze-Guiding and Control group in terms of production outcome (interaction of time*group: F(1, 33) = 0.82, p = 0.372, \( n_{p}^{2} = 0.0 2 \)). The analysis of start-up mistakes showed that the Gaze-Guiding group made significantly fewer start-up mistakes than the Control group (interaction of time*group: F(1, 32) = 17.60, p < 0.001, \( n_{p}^{2} = 0.0 4 \)).

Study 3: Parallel-Sequence Task (N = 40; Fig. 2 ). The analysis showed that the Gaze-Guiding group produced significantly more purified gas than the Control group (interaction of time*group: F(1, 38) = 7.98, p = 0.008, \( n_{p}^{2} = 0.17 \)). The analysis of the start-up mistakes showed that the Gaze-Guiding group made significantly fewer mistakes compared to the Control group (interaction of time*group: F(1, 38) = 4.96, p = 0.032, \( n_{p}^{2} = 0.12 \)).

Studies 1–3 (N = 110). Only start-up mistakes were analyzed for Studies 1–3 as a whole due to different production outcome criteria (and start-up times) in the three studies. The analysis showed that all three Gaze-Guiding groups (treated as one group) made significantly fewer mistakes than the Control groups (treated as one group): F(1, 108) = 29.77, p < 0.001, \( n_{p}^{2} = 0.22 \) (interaction of time*group).

4 Discussion

The results show that the gaze guiding tool supported the cued skill recall after a period of non-use for the correct execution of all three tasks. Using the gaze guiding tool results in comparable start-up mistakes to the initial training and in fewer start-up mistakes compared to the Control group in all tasks. This indicates that the gaze-guiding tool has the potential to support the execution of the tested task types.

Nevertheless, the effectiveness differs between the tasks that are supported. The results indicate that gaze guiding supported a higher production of purified gas for the parallel-sequence task compared to the Control group and when comparing two measurement time points, but this was not shown for the fixed-sequence and the contingent-sequence task. Especially in the contingent-sequence task, the gaze-guiding tool was not found to be effective (production outcome M = 5.25). It can be assumed that gaze guiding decreases in efficacy with increasing number of skill elements in a task. In this respect, the affordances of the contingent sequence were higher than in the other two sequences: The contingent sequence consisted of 23 steps (both conditions; fixed sequence: 13 steps) and the participants had to decide which procedure to follow depending on predefined cues that needed to be acknowledged. Accordingly, an incorrect situation assessment could result in the execution of the wrong follow-up sequence, thus leading to zero production outcome. In sum, it can be concluded that the gaze-guiding format chosen in the presented studies is not a “one-size-fits-all”-sequences solution. Instead, further research is needed to determine the benefits and limitations of different gaze-guiding design principles for different sequences.

The results also suggest that the gaze-guiding tool cannot support the performance with regard to production outcome, which depends on the speed of sequence execution: In all groups, the skill level decreased from initial training to retention assessment, and for the contingent-sequence task in particular, no benefit of the gaze guiding could be shown. The descriptive statistics of the control variable start-up time suggests that the groups using gaze guiding started up WaTrSim more slowly at the retention assessment than at the initial training. This may be a reason why the production outcome was lower at the retention assessment even if the start-up procedure was executed correctly. It can therefore be suggested that gaze guiding is a helpful tool for the correct execution but that there is also some potential for development in terms of optimizing the timing of the gaze guiding. For example, the timing could be adjusted so that gaze guiding is faded in earlier, enabling the participants to act earlier.

In cases of emergencies, incidents or breakdowns, the correct execution is important for controlling the plant safely. The present results represent a promising start point for developing a gaze-guiding tool that is able to support operators in executing SOPs after longer periods of non-use.