Keywords

1 Introduction

Nuclear power plant operators use a complex human-machine interface (HMI) in the form of a control room with control boards containing thousands of indicators and controls (Boring et al. 2013). Operators face the challenging task of monitoring and controlling the plant to ensure safe, efficient, and reliable electrical power production. The operators’ process control task places considerable demands on the operators due to the complex relationships between the multitudes of systems involved with the nuclear power production process. Of the numerous approaches to evaluating HMI interactions, situation awareness is the most prominent method employed (Endsley and Kiris 1995). Acquiring situation awareness (SA) requires many perceptual and cognitive constructs, such as attention, visual perception, working memory, and decision making. All these underlying concepts play a role in building SA, but attention is particularly relevant, since it drives the selection of important information from the plethora of status and control information displayed across the control boards (Wickens 2008). Due to attention’s prominent role in acquiring SA, a new measure of SA based on an attention acknowledgment measure is proposed to augment existing measures of attention allocation, such as eye tracking.

2 ALARA

Within human-computer interaction, ALARA is the acronym for “as low as reasonable assessment” which is a wordplay on an existing ALARA acronym within the nuclear industry for maintaining personnel exposure to radiation to levels “as low as reasonably achievable”. Henceforth, ALARA is in reference to the as low as reasonable assessment, which is intended to convey the idea that simple measures can and should be used over more complicated measures. This is particularly important in complex human-computer interaction domains such as nuclear power plant control room usability studies, which are problematic for research. Operators have limited time due to their demanding work and training schedules and the simulator facilities typically used to support or directly conduct studies have limited availability due to their primary use for training operators. As such, making the most advantageous use of the time researchers have with operators to collect data is crucial. Discount usability and ALARA encompass this rationale and mandate using simple and easy to administer measures as opposed to more complicated measures. This new attention acknowledgment measure is intended to provide a simple and easy to administer method to human factors practitioners following the ALARA ideology. This simple measure is in direct contrast to an existing physiological measure of attention, eye tracking, which is traditionally used in usability studies on nuclear control rooms.

3 Eye Tracking Measure of Attention

Eye tracking is a popular technique to measure attention and its allocation through a visual scene based on the assumption that attention is typically yoked to the gaze position of the eyes (Duchowski 2011). Eye tracking entails measuring the gaze position using infrared camera systems. In the most common technique employed with commercially available eye trackers, the pupil and corneal reflection are captured to calculate where the eye is pointed (Holmqvist et al. 2011). Incorporating relative head position to a visual plane with the calculated direction of the eye provides gaze location within a visual plane.

Eye trackers are a useful research tool; however, they also suffer from several technical issues that make it challenging to use effectively in some environments. First, eye tracking suffers from numerous sources of errors that can lead to difficulty in accurately and reliably measuring each participants’ gaze position. For example, a large portion of commercially available eye trackers rely on infrared cameras to detect the pupil and corneal reflection of each eye to determine the gaze position (Holmqvist et al. 2011). The process of capturing the pupil and corneal reflection suffers when the camera cannot accurately capture either of these two items. Some individual differences that can interfere with this process include drooping eyelids that occlude the pupil, contact lenses that diffuse the corneal reflection, and mascara or eye makeup that generate false corneal reflections (Holmqvist et al. 2011). Additionally, for stationary camera based systems, the head position must also be tracked along with the eye position which suffers from other sources of errors such as excessive participant movement and improper positioning away from the eye tracker (Holmqvist et al. 2011). Both the eye and head position tracking also suffer from interference based on lighting conditions (Holmqvist et al. 2011). Beyond accurately recording the gaze data, the analysis can prove cumbersome for eye tracking. The data generated by eye tracking must undergo extensive processing to manipulate it into a more human digestible format necessary to answer research questions (Holmqvist et al. 2011).

In addition to these general challenges associated with eye tracking, some environments pose specific challenges for eye tracking, such as the HSSL. The HSSL platform has been primarily used to perform applied research in collaboration with nuclear power utilities. As such, the timeline for running the experiments is tight and the cost of these experiments can be large (Ulrich et al. 2016). With the brief time course it is important to collect the needed data as quickly as possible. Often the simpler subjective response measures provided by the operator participants provide the most valuable insights to improve upon the usability of new interface designs undergoing evaluation within the simulator (Ulrich et al. 2016). The HSSL presents a challenge for eye tracking methodologies (Kovesdi et al. 2015), due to its complex three-dimensional environment containing many depth planes and spanning across 45 large displays with thousands of indicators and controls. Furthermore, several technical issues impede the use of eye tracking in this environment, including battery life constraints for the portable eye tracking glasses and their processing units worn by the operator participants and frequent recalibrations required between experimental trials to ensure the accuracy of the eye tracking. Furthermore, some eye tracking units use conflicting infrared camera systems and markers placed on the participant to determine head position. This type of infrared camera system for tracking head position is incompatible with the simulator touchscreen technology which also relies on an infrared camera system embedded within bezels mounted over the displays to detect touch positions. The touch capabilities were rendered functionless when this eye tracking system was operating due to the interference from the conflicting infrared camera systems. From a human perspective of managing participants, the operators do not enjoy wearing the bulky glasses-based systems that are compatible with the HSSL. These issues and others are encountered in other labs as well (Holmqvist et al. 2011), which provides the impetus to develop new measures that can answer the same questions in another fashion. The proposed attention acknowledgment measure would provide a way to simply identify where participants are attending to within an interface without relying on eye tracking techniques.

4 Attention Acknowledgment Measure

Ulrich et al. (2016) proposed a new attention acknowledgment measure consists of presenting visual attention markers that participants are instructed to acknowledge upon detection of the marker in its target state. The act of acknowledging the target via a response serves as an indication that the marker was attended to and underwent the necessary cognitive processing to elicit a response. This measure allows researchers to evaluate human-computer interactions by identifying what aspects of the interface were attended to while performing a task, see Fig. 1, for an example implementation of the attention acknowledgment measure embedded within an interface. The relative proportion of marker acknowledgments serves as an indication of the distribution of attention while the participant interacted with the interface. Within this example implementation, the markers are presented as part of or near interface elements to capture how often and when an individual attended to these interface elements. Using marker acknowledgments is a secondary task, which is inherently accompanied by some primary task intrusion, though this measure was developed to minimize any intrusion. The attention acknowledgment measure provides an easy to implement and assess method for measuring attention allocation. The setup involves overlaying the markers on the interface. The attention acknowledgment software system is configurable and allows the researcher to adjust the presentation and timing of the markers throughout the display. Furthermore, areas of interest can be defined and markers can be assigned to these areas of interest. The markers record correct acknowledgments via mouse clicks to yield total acknowledgment scores for each defined area of of interest. A primary advantage, in line with the concept of discount usability, is the simple to interpret results, which consist of acknowledgment counts for each area of interest.

Fig. 1.
figure 1

Example implementation of attention acknowledgment markers embedded within an experimental interface used to assess situation awareness. Markers are positioned within areas of interested to identify distributions of attention while interacting with the interface.

Full size image

4.1 Attention Acknowledgment Measure Development

This study is the latest in a series of studies conducted to develop an appropriate marker to be used for the attention acknowledgment measure, see Ulrich et al. (2016) for more details on these prior studies. These prior studies focused on establishing the viability of a rotating bar stimulus to serve as an attention marker for assessing where attention was allocated during a simple crosshair tracking task. In these experiments, participants were instructed to maintain the position of a crosshair in the center of its axis while detecting a single rotating marker among a matrix of stationary marker distractors. Participants demonstrated greater accuracy in correctly detecting rotating markers when the marker was located in close proximity to the crosshair task as opposed to located at further distances away in the display. Furthermore, the time to identify rotating markers was shorter for close proximity markers as opposed to markers that were located more distantly away from the crosshair task. Since the primary task required participants to attend to that location within the display, the greater accuracy and shorter times to detect rotating markers near the primary crosshair task over more distant markers provides evidence that acknowledging nearby markers is a potentially viable method for localizing where attention was allocated during a simple primary task. In other words, the markers serve as an effective way to tag where attention is localized within a display.

This current study aimed to extend the prior research in two important ways to further establish the viability of the markers to serve as a measure of attention allocation. First, this study examined whether markers located nearest to the primary task were detected over more distant markers. In order to serve as a marker of the locus of attention for a primary task, it is important to establish that the marker positioned nearest the primary task is detected and acknowledged over more distant markers. Eye tracking was used to capture fixations prior to the detection and acknowledgement of the target marker. Using an established measure of attention, such as eye tracking was done to further verify that participants were in fact directing their attention to the primary crosshair task location and the acknowledged marker and ensure that attention was not directed to the more distant target marker that went unacknowledged. Establishing this pattern of attention corroborates the rationale for using a secondary task such as marker acknowledgment as a measure of attention.

4.2 Method

Eight participants required from an undergraduate psychology program were recruited for the study. The study consisted of a single factor three-level within subjects design. The distance of two target markers was manipulated, resulting in a total of three different conditions of target marker pairs presented to each participant. The conditions differed in the distance each of the two target rotating markers was presented from the primary crosshair task. The test stimuli consisted of the primary crosshair task displayed within a grid of 32 total markers, two of which were in the rotating target state for each trial. Within each trial, the grid of 32 markers were categorized into three circular regions based on each marker’s distance from the primary crosshair task, as can be seen in Fig. 2 to create a near, middle, and far region of markers. The three conditions were defined as target marker pairs consisting of one target marker in the near and middle region, near and far region, and middle and far region. Therefore, the three condition were termed near-middle, near-far, and middle-far region pairs. Participants completed a total of 162 trials in which a pair of target markers were presented at various distances from the primary crosshair task against the grid of nontarget stationary distractor markers. Participants were instructed to select the first target marker they detected during the four second trail. While completing the primary crosshair and secondary target marker acknowledgment tasks during each trial, each participants’ fixations were recorded using a Tobii X2-60 Eye Tracker, which consists of a desktop monitor mounted eye tracking camera.

Fig. 2.
figure 2

Matrix of markers with the crosshair tracking task. Two markers were in the target state and presented at various distances from the crosshair task.

Full size image

General Procedure.

Participants to were instructed to select the first marker they detected in the target rotating state, of two total markers in the rotating target state, as they performed the manual crosshair tracking task. During each trials participants responded with a mouse click to select the rotating marker upon detection.

Crosshair Manual Tracking Task.

The manual tracking crosshair task required participants to maintain the vertical crosshair in the center of the horizontal crosshair while undergoing a pseudorandom disturbance. The horizontal disturbance was generated using a sum of sines method (Lew et al. 2014). Participants were instructed to use the left and the right arrow keys to counteract the disturbance and maintain the vertical crosshair at the center position of the horizontal crosshair. The instructions emphasized the importance of the crosshair task and explicitly stated the lower prioritization of identifying the marker objects in order to ensure participants directed their attention to the crosshair.

Marker Detection Task.

The attention markers were organized into a 4 × 8 matrix spanning the entire display. The markers were categorized into near, mid, and far regions defined by the pixel distance away from the location of the crosshair task during each trial. The display consisted of a full matrix of 32 change detection objects as can be seen in Fig. 1. The marker detection task resembled a standard search task (Wolfe 1994) in which the participant had to find one of the two rotating markers, which provided both reaction times to detect the first rotating marker.

5 Results

To examine the effect of target distance pairings, i.e. near-middle, near-far, and middle-far region pairs of target markers on participant’s acknowledgment rates of the closer target marker, a Chi-square test of independence was calculated. No significant interaction was found X 2 (3, N = 453) = 90.724, n.s.. Across all conditions the participants selected the closer target marker over the more distant target marker in more than 80% of the trials as can be seen in Table 1 below.

Table 1. Percentage of trials in which participants acknowledged the closer of two target markers presented within a matrix of nontarget stationary markers and a centrally located primary crosshair task.
Full size table

The eye tracking gaze data was processed to determine the point of fixation at the time the participant acknowledged a detected target marker with a mouse click selection. Areas of interest were defined as the crosshair, all nontarget markers, and the two target markers. There were numerous trials in which valid eye tracking data was not obtained. Due to eye tracking errors that results in trials without valid eye tracking data, trials in which the recorded fixation point at the time of marker acknowledgment was not categorized as a valid area of interest were removed. This resulted in the removal of 139 out of the total 453 recorded trails, for a data loss total of 30.68%. Table 2 depicts the percentage of trials in which the fixation point at the time of target marker acknowledgment resided on the crosshair, acknowledged target and unacknowledged target.

Table 2. Percent of trials in which participants were fixated upon the crosshair, acknowledged target, and unacknowledged target at the time of a target marker acknowledgement with a mouse click selection.
Full size table

6 Discussion

The results of the current experiment further validated developed attention acknowledgment measure as a method to capture the locus of attention while performing a primary task. Participants reliably selected the closer target marker over the more distant target marker in the vast majority of trials. Indeed, in the near-far condition, participants selected the closer target marker in every trail. This provides strong supporting evidence that participants locus of attention centered around the primary task afforded them the ability to consistently detect and acknowledge the nearest most target marker. This result is quite promising for the attention acknowledgment measure for an important reason. The attention acknowledgment measure relies on a matrix of markers embedded within the interface to assess attention. In this configuration, multiple markers will be simultaneously in the target state to allow the individual to detect and acknowledge any of these target state markers. The location of attention within the interface drives which markers are detected and acknowledged. Therefore, a strong and reliable preference for selecting target marker nearest the locus of attention indicates that marker acknowledgments function as a means to tag where attention is allocated within the interface at a given point in time. The eye tracking results further corroborate these findings.

The eye tracking data illustrates the distinction between two visual strategies participants’ used to detect and acknowledge target markers in close proximity to primary crosshair task in contrast to target makers in far proximity to the primary crosshair tasks. In both the near-middle and near-far conditions, which represent target markers in close proximity to the primary crosshair task, participants fixated upon the crosshair at the time of target marker acknowledgment in 70.70% and 79.80% of trials, respectively. The acknowledged target itself was fixated upon 24.24% and 20.20% of trials for these same two near-middle and near-far conditions. When taken together, this pattern of fixating primary upon the crosshair with a modest percentage of trials fixated upon the acknowledged target marker indicates that participants were actively engaged in the primary task and their locus of attention resided on the primary crosshair task at the time of the target marker acknowledgment. The close proximity of the closest of the two target markers in the near-middle and near-far conditions allowed the locus of attention to encompass the nearest target marker. Participants did not need to redirect attention away from the primary crosshair task to detect these close proximity target markers. The opposite visual strategy is apparent for the eye tracking data in the middle-far condition in which participants were forced to search for the closest target marker since it was outside of their locus of attention on the primary crosshair task. As a result the percentage of trials in which participants fixated upon the acknowledged target marker is much higher than in the near proximity target marker conditions with the 45% and the percentage of trails in which participants fixated upon the crosshair at the time of target marker acknowledgment is much lower at 52.50%.

With this latest study, the attention acknowledgment measure has been thoroughly vetted as a valid measure of attention and is now ready for use in a variety of applications. The next phase of research entails examining the scalability of the attention acknowledgment measure. This evaluation will determine if the attention acknowledgment measure can be scaled up from assessing attention in a simple primary crosshair tracking task to assessing attention in more complicated primary tasks. A microworld simulator consisting of a simplified nuclear process control task is the intended to serve as the next test application. The microworld requires participants to monitor and adjust various plant components to operate a simplified pressurized water reactor to generate steam, turn a turbine, and ultimately produce electrical power. This nuclear microworld application is a considerable increase in complexity from the simple crosshair task used to develop and test the attention acknowledgment measure thus far. The time course for assessing attention is considerably longer in the microworld with time spans on the order of minutes as opposed to seconds with the simple crosshair task. Secondly, the microworld entails a constantly shifting locus of attention as participants monitor the components and make any necessary adjustments while performing the primary electricity production task.