Keywords

1 Introduction

1.1 Operating Procedures in Nuclear Domain

Operating procedures provide a description of the actions that are needed to operate a particular system in a safe and efficient manner. According to some estimates, approximately 70% of the incidents in the nuclear domain have been associated with failures in procedure usage [6]. Procedures can be presented in different formats such as step-by-step text-based instructions, decision trees and flowcharts [12]. Traditionally, operating procedures have been presented in paper format, but today procedures are presented to a larger extent on a computer [8]. Procedures guide operator behavior and set constraints of what is acceptable and what is not. There is a trade-off between too strong level of guidance and too weak guidance: Too strong guidance restricts adaptive way of working, needed in varying situations, but too weak guidance may have detrimental effects on a crew’s performance and shared understanding of the situation [13].

Traditionally, it has been thought that in safety-critical domains, human-system interfaces and procedures are designed with the aim of ensuring safe practice, and the aim of training is to provide to the personnel enough knowledge and ensure that tools are used as planned (see Fig. 1). However, when designing human-system interfaces for complex domains, it is not possible to anticipate all possible work situations, procedures and training. Instead, arriving to the optimal solution requires flexible thinking and problem solving. There is always a tension and gap between work-as-imagined and work-as-done [5]. Human-system interfaces (HSIs) and procedures are designed based on the designers’ view of the work of operative personnel - which is only partly grounded on the work-as-done. Since human-system interfaces and procedures are designed from the perspective of work as designed, personnel must make continuous adaptations in order to manage and accomplish their work tasks. To succeed in this, they need resources for actions provided by their skills and by organizational support. Regarding skills, resilience is especially needed for enabling intelligent use of procedures [7]. The organization, in turn, can support the crew by, e.g., providing technical backup from other workers and professionals, when additional expertise is needed. Even though HSIs and procedures are used in daily work, the larger the gap between work-as-imagined and work-as-done the more problems there may emerge in their usage and a larger role the resources for action plays. Normally, training plays a key role in the pursuit of reconciling the gap between work-as-designed and work-as-done.

Fig. 1.
figure 1

A traditional view according to which training has an important role in aligning the work-as-imagined and work-as-done.

Full size image

The aim of our study was to explore the applicability of the Functional Resonance Analysis Method (FRAM) method for procedure development [4]. The FRAM method can be used as a tool for understanding the reasons behind the gap between work-as-imagined and work-as-done, from the perspective according to which procedures represent work-as-imagined, and the actual crew activities represent work-as-done.

A FRAM model was developed by a free software tool called the FRAM Model Visualizer [3]. The basic idea behind the FRAM is to develop questions that are discussed with those who will use the procedure in their work. The objective of these questions is to identify the critical functions related to nuclear process control and the interactions of these functions. The model was further developed, and its level of appropriateness was tested by analyzing video recordings of simulator tests in which actual operator performance (work-as-done) could be observed. In the future, we will also experimentally evaluate the framework in simulator tests in which two different versions of the procedure are compared in a quasi-experimental set-up.

In this paper, we first present 1) the results of single-person interviews in which a number of key issues related to procedure development were discussed and 2) the results of two focus group meetings in which a FRAM model for one proceduralized activity was developed. The model describes the potential variability of the system in terms of functions. Each function has been defined using six aspects: Input, Output, Time, Resource, Control and Precondition [4]. Second, we present the preliminary results of observations about a simulator study at one of Finnish nuclear power plants, in which the analyzed procedure was used. Video-recordings from thirteen simulator runs were analyzed. The aim of these observations was to verify the conclusions made on the grounds of FRAM modelling.

2 Methods

2.1 Data Collection

Data was collected by focus groups, single-person interviews and video-based observations. Two focus group sessions were arranged with ten participants. Most of the participants were procedure developers and simulator trainers in one Finnish nuclear power plant. The aim of the focus groups was to develop a FRAM model for a particular incident situation and to analyze the procedure designed for managing this situation. In addition, we carried out four single-person interviews by interviewing four experts from three different organizations (from two Finnish NPPs and from the Finnish Nuclear Regulatory Authority).

In order to compare work-as-designed and work-as-done, video-based structured observations were conducted, in which video recordings of simulator test sessions were observed. Simulator runs of thirteen operator crews were analyzed.

2.2 Data Analysis

Interviews were transcribed, and the notes were analyzed in order to explore the debates conducted around the critical functions and their couplings. Notes were made during observations of simulator test sessions, and answers to key research questions were tabulated on an observation table. Audio recordings were first transcribed, and then analyzed by using the grounded theory approach by one researcher.

3 Results

3.1 Description of the Procedure

The target procedure (i.e., triggering of the boron chain) consists of about twenty procedural steps in each of which a particular action is associated. The goal of the procedure is to drive the plant to a safe end state. Most of the operator tasks are inspections of correct functioning of automation systems.

3.2 Interview Results

According to single-person interviews, power companies are responsible for their own procedure development, and STUK, the Finnish nuclear authority, plays a supervisory role. Practices and processes of procedure development somewhat differ from one power plant to another.

According to the experts’ interviews, methods and tools they are using nowadays are quite good and sufficient for their purposes. However, they also felt that that more formal methods and tools for procedure development could be beneficial. Typically, task analysis has been thought as an important method in procedure design. However, according to the interviewees, formal task analysis methods are not systematically used in nuclear power plants. Instead, designers use safety analysis reports and their own less systematic notes as a starting point for their design work. Some of them, however, thought that a systematic application of task analysis might be useful for determining whether the tasks can be performed or not.

The FRAM method was first briefly described to the interviewees. Some of the interviewees were familiar with FRAM, but none of them had ever used it. Differing opinions were expressed regarding the utility of the FRAM method in procedure development: while some were skeptical, others thought that the method may be helpful. For instance, FRAM was seen as potentially useful in the training of new procedures. By using FRAM one could go through all the functions and their justifications, and at the same time, evaluate how these justifications were derived.

3.3 Building a FRAM Model

Incident Scenario Modelled in FRAM.

The scenario starts out as routine testing, that is then interrupted by a sudden, unexpected launch of an emergency shutdown following a control rod drop -failure, leading to a launch of chemical shin (boric acid) to bring the reactor to zero power level and sub-critical state.

The first focus group considered the target scenario from the process point of view. In the first round, fourteen functions for either the Reactor Operator (RO) or the Turbine Operator (TO) were identified. A failed reactor shutdown is the trigger for all the subsequent events in the scenario run. Three partial conditions can lead to the triggering of the boron chain. After the boron release, the operators’ main aim is to bring the reactor to a sub-critical state. SIRM measurements provide critical information for this interpretation.

To build the FRAM model, the main functions and their interrelations were first identified and presented by the FRAM notation (see Fig. 2). Second, the couplings between functions were defined and illustrated by thin lines connecting the functions.

Fig. 2.
figure 2

A simplified FRAM model, describing the critical functions related to the triggering of the boron chain, designed on the basis of the first focus group meeting. The model was developed by a free software tool named the FRAM Model Visualizer [3].

Full size image

The FRAM model was evaluated with regard to the number of couplings (antecedents and consequences), since according to [4], their number is associated with the variability of a particular function, and the variability increases with the number of couplings. Table 1 shows the number of couplings for some functions. Only the most critical upstream and downstream functions are presented. Upstream functions belong to those functions, which have already been completed, and downstream functions belong to those, which follow a particular function.

Table 1. Summary of the variability associated with some key functions.
Full size table

As can be seen, the number of upstream couplings is largest for the functions of triggering of the boron chain and reactor sub-criticality. This means that the variability and adjustments of upstream functions have a quite large effect on these two functions.

Operator’s Perspective.

In the second focus group meeting, the scenario was analyzed from the operators’ perspective. The model became more detailed in this phase, and a larger number of functions were identified and described (see Fig. 3). Most of the functions were allocated to the RO or TO, who perform a majority of the operations, but there were also functions allocated to the shift supervisor (SS), who has an important role as a superior.

Fig. 3.
figure 3

A more complex FRAM model that was designed on the basis of the second focus group meeting, to clarify the operator perspective (operations). The text is not intended to be readable - the aim is only to illustrate the complexity of the model when the number of functions is higher (cf. Fig. 2). The model was developed by the FRAM Model Visualizer [3].

Full size image

All in all, the variability was more pronounced at the beginning of the emergency situation than in its later phases. The participants emphasized that it is particularly important to complete actions correctly immediately once the failure is detected.

Based on the discussions, the participants recommended some improvements to the target procedure, which can be considered as the main benefit of the modelling work. For example, it was found that a description of the end-state of a procedural action was missing from two procedure steps.

3.4 Observations of Simulator Training Exercises

The main goal of the observations of simulator runs was to verify the conclusions made on the grounds of FRAM models.

In the simulator runs, triggering of the boron chain was caused by a failure in the insertion of control rods, requiring the cessation of reactor activity chemically. Once the boron was launched, the RO and TO confirmed the stability of the processes and also followed the decrease of nuclear activity, and finally on the basis of SIRM measurements confirmed that the plant has been brought to a sub-critical state.

All crews applied the failed reactor shutdown procedure in the simulator run. However, contrary to our expectations, only six crews applied the triggering of the boron chain -procedure, which has been particularly developed for this incident. The failed reactor shutdown procedure was used instead, mainly because it is more familiar to the operators, and it includes the major part of the operator tasks found in the triggering of the boron chain procedure. In addition to these two, operators used several other procedures in the simulator run.

The crews performed the functions allocated to them in accordance with the procedures. The Shift Supervisor (SS) accomplished his/her own procedure and supervised the RO’s and TO’s task execution. He/she could also share additional tasks with the other operators, and declare Site Emergency. The SS typically coordinated the use of procedures and checked that all tasks were adequately completed. Some proceduralized tasks were not completed in a chronological order. It was also found that the operators did not accomplish their tasks completely independently, but some of the actions were performed in collaboration with other operators.

All the crews achieved the main goals of the situation. The overall variability of performance time between crews was moderate: the average completion time was 31 min (range 27 to 33 min). However, within the scenario, there was considerable variability in the time of observation for certain critical activities (Fig. 4). For example, the triggering of the boron chain was observed at fastest in 30 s and at slowest in 9 min 10 s after the start of the simulator run. The average duration was 5 min 10 s.

Fig. 4.
figure 4

Elapsed time until observing two critical functions in minutes for thirteen crews. Two crews (i.e., D and H) did not observe one of the two functions. Function 1: Triggering of the boron chain. Function 2: Reactor sub-criticality.

Full size image

We analyzed the amount of variability between crews regarding some critical functions.

The amount of variability seemed to be largest for two functions, identification of triggering of the boron chain and verification of pumping of boron. Other functions for which some variability was observed were time for attainment of reactor sub-criticality (see Fig. 4), performing a reactor shutdown and stopping manually in-pumping.

With regard to the identification of boron release, there were quite big differences between the crews in the time they needed to observe the release. With regard to the verification of pumping of boron, the differences between crews were mainly associated with the adjustment of the water level of the reactor. Some TOs considered it a failure, if the level exceeded 5 m. There were also some variability in initiating reactor shutdown: while some crews tried to drive the control rods into the core several times, some crews did it only once or twice.

Basically, the same sources of variability were identified with both methods (i.e., with observation or modelling). However, it was also found that FRAM modelling identified variability in some functions for which little variability was observed during the simulator runs.

4 Discussion

4.1 FRAM Method in the Analysis of EOPs

We were able to demonstrate the gap between the work-as-design and the work-as-done with regard to one particular procedure/operational condition. We were also able to identify the most critical functions and sources of variance in procedure execution. The results are thus in line with some earlier studies, in which the FRAM model was able to identify the most critical functions and their couplings [9, 14].

In addition, one concrete suggestion for improvement emerged in the focus groups, and the change was planned to be implemented in the proposed procedure step. It was also found that the SS’s role has not been identified in a sufficient degree in existing job descriptions.

FRAM could be especially helpful in the analysis of novel situations for which there is no existing procedures available. The method could also be helpful in safety analysis to provide information on potential successes and failures. The method seems to be quite sensitive, and it is able to identify less obvious, hidden couplings and low variability in functions, which may be difficult to identify by other means. On the negative side, it can also identify couplings, which are less relevant and meaningful. In order to prevent identification of this kind of ‘imagined variability’ [11], it is important to discuss with operators whether they consider the variability real or not - indicating that the results should be always verified against an independent method, preferably with a party with complementary knowledge than those having created the model.

The presentation of the model becomes easily quite complex as the number of functions increases. If the number of functions is quite big, it is quite difficult to present the results in a simple way, and visualizations based on the FRAM Model Visualizer become quite difficult to interpret. A model’s value may thus deteriorate, if modelling is conducted in too detailed level. Overall, there is an urge for more illustrative ways to present the modelling results [9].

One problem is that a hierarchical structure is missing from FRAM models, and therefore it is difficult to zoom in or out on a model. To find the optimal level of detail, it would be necessary to conduct the modelling work at different levels of detail as we did in our work. Also, it should be more explicitly specified the type of a coupling between functions, i.e. what is transferred between functions, matter, energy, information or something else. One possibility would be to use colors to specify different types of couplings.

It was found that the application of the method was quite laborious, and it takes time to learn to apply the method in an efficient manner. Therefore, it is important to allow the personnel participating in FRAM workshops enough time for familiarization and training with the method. These observations are in line with some previous studies on FRAM [8, 14].

Overall, as compared to some earlier studies in other domains in which the method has been used in procedure design, the findings seem to be quite modest [2, 10]. One reason for these unremarkable findings is that the role of procedures is different in nuclear domain: there is more rigor in the development of procedures in nuclear field, and strict adherence to procedures is more or less a norm [1].

5 Conclusions

FRAM models were able to describe operator work as it is done through functions and links between these functions. When comparing the variability in performance, predicted by the FRAM and observed in simulator runs, it was found that the method was able to identify main sources of variability, those small adjustments in a daily work characteristic of Safety-II. Also, the experts thought that FRAM might be useful especially in the early stages of procedure design.

Some areas for development were identified, and it was thought that the method may be more useful for work analysis method than as a means to develop procedures. Yet, the FRAM method has some potential for use in procedure design, especially in designing procedures for novel situations. One of the most favorable feature is its versatility: the method can be applied for many purposes.

Overall, formal methods are able to tighten up the development of procedures, and FRAM, as one example of these methods, may provide added value to the development of procedures also in nuclear domain.