Developing Process Performance Indicators for Emergency Room Processes

Abstract

In a healthcare environment, it is essential to manage the emergency room process since its connectivity to the quality of care. In managing clinical operations, quantitative process performance analysis is typically performed with process mining, and there have been several approaches to utilize process mining in emergency room process analysis. These research provide a comprehensive methodology to analyze the emergency room processes using process mining; however, performance indicators for directly assessing the emergency room processes are lacking. To overcome the limitation, this paper proposes a framework of process performance indicators utilized in emergency rooms. The proposed framework starts with the devil’s quadrangle, i.e., time, cost, quality, and flexibility. Based on four perspectives, we suggest specific process performance indicators with a formal explanation. To validate the applicability of this research, we present a case study result with the real-life clinical data collected from a tertiary hospital in Korea.

This work was supported by clinical research grant from Pusan National University Hospital and the MSIT (Ministry of Science and ICT), Korea, under the ITRC (Information Technology Research Center) support program (IITP-2018-0-01441) supervised by the IITP (Institute for Information & communications Technology Promotion).

1 Introduction

In a healthcare environment, systematic management of clinical processes is inevitable since it is closely connected to medical service quality [13]. In particular, the Emergency Room (ER) process is one of the main processes in the healthcare system [18, 20]. The ER process is as highly complicated as other major healthcare processes, including outpatients and inpatients [1, 15]. Besides, it is often overcrowded and out of control since it is exposed to the risk of unexpected factors [6]. For these reasons, comprehensive process management for efficient operation and care quality management is considered to be essential in the ER context.

Performance measurements are of paramount importance in managing processes [7]. Above all, the quantitative performance analysis is getting a keen interest due to the abundance of data and advances of data-driven methods [2, 9]. Process mining, i.e., a relatively young discipline focused on deriving knowledgeable process-related insights from event logs, has enabled to perform data-driven process analyses [14]. In the ER environment, there has been numerous research utilized in process mining. For example, Rojas et al. [20] proposed the comprehensive question-driven methodology for analyzing the ER processes using process mining with the four schemes, including process discovery, conformance analysis, performance analysis, and organization analysis. Despite its novelty and applicability in practice, process performance indicators to directly assess the ER process are lacking. In other words, it has revealed what to analyze using process mining, but it is insufficient to provide details on what aspects of the emergency room will be analyzed for a specific purpose. Therefore, it is necessary to develop indicators and analytical methods that can measure ER performances (e.g., time, cost, quality, and flexibility) considering the purpose of analysis.

Using the devil’s quadrangle [7], i.e., time, cost, quality, and flexibility, this paper proposes a framework of process performance indicators (PPIs) utilized in ER processes. Our previous work suggested PPIs that can measure the effects of business process redesigns using process mining functionalities [3]. Based on the approach, this study develops performance indicators for evaluating ER process management. To this end, we first prepare PPIs that can be analyzed using clinical event logs and verify them with a thorough discussion with clinical experts in the emergency department. To validate the applicability of our framework, we performed a case study with the real-life clinical data collected from a tertiary hospital in Korea.

The remainder of this paper is organized as follows. Section 2 summarized related works. Then, in Sect. 3, we describe our framework and process performance indicators in detail. Section 4 shows the application of the proposed framework in the case study. Finally, Sect. 5 concludes the paper and describes future directions.

2 Related Work

There have been numerous research efforts to apply process mining into a healthcare setting, e.g., outpatients, inpatients, and emergency room processes [8, 14, 15, 17,18,19,20]. Especially, regarding the ER process, existing research have presented several approaches to analyzing the care flows and the insights by applying them. In [20], the authors proposed a comprehensive question-driven methodology, which provides a data reference model, frequently-posed questions, and the detailed stages to solve the questions using process mining. Also, there was an approach to evaluate the capabilities of process mining to the ER process on the stroke case [8]. [18] proposed the six-phase method for performance analysis of emergency room episodes. However, they do not focus on detailing process performance indicators to evaluate the ER process.

Regarding the process performance indicators, some related works exist in the context of the quantitative process analysis in business process management. [11] proposed how to determine the performance indicators and suggested six requirements on indicators: quantifiability, sensitivity, linearity, reliability, efficiency, and improvement-oriented. [5] proposed PPINOT, i.e., a metamodel to define PPIs comprehensively, which includes how to connect elements in business processes and PPIs and provides an implementation of the metamodel using description logics. [3] presented a framework of process performance indicators to assess the effects of business process redesigns in four perspectives: time, cost, quality, and flexibility.

The studies reviewed above do not sufficiently suggest process performance indicators to be utilized immediately to manage the ER processes using process mining techniques.

3 Method

In this section, we explain a framework for defining emergency room process performance indicators (i.e., ERPPIs) for managing emergency department processes. Figure 1 presents an overview of the proposed framework. In developing EDPPIs, this research employed four perspectives mainly utilized in the quantitative business process performance analysis, i.e., devil’s quadrangle: time, cost, quality, and flexibility [7]. Starting from the four perspectives, we suggested 9 performance indicators as presented in Fig. 1. To define these indicators, we prepared process performance indicators that can be derived from clinical event logs collected in electronic health records and by process mining techniques. After that, they were verified by clinical experts in the emergency department, whether they are meaningful and applicable in practice. The remaining section will describe the clinical event logs as preliminaries and the detailed explanation about the proposed ERPPIs for each perspective.

Fig. 1.

Overview of the proposed framework

3.1 Preliminaries

Prior to detailing on PPIs, we explain clinical event logs, i.e., the inputs of process mining in healthcare, utilized for a formal definition of them. Definition 1 presents a formal explanation of event logs, variants, emergency values, and activity relations.

Definition 1 (Event, Case, Event Log)

[Event, Case, Event Log] Let A, O, T, ET be a finite set of activities, originators, timestamps, and event types, respectively. $E= A \times O \times T \times ET$ is the set of events, i.e., combinations of an activity, an originator, a timestamp, and an event type (e.g. $e_i = \{ a_i, o_i, t_i, et_i \})$. Let L be an event log which has a multiset of traces, and $C = \{ c_1, c_2, c_3, ... , c_k \}$ be the set of cases. A trace ${\sigma }_k = \{ e_{k,1}, e_{k,2}, e_{k,3}, ..., e_{k,n} \}$ is mapped into a case $c_k$, where $e_{k,n}$ denotes n-th event of the k-th case. Let $V = \{ v_1, v_2, v_3, ..., v_o \}$ be a finite set of variants where $v_i$ is a nonempty subset of all possible combinations of activities. var is a function mapping each case to a variant (e.g. $var(c_k)$ is the variant of k-th case). Let $EM = \{ 1, 2, 3, 4, 5 \}$ be a finite set of emergency values. emer is a function mapping each case to a emergency value (e.g. $emer(c_k)$ is the emergency value of k-th case). Activity Relation $(AR) \subseteq A \times A$ is a set of activity relations where two events have causal relations (e.g. $ar_{k,ij} = \{(a_{k,i}, a_{k,j} )|a_{k,i}, a_{k,j} \in A \}$ where $e_{k,i}$ is the predecessor of $e_{k,j}$ (i.e. $e_{k,i} > e_{k,j}))$.

3.2 Time-Related ERPPIs

Most organizations aim at managing their business processes by improving time-related indicators, such as decreasing the processing time and waiting time. In the clinical processes for the emergency room, time-related performances are essential since they are highly relevant to the clinical results. In this research, we suggest three indicators in the time perspective: length of stay for patients (ERPPIT1), length of stay for patients of a variant (ERPPIT2), and cycle time of a clinical activity (ERPPIT3). Followings are the definitions of time-related process performance indicators (Definitions 2, 3 and 4).

Definition 2

(ERPPIT1: Length of stay for patients). Let LOS be the length of stay for patients in an event log L.

- $LOS = \{t_{k,n}-t_{k,1}|\forall _{0< k \le |c|} \forall _{0 < i \le n} c_k \in L \wedge e_i \in c_k \}$

Definition 3

(ERPPIT2: Length of stay for patients of a variant). Let $LOS(v_1)$ be the length of stay for patients of a variant $v_1$ in an event log L.

- $LOS(v_1) = \{t_{k,n}-t_{k,1}|\forall _{0< k \le |c|} \forall _{0 < i \le n} c_k \in L \wedge e_i \in c_k \wedge var(c_k)=v_1 \}$

Definition 4

(ERPPIT3: Cycle time of a clinical activity in a event log). Let $CT(a_1)$ be cycle time of a clinical activity $a_1$ in an event log L.

- $CT(a_1) = \{t_{k,i+1}-t_{k,i}|\forall _{0< k \le |c|} \forall _{0 < i \le n} c_k \in L \wedge e_i \in c_k \wedge a_{k,i}=a_1 \}$

The first two time-related indicators, i.e., ERPPIT1 and ERPPIT2, are the performance of passing the entire clinical processes from start to end. Thus, it is necessary to manage them since they are highly related to the congestion of the emergency department. ERPPIT3 measures the performance of leading clinical activities, including treatments, medical tests, and consultations, which are of great importance to prohibit bottlenecks in the process. All time-related indicators are combined with aggregation functions, e.g., average ($f^{AVG}$), median ($f^{MED}$), minimum ($f^{MIN}$), and maximum ($f^{MAX}$).

3.3 Cost-Related ERPPIs

As far as the cost-related PPIs are concerned, clinical event logs have to include cost information to identify how expensive the emergency clinical process is for operating it. However, it is often unfeasible to obtain clinical data enhancing the cost information. Therefore, based on the assumption that all resources are full-time equivalents, we develop an alternative indirect cost-related PPI, i.e., the total number of originators in the log (ERPPIC1); thus, it can be calculated from the commonly available clinical event logs. Definition 5 gives the formal definitions of the cost-related PPI.

Definition 5

(ERPPIC1: Total number of originators). Let $N_o$ be the total number of originators in an event log L.

- $N_o = \sum _{q=1}^{m} {\left\{ \begin{array}{ll} 1 &{} \quad \text {if } O_q \in \{ \sum _{0< k \le |c|}\sum _{0 < i \le n}\pi _{o}(e_{k,i}) \} \\ 0 &{} \quad \text {otherwise} \end{array}\right. }$

With the assumption that wages are similar among full-time employees, ERPPIC1 becomes the significant cost indicator since labor cost is usually essential.

3.4 Quality-Related ERPPIs

In business process management, quality-related performance analysis can be differentiated as external and internal aspects. More in detail, the external quality focuses on client’s angles (e.g., patients), while the process participant’s viewpoint is relevant to the internal quality. In an emergency room environment, external quality can include patient’s satisfaction and clinical results (e.g., mortality rate or re-visit rate). These indicators, however, cannot be easily derived since clinical event logs generally do not hold the relevant information. Therefore, this research proposes three indicators in the internal quality perspective that can be measured from clinical data.

First, the workload of resource (ERPPIQ1) indicates how much works an originator gets. In an emergency room environment, workload management is essential since it is highly crowded in general. Definition 6 gives a detailed explanation of how to measure the workload of resources. As described in the Definition, it requires two different values within a specific time period ($tp_j$): frequency of events started ($o_q, start, tp_j$) and terminated ($o_q, complete, tp_j$) by a specific originator ($o_q$). In the initial stage ($tp_1$), the workload is computed by checking the difference between the number of started and completed events. From the second time period ($j>1$), the workload of the immediately preceding stage is also considered.

Definition 6

(ERPPIQ1: Workload of Resources). Let $C = O \times T \times TP \rightarrow \mathbb {R}$ be a function that computes the number of events from a log L for a given resource $(o_q \in O)$, a type $(\{start,complete \in T\})$, and a time period $(tp_j \in TP)$. $C(o_q, start, tp_j)$ denotes the number of events started by the resource $o_q$ within the time period $tp_j$. Here, if the event type only holds complete, the complete time of the immediately preceding event in the same case becomes the start time. $C(o_q, complete, tp_j)$ denotes the number of events completed by the resource $o_q$ within the time period $tp_j$. The workload for the resource $o_q$ within the time period $tp_j$ is defined as follows.

- $Workload_{o_q, tp_j} = \sum _{0< q \le |m|} \sum _{0 < j \le |p|} {\left\{ \begin{array}{ll} C(o_q, start, tp_j) - C(o_q, complete, tp_j) &{} \text {if } j=1 \\ Workload_{o_q, tp_{j-1}} \\ \qquad +\,C(o_q, start, tp_j) - C(o_q, complete, tp_j) &{} \text {otherwise} \end{array}\right. }$

The second quality-related indicator, i.e., ERPPIQ2, shows how diverse are the variations of the cycle time in the emergency clinical process. It implies to identify whether the process is stable and standardized. Definition 7 presents the formal explanation of ERPPIQ2.

Definition 7

(ERPPIQ2: Variation of length of stay for patients). Let $\sigma (LOS)$ be the standard deviation of the length of stay for patients in an event log L.

- $\sigma (LOS) = f^{STD}(\{t_{k,n}-t_{k,1}|\forall _{0< k \le |c|} \forall _{0 < i \le n} c_k \in L \wedge e_i \in c_k \})$

The last indicator of the quality perspective, i.e., ERPPIQ3, is closely linked to the emergency department. In the initial step of the emergency room process, the patient’s emergency degree is measured by the triage activity, e.g., Korean Triage and Acuity Scale (KTAS) [12], and hospitals provide the clinical services based on it. That is, it is a fundamental policy that higher emergency of the patient acquires a higher priority. In this paper, we suggest the triage-based patient response rate by considering two policies: (1) first come, first served (FCFS) and (2) more urgent people are allowed to be treated first even if they are late. Definition 8 presents the formal explanation of this indicator.

Definition 8

(ERPPIQ3: Triage-based patient response rate). Let $emer(c_k)$ denotes the emergency degree of a case $c_k \in C$. Assume that all cases $c_k \in C$ are sorted by the completed time of their first event $t_{k,1}$. Then, triage-based patient response rate $TPR_{tp_j}$ for a specific time period $tp_j \in TP$ is formally defined as follows.

- $TPR_{tp_j} = 1 - \frac{\sum _{0 < k \le |c|} {\left\{ \begin{array}{ll} 1 &{} \quad \text {if } t_{k,2}> t_{k+1,2} \\ &{} \qquad \wedge \text { }emer(c_k) > emer(c_{k+1}) \\ &{} \qquad \wedge \text { }min(tp_j) \le t_{k,2}, t_{k+1,2} \le max(tp_j) \\ 0 &{} \quad \text {otherwise} \end{array}\right. }}{|c| - 1}$

3.5 Flexibility-Related ERPPIs

To assess the flexibility of emergency room clinical processes, we introduce two indicators, i.e., ERPPIF1 and ERPPIF2, which evaluate whether a process can react to changes. Definitions 9 and 10 provide the formal explanation of each flexibility-related indicator.

Definition 9

(ERPPIF1: The total number of variants in a log). Let $N_v$ be the total number of variants in the log.

- $N_{v} = \sum _{r=1}^{o} {\left\{ \begin{array}{ll} 1 &{} \quad \text {if } V_r \in \{ \sum _{0<k<|c|} \pi _{var}(c_{k}) \} \\ 0 &{} \quad \text {otherwise} \end{array}\right. }$

Definition 10

(ERPPIF2: The total number of relations in a process model). Let $N_{ar}$ be the total number of relations in the process model.

- $N_{ar} = \sum _{0<k \le |c|} \sum _{0<i<j \le n} {\left\{ \begin{array}{ll} 1 &{} \quad \text {if } c_k \in L \wedge e_{k,i}, e_{k,j} \in c_k \wedge a_l, a_m \in A \\ &{} \qquad \wedge e_{k,i} > e_{k,j} \wedge \pi _{a}(e_{k,i})=a_l \\ &{} \qquad \wedge \pi _{a}(e_{k,j}) = a_m \\ 0 &{} \quad \text {otherwise} \end{array}\right. }$

These two indicators signify to identify whether the process model has an ability to handle a higher variety of cases with different control-flows.

4 Case Study

To demonstrate the applicability of the proposed process performance indicators, we performed a case study with the real-life clinical data collected from the electronic health records (EHR) system in a tertiary hospital in Korea.

4.1 Context

In the case study, we collected the clinical event log of the ER patients during 2018. The event log contained 15 medical tasks: entry, basic treatment, first aid treatment, other treatment, diagnostic test, visual test, consultation, cooperation request, cooperation arrival, the decision on hospitalization or discharge, prescription request, prescription receiving, certificate issuing, discharge, and hospitalization. Also, in the log, around 460,000 events were included for about 30,000 patients who visited the emergency room. To measure the PPIs, we used the Fluxicon Disco [10] and ProDiscovery [16].

4.2 Results

Time Perspective. For the time perspective, we first measured the length of stays for ER patients, i.e., ERPPIT1. As a result, we identified that patients stayed 8.35 h in the ER on average; while the median is 4.8 h. Figure 2 presents the distribution of the length of stays of patients. In the figure, we can identify that most of the patients, i.e., 97%, stayed just within a single day; while only 3% of patients stayed over a day. Here, interestingly, we were able to find out a couple of patients who remain for around 40 days.

Fig. 2.

The length of stay analysis result using the dotted chart

Concerning the second time-related indicator, there was a significant difference of variants according to its characteristic on whether it connects to hospitalization or discharge. More in detail, inpatients tended to stay longer than discharged patients (hospitalized: 11.52 h and discharged: 5.42 h on average).

Also, we measured the cycle time for clinical activities in the ER process, i.e., ERPPIT3, and several points required relatively long cycle time: decision on hospitalization and discharge, hospitalization, and prescription receiving. To identify the preceding activities that cause problematic points, we performed the model-based performance analysis as depicted in Fig. 3. As a result, we identified that it takes a long time from the diagnostic test to hospitalization; besides, it was followed by the decision on hospitalization and discharge to discharge, visual test to prescription request, and diagnostic test to the decision on hospitalization and discharge.

Fig. 3.

The time-related analysis result with the process model

Cost Perspective. Regarding the cost perspective, we measured the number of doctors. From the log, we identified that there were 6.8 doctors on average in a single day. More in detail, we identified that it has a time pattern of the value; thus, there was a variation according to time. Figure 4 depicts the resource-related analysis result. In the figure, blue dots and grey lines signify the number of doctors working in the emergency room by each hour and patients staying in the ER, respectively. Thus, it was confirmed that there were many medical personnel as the number of patients increased during the daytime.

Fig. 4.

The resource-related analysis result for the cost and quality perspectives (Color figure online)

Quality Perspective. With regard to the quality perspective, we calculated the workloads for doctors in the ER, i.e., ERPPIQ1. Due to the limitation of the collected data, the number of patients in the emergency room, rather than the number of medical activities performed by the physician, was assumed as a workload. As a result, we identified that each person averagely managed 11.9 patients in a single day. Also, there was a variation of workload by each hour as depicted in Fig. 4. More in detail, there was a trend that clinicians were busier at dawn despite the small number of patients, as shown in the yellow line.

Then, the variation of the length of stay for ER patients was computed, i.e., ERPPIQ2, and the standard deviation was 8.61 h. More in detail, we analyzed that the value of hospitalized patients (9.99 h) was more diverse than discharged patients (5.44 h).

For ERPPIQ3, the triage-based patient response rate was measured, and the average value was 0.74. To this end, we performed further analysis with the dotted chart, as depicted in Fig. 5. The figure shows the response pattern of the well-managed day with the high value of 0.91. As presented in the blue box in the figure, we observed that a more urgent person, i.e., the red dots, was treated faster than a less urgent person, i.e., the yellow dots.

Fig. 5.

The triage-based patient response analysis result (Color figure online)

Flexibility Perspective. Lastly, in the flexibility perspective, we identified that there were 25,004 variants in the log (i.e., ERPPIF1). Thus, considering the total number of patients, i.e., 30,000 patients, it was confirmed that most people have different variants from each other. Then, we produced the ER process model with a discovery technique, i.e., frequency mining [4], and measured the number of relations in the model, i.e., ERPPIF2. As a result, there were 188 activity relations in the discovered model.

Discussion with ER Experts. These case study results were discussed with ER experts to assess the overall state of the ER and identify the rationales of them. First, the hospital has well managed the time-related indicators. Notably, it was confirmed that a couple of points that take a long time are the results of not determining hospitalization at night for the convenience of other inpatients. Regarding the cost perspective, the number of clinicians was maintained at the appropriate level. In terms of the quality, the variation of the length of stay and the response to the emergency patients were satisfactory, but the proper allocation of the work considering the number of patients was necessary. Finally, we confirmed that the emergency room was operated with sufficient flexibility to care for various patients.

5 Discussion and Conclusion

In this paper, we suggested ERPPIs (Emergency Room Process Performance Indicators) for the emergency room to assess emergency room processes based on four perspectives: time, cost, quality, and flexibility. Also, we validated the proposed indicators by applying to the real-life clinical event log collected from a tertiary hospital in Korea.

Our work has an outstanding implication of presenting a method to directly measure the performance of the ER processes with the four perspectives, unlike other existing general frameworks, which consist of data preparation, preprocessing, analysis, and evaluation. As such, we believe that our framework has sufficient applicability in practice, and its results can become a solid basis for decision making in the ER.

Our work also has several limitations. First, it is necessary to advance ERPPIs by diversifying and elaborating them. The indicators presented in this study are just the most fundamental metrics. Therefore, it should be expanded in consideration of various aspects of the context in the emergency room, as well as aspects that can take advantage of the advanced techniques of process mining. Besides, despite that the criteria for determining the analysis results are needed to evaluate the emergency room process, this study did not present them but merely chose to discuss them with the medical experts. Therefore, we will work on developing a method to derive the appropriate criteria, based on the results of analysis by period (e.g., weekly or monthly).

Furthermore, for future works, we will implement a tool to support the application of the proposed framework. Also, we will make a connection of this research to the process redesign that presents improvement plans according to the results of the performance analysis. Lastly, more case studies will be conducted to validate our framework.