Keywords

1 Introduction

Many enterprise repositories comprise large collections of process models, which represent business processes serving to achieve specific goals with their corresponding actors, tasks, and decisions. Usually, process models vary in respect to their quality and level of granularity. As a consequence, these models face a wide range of challenges affecting model comprehensibility and error probability. However, the comprehensibility of process models is crucial for enterprises to enable an overall understanding of respective processes for all involved actors.

A process model may be represented either as a textual or a graphical documentation, whereas the latter provides specific advantages compared to textual descriptions [1]. Focusing on the graphical specification of processes, there exist various modeling languages like, for example, EPCs [2], BPMN [3], and Flow Chart [4]. Each of these languages is defined through its syntax and contains a set of graphical symbols for documenting processes. Many studies have shown that the use of graphical symbols foster process model comprehension [5].

Putting an emphasis on EPCs, this paper presents an experiment on process model comprehension using eye tracking. Usually, processes expressed in terms of EPCs consist of three different elements, i.e., functions, events, and logical connectors. In general, an EPC is a chain of alternating events and functions. Their specification, in turn, primarily describes the business logic of the process, thus having a positive impact on process model comprehension [2].

The presented experiment is part of a series of experiments using a conceptual framework that incorporates concepts from cognitive neuroscience and psychology for process model comprehension [6]. Regarding the influence of process modeling expertise on model comprehension, the results are similar compared to a priorly conducted experiment addressing BPMN process models. However, the results additionally revealed that process models expressed in terms of EPC are perceived differently than BPMN models. Due to the fact that the obtained results may be considered as preliminary, however, they provide promising insights with respect to the reading and comprehension of process models.

The paper is organized as follows: Sect. 2 describes the context of the experiment. The experimental setting and operation is introduced in Sect. 3. In Sect. 4, the obtained results are analyzed and hypotheses are tested for statistical significance. Finally, Sect. 5 discusses related work and Sect. 6 summarizes the paper.

2 Context of the Experiment

The complex biological and cognitive processes in the head of an individual, whether being conscious or subconscious, ultimately decide how the environment is perceived and, hence, influence the decisions on the further actions to be taken [7]. In the domain of process modeling, the application of cognitive neuroscience and psychology entails auspicious prospects [8,9,10]. Focusing on issues related to the comprehension of process models, we currently conduct a series of experiments, utilizing advantages of different concepts from cognitive neuroscience and psychology. Among others, we strive for a comparison between existing process modeling languages to yield the perceived pros and cons of respective languages. In order to achieve these objectives, we make use of a conceptual framework we developed [6]. Table 1 presents concepts, for which we already conducted experiments using the conceptual framework. Furthermore, Table 1 illustrates the number of involved subjects and analyzed process models. Table 2, in turn, depicts the evaluated process modeling languages and relates them to the mentioned concepts, these languages have been evaluated with. In more detail, we observed and measured changes in the electrodermal activity as well as in the heart rate of subjects while confronting them with different modeling related issues (e.g., level of difficulty, used process modeling languages) with respect to the process models [11, 12]. Furthermore, for maybe lowering the needed amount of mental effort and to may reduce the perceived difficulty, while reading and comprehending process models, we applied the Cognitive Load Theory and the Construal Level Theory in this context [13, 14].

Table 1. Experiments using the conceptual framework
Full size table
Table 2. Process modeling languages evaluated with the conceptual framework
Full size table

3 Experimental Setting

The expertise in process modeling might be a factor influencing the comprehension of process models. This leads us to the following research question:

figure a

To investigate this research question, an experiment using eye tracking is conducted. Note that the experiment is conducted as a quasi-experiment since the subjects are assigned by judgment with respect to the level of expertise a subject has in process modeling [15]. Generally, eye tracking is a reliable measurement method to determine differences between subjects while comprehending a visual stimulus (e.g., picture) [16]. Furthermore, the use of eye tracking reveals insights about cognition as well as cognitive development of a subject and provides non-invasive indices of brain functions. By using this measurement method, we want to obtain insights into how subjects are comprehending process models, while, for example, monitoring attention shifts. In our context, the eye movements of the participating subjects were recorded, while three EPC process models had to be comprehended, along with a set of comprehension questions. In the experiment, the recorded types of eye movements were fixations, saccades, and gaze paths [17]. Fixations are very slow eye movements at a specific point in a stimulus, whereas saccades represent fast eye movements. To be more precise, a saccade constitutes a change of fixation of the eyes in a stimulus. The chronological order of fixations and saccades creates a gaze path. It is found in eye tracking experiments that experts comprehending a stimulus are more likely to have a smaller number of fixations, saccades, and consequently a shorter gaze path length compared to novices [18].

3.1 Hypothesis Formulation

The following six hypotheses were derived to investigate whether or not expertise in process modeling has a positive impact on process model comprehension. More precisely, we focus on the question whether intermediates (i.e., individuals with more expertise in process modeling) are more effective regarding the comprehension of process models expressed in terms of EPCs compared to novices:

figure b

3.2 Experimental Setup

This section describes the subjects and objects of the experiment, together with the independent and dependent variables.

Subjects. There were no prerequisites for participating in the experiment. Therefore, subjects with diverse backgrounds (i.e., students, academics, and professionals) were invited to participate in the experiment. Subjects were informed that the experiment takes place in the context of process model comprehension and anonymity was guaranteed for all subjects. As done in other scientific fields, the categorization into groups (i.e., intermediates and novices), in turn, was accomplished by a median split, i.e., based on time spent on process modeling provided by self-reporting of the subjects.

Objects. In the experiment, three process models reflecting different levels of model difficulty (i.e., easy, medium, and hard) are presented to the subjects. In particular, the used process models were created in collaboration with several experts in the domain of process modeling. The models were expressed in terms of EPCs. The easy process model comprises only basic modeling elements (i.e., events and functions). With rising level of difficulty, new EPC elements were introduced and the total number of elements was increased. The eye movements were recorded throughout these comprehension tasks. After comprehending a process model, four true-or-false comprehension questions, referring solely to the scenario semantics, had to be answered by the subjects. The questions were used to evaluate whether or not the process models were correctly interpreted.Footnote 1 In addition, experts and novices in the domain of process modeling, who were not participating in the experiment, ranked the used process models with respect to their level of difficulty. Moreover, they were asked to compare the EPC models with the priorly used BPMN models to ascertain a comparability between these two modeling languages [6].

Independent variables. The experiment contains two independent variables; i.e., the level of difficulty and expertise level of subjects.

Dependent variables. Regarding the level of difficulty, the dependent variables include the duration needed for comprehending a process model, achieved score based on the comprehension questions, and needed response time for answering the questions. In the context of eye tracking, the dependent variables include the number of fixations, number of saccades, and length of the gaze path. Figure 1 summarizes the research model of the experiment.

In general, since the experiment consists of pure comprehension tasks, the results may be considered as preliminary, i.e., their generalization needs to be confirmed by additional experiments.

Fig. 1.
figure 1

Research model of the experiment

Full size image
Fig. 2.
figure 2

Experimental design

Full size image

3.3 Experimental Design

The experimental setting is based on the guidelines set out by [19]. The experiment was conducted in a designated lab at Ulm University. Prior to the experiment, three pilot studies with 12 subjects were conducted to improve the experimental design and material as well as to eliminate potential ambiguities, e.g., optimization of the used process models. Figure 2 shows the procedure of the experiment: An introduction was given, subjects had to sign a consent form, and demographic data (e.g., expertise in process modeling in general, familiarity with particular modeling languages) were collected. Subsequently, the eye tracking appliance was calibrated and subjects completed a tutorial in order to familiarize them with the functionality of the eye tracker and the procedure of the experiment. Therefore, an exemplary task based on the actual experiment was shown to the subjects. The experiment could be done either in English or German. After completing these mandatory steps, subjects needed to comprehend three EPC process models. First, the process model reflecting an easy level of difficulty was presented, followed by the medium and, finally, the difficult model. After studying each of the models, subjects had to answer four comprehension questions related to the respective model. The questions could be answered with ‘true’, ‘false’, or ‘uncertain’. When answering the questions, the process models were not visible. The pure comprehension of process models (i.e., without any guidance) is uncommon, but we wanted to deliberately disclose the approaches for the pure comprehension on EPC process models. Finally, subjects could provide textual or oral feedback.

Instrumentation and data collection procedure. For eye tracking, we used the SMI iView X Hi-Speed system at a sampling rate of 240 Hz.Footnote 2 The tracking appliance was placed in front of a monitor that provides the process models to subjects. Subjects used a keyboard with three predefined keys providing the options for answering the comprehension questions. Eye tracking data collected during the experiment were analyzed, visualized, and exported with SMI BeGaze software [20]. Demographic data and qualitative feedback were gathered with questionnaires.

3.4 Data Validation

Overall, data from 36 subjects (12 female participants) were collected, i.e., 20 students, 12 academics, and 4 professionals participated. Specifically, 19 of them were computer scientists; additionally, 4 psychologists, 4 economists, and 4 social workers participated. Finally, 5 subjects did not provide a statement. The median of hours spent for process modeling, which is used to divide the participants in intermediates and novices, was 20.5 h. This resulted in a number of 21 intermediates and 15 novices. Furthermore, eye tracking data from 4 subjects were excluded due to invalidity, i.e., eye movements were not recorded properly. Finally, the group of novices included 14 subjects and the one of intermediates consisted of 18 subjects.

4 Data Analysis and Interpretation

Table 3 presents mean and standard deviation (i.e., STD) for novices and intermediates. It shows the process model comprehension duration (in ms) as well as the achieved answering scores. Thereby, specific values to each answering option were assigned, i.e., ‘true’ = 1, ‘false’ = \(-1\), and ‘uncertain’ = 0. Furthermore, response times for answering related questions (in ms), number of fixations as well as number of saccades and, finally, length of the gaze path (in px) are listed in Table 3 (i.e., theoretical factor and operationalization of factor).Footnote 3

Table 3. Obtained experimental results
Full size table

Generally, all values, except the answering scores, increase with rising level of difficulty, as expected by us. For the process model with an easy level of difficulty, the results reveal that intermediates are more effective in terms of process model comprehension compared to novices. The comprehension duration is shorter and, in average, intermediates made less mistakes in answering the questions. Furthermore, they needed less fixations and saccades, resulting in a shorter gaze path. Concerning the process model with the medium and the one with the highest difficulty, the results between novices and intermediates are approaching a similar level and only slight differences are observable. It appears to be that novices and intermediates perform equally regarding the comprehension of EPC process models. However, it is noteworthy that the results do not differ significantly considering the fact that few novices have had no experience in EPCs at all. Figures 3, 4, 5 and 6 show selected results of the experiment. Figure 3 indicates that the time needed for process model comprehension increases with rising level of difficulty. Figure 4 illustrates that the answering scores are decreasing with rising level of difficulty. Moreover, for the process models with the medium and highest level of difficulty, novices achieve a slightly better score. The response times for answering the questions increase with rising level of difficulty (cf. Fig. 5). Especially between the easy and medium process model, a difference is discernible. The fixation number for intermediates is always lower than the number for novices (cf. Fig. 6). Altogether, EPC process models seem to be fairly comprehensible without previous knowledge.

Fig. 3.
figure 3

Comprehension duration

Full size image
Fig. 4.
figure 4

Answering score

Full size image
Fig. 5.
figure 5

Response time

Full size image
Fig. 6.
figure 6

Fixation number

Full size image

4.1 Hypotheses Testing

The stated hypotheses (cf. Sect. 3.2) are tested for statistical significance using a Student’s t-test (cf. Table 4). Particularly, we refer to the rule of thumb, according to the literature, that the use of the t-test is appropriate at a sample size of \(n > 30\) [21]. In this context, a successful t-test (with \( p < p_{0} \) at risk level \( \alpha = 0,05 \)) will reject a null hypothesis [22]. Only for hypothesis \(H_{1,2}\) for the easy process model, a significant result emerged. In addition, tendencies are discernible in \(H_{1,1}\) and \(H_{1,4}\). However, the results confirm our observations from the first experiment that process model comprehension not necessarily correlates with expertise in this domain [6]. This is also evident in the results obtained from the process models with a higher level of difficulty.

Table 4. Hypotheses testing results
Full size table

4.2 Threats to Validity

Any experiment bears risks and, hence, its levels of validity need to be checked and discussed. The level of difficulty reflected by the respective process models constitutes a threat to validity. The gap between the single difficulties might be not alike. Furthermore, the comprehension of process models without any concrete task or guidance is uncommon in practice. Moreover, process models must be memorized for answering the comprehension questions. Consequently, the risk arises that process models are wrongly memorized. Furthermore, process scenarios are perceived differently based on various factors (i.e., familiarity). Therefore, the considered scenarios constitute an additional risk. Finally, splitting and defining novices and intermediates, based on the hours spent on process modeling, is another threat to validity. Obviously, an individual with a high number of hours spent with process modeling can be considered as an expert, but it is questionable whether 20.5 h are sufficient to denote an intermediate. Moreover, the representativeness of the results is limited by the relatively small sample sizes. The sizes of the samples also limit the statistical power and there might be significant differences between novices and intermediates, which we could not show in this experiment, but which might become apparent in larger samples. Regarding the statistics, it has to be mentioned that multiple t-tests were performed and no correction for multiple testing was applied. We justify this by the fact that this experiment was an explorative one instead of an experiment aiming to replicate findings of a previous experiment on EPC.

4.3 Discussion and Comparison with Prior Results

Prior to this experiment, a similar experiment (i.e., same setting and operation) was conducted using process models expressed in terms of BPMN 2.0 [6]. Furthermore, special care was taken that the used process models are comparable between the two modeling languages as well as their levels of difficulty. The main differences between both experiments were the various process scenarios captured in the used process models. However, prior experiments showed that a familiar or unfamiliar scenario does not have an influence on the comprehension of process models [23]. In general, two common effects could be observed in both experiments with respect to process model comprehension. First, the performance of subjects decreases with rising level of difficulty and, second, the performance of novices and intermediates approximates with each other as well with rising level of difficulty. It is noteworthy that the overall performance of subjects confronted with EPC process models indicates better results compared to BPMN models. Descriptively, comprehension duration and response times for answering related questions are almost the same, but the final answering scores are substantially higher for EPCs, despite expertise in process modeling and level of difficulty. It appears that subjects cope better with EPC process models than BPMN models. However, in the end, the stated observations need to be investigated by inferential statistics and by further research either through replication or similar experiments. Finally, based on a set of stated categories that can foster experiments on process model comprehension, and with the use of the conceptual framework, further experiments will be subject of future work [23].

5 Related Work

With a focus on process model comprehension, [24] evaluates different process modeling languages, whereas [25] presents factors that influence the comprehension of process models. The influence of process model complexity on related model comprehensibility is investigated in [26]. In [27], factors that affect the comprehension of process models are discussed. A state-of-the-art report on empirical research on process model comprehension can be found in [28].

Regarding cognitive aspects in process modeling, [29] discusses how the comprehension of conceptual models is influenced by a reduced cognitive load. [30] shows the difficulty of comprehending different relations between the elements in a process model. [31], in turn, discusses principles for the design of a cognitively effective visual modeling language, whereas [32] attempts to operationalize perceptual properties of modeling languages to improve their cognitive effectiveness. Moreover, [9] discusses individual preferences for process representations based on their cognitive style.

In line with eye tracking, [33] concludes that a higher mental effort can be measured by the change of pupil dilation for task-based process models. Furthermore, [34] presents results on how eye tracking leads to a more fine-grained understanding of process models. [35] studies the factors that influence process model comprehension using eye tracking.

Comparing process models expressed in terms of BPMN and EPC respectively, [36] describes an experiment in which subjects ranked the modeling languages according to their subjective comprehension difficulty. The results conclude that BPMN models are easier to comprehend. Finally, [37] investigates the differences between BPMN and EPC process models, yielding that no final message can be made regarding a positive perception.

6 Summary and Outlook

This paper investigates whether or not expertise in the domain of process modeling has a positive impact on process model comprehension. Particularly, an eye tracking experiment, using process models in terms of EPCs and related results were presented. The preliminary results indicate that expertise in process modeling not necessarily implies a better comprehension of process models. In detail, intermediates as well as novices are struggling similarly once process models exceed a certain level of difficulty. The obtained results are in line with results from a priorly conducted experiment using BPMN process models. However, it appears that EPC process models are easier to read and comprehend. Further, with broader and more detailed investigations, we attempting to confirm respective generalization. The presented experiment is part of a series of experiments, in which a conceptual framework is used that incorporates methods and theories from cognitive neuroscience and psychology [6]. Thereby, eye tracking constitutes only one measurement method from the pool of existing methods in this context. By using the conceptual framework, additional concepts from neuroscience and psychology (e.g., electrodermal activity, Cognitive Load Theory) will be used in future experiments with a particular focus on how to foster the reading and comprehension of process models.