Keywords

1 Introduction

In the Business Process Management (BPM) life cycle, the success of business process (re)design, analysis, and simulation are all underpinned by the assumption that the business activities are well understood. This understanding is extracted from graphical process models, which mainly focus on the temporal or logical relationships between business activities, as well as business rules, which are constraints and mandates that control the behavior of the process and business activities. Lack of good understanding of a business process and business rules that constrain the process can give rise to many risks. Users may inadvertently breach required standards of operation or make ill-informed decisions. Different stakeholders, such as process designers, information systems developers, and process participants may have inconsistent, or even conflicting, understanding of the same process. Ultimately, such inconsistencies hinder the effectiveness of important organizational activities and introduce risks of noncompliant process execution.

While all graphical process models generally integrate some aspects of rules (e.g. through control flow of the process), business rules can be represented in an integrated manner or in a separated manner. When represented in an integrated manner, they are shown graphically in a process model, either as textual annotations [1], as graphical links to external rules [2], or diagrammatically using the native notation of the graphical model [3], e.g. through a combination of sequence flows, activities and gateways. When modeled in a separated manner, rules are captured in separate documents or rule engines, and the relationships between the business process models and the rules are not explicitly represented in the process models. Traditionally, due to limited support for representation of business rules in graphical process modeling techniques [4], organizations often store such representations in separate text documents, spreadsheets, or disconnected business rule repositories [5]. Over the past two decades, prior work has argued for the need to model business rules in an integrated manner with business processes [6, 7], and a variety of integration methods [1,2,3, 6, 8,9,10,11] and initial guidelines on rule integration [5] have been developed.

Arguments for such integration are typically based on an assumption of process improvement and shared understanding [5]. However, despite such arguments, and despite the different integration methods developed, if and to what extent such integration improves user understanding of the process models has not been investigated. In particular, while researchers have argued that integrated modeling can improve the understanding of business processes [5], this proposition has not been empirically evaluated. In this paper, we first present the theoretical foundation of the effects of rule integration on the cognitive activities of process model comprehension. With a focus on linked rules, a type of rule integration with process models, we then hypothesize the relationships between linked rules and process model understanding and report the results of our experiment to determine if linked rules can improve the understanding of process models.

2 Background and Related Work

A business process is a structured collection of activities that accomplishes a specific goal [12]. Such structures also involve business rules, which specify obligations, permissions, and restrictions that will limit the choice of approaches toward achieving a given goal [13].

Business process modeling and business rule modeling both focus on creating a representation of the organization’s current and future practices. They are complementary approaches as they address distinct aspects of organizational practices. The overlap between business process models and business rules indicates a need to model the two related aspects together. Researchers argue that the integration of business rules into business process models can achieve better process model understanding [14,15,16], and improved governance, risk management and control [1, 17]. At the same time, however, researchers have identified a general lack of capability among process modeling languages to adequately represent business rules [4, 18, 19].

To solve this problem, a variety of integration methods and techniques have been developed since the publication of the first paper suggesting that business process and rule modeling approaches should be merged [20]. To name a few, McBrien et al. defined the structure of rules to couple business process models and rules [21]. Knolmayer et al. refined process modeling and linked the resulting models to workflow execution through layers of so-called Reaction Business Rules [22]. Kovacic et al. developed a meta-model to demonstrate how rules can link process, activity, events, data objects, and software components [13]. To summarize, three forms of integration of business process models and rules have been developed in literature viz. link integration, text integration, and diagrammatic integration. These approaches are summarized below and illustrated in Fig. 1.

Fig. 1.
figure 1

Integration methods illustration

Full size image

Link integration.

Link integration approaches incorporate information about the location of a related, externally documented, rule in a process model. Links can be static or automatic. In static link integration, the location information can be the section number and id, or the page number of the rule in a rulebook, thus allowing process users to locate the rule. Automatic link integration means the location information can be implemented as links, which will automatically navigate to the rule in the rule repository when the link is clicked. Notable contributions on link Integration are [2, 9].

Text integration.

Text integration approaches represent the content of a rule textually in a business process model. For example, BPMN has a text annotation construct which allows users to put business rules into such an annotation construct in sentential format. Notable contributions on text integration are [1, 11].

Diagrammatic integration.

While rules in link integration and text integration are represented in a sentential format, diagrammatic integration approaches represent rules in a diagrammatic format in a process model, using process modeling constructs such as sequence flows and gateways. A notable contribution on diagrammatic integration is [3].

According to [23], the fundamental purpose of conceptual models is to improve users’ understanding of the static and dynamic phenomena in a domain, and then to help developers and users to communicate and to serve as a basis for design. Process models are a typical type of conceptual model, and the factors affecting the understanding of process models have been well studied. Factors affecting the understanding of process models can be classified into two categories: process model factors and individual factors. Process model factors relate to the metrics of the process models, such as modularization [24], block structuredness [24], and complexity [25]. Individual factors, or personal factors, relate to process model users, such as an individual’s domain knowledge [26], modeling knowledge [27], modeling experience [24], and education level [24]. Figl et al. [28] provided a comprehensive overview of the literature on process model comprehension.

The argument that rule integration can improve process model understanding is the foundation that has motivated the development of different integration methods and techniques. The evaluation of the argument is critical to progress this research field. However, despite a considerable number of integration methods have been introduced using existing process modeling constructs, and despite many factors that can effect process model understanding have been identified, the question of whether integrating business rules into process models can improve the understanding of process models has not been theoretically analyzed nor empirically evaluated.

3 Theoretical Background

The limitations of diagrammatic integration are widely known due to the expressibility limitations of process modeling languages [1]. Similarly the drawbacks of rule integration through text annotations are duplicate and potentially inconsistent rule representations [29]. Hence in this paper we focus on a specific form of rule integration, namely link integration – an approach that points the model to the relevant rule, rather than duplicating that rule in the process model in either text or graphical form.

Link integration approaches incorporate visual links that connect the relevant rules to a section of the model – i.e. the links are explicitly represented on the activities or gateways that the rules constrain. This approach thus makes the connections of rules and corresponding activities explicit, presumably reducing cognitive load required to mentally connect rules to the appropriate part of the process model [16]. When rules are modeled in a separated manner, on the other hand, they have to be semantically interpreted and manually matched by the model user to the relevant parts of the model. This is an error-prone process that requires the user to interpret the business rule against the background of the entire model to determine best fit. Accordingly, our first aim is to investigate the effect of link integration on process understanding accuracy, which means how well a process model is understood:

Hypothesis 1:

Process models with linked rules are associated with better understanding accuracy compared with separated rules.

When rules are separated, all rules are organized as one set of rules, represented in some textual form (either plain text or in one of the business rule modeling languages). Finding the relevant rules that constrain a specific activity or gateway requires a comprehensive search and semantic interpretation of the set (e.g. linearly down the entire list of rules), which takes more time to mentally connect rules and a process model.

Accordingly, our second aim is to investigate the effect of rule linking on process understanding efficiency, focusing on how much time it takes a participant to review the process model and related rules to demonstrate understanding accuracy.

Hypothesis 2:

Process models with linked rules are associated with better understanding time efficiency compared with separated rules.

As extra cognitive activities such as search and semantic interpretation are needed with rule linking, our third aim is to investigate the mental effort:

Hypothesis 3:

Process models with linked rules are associated with less mental effort needed for understanding.

Despite the benefits, link integration is not without limitations. First, people using linked rules may focus on the interactions of specific rules and process components, without a holistic understanding of the process model and rules as a prerequisite, thus may have inaccurate understanding. Second, it can cause the attention switching effect [30], which means that users need to split their attention among multiple sources of information and mentally integrate them. Given separated rules as a whole list, one can choose to learn and assimilate more rules before switching attention to a process model, thus to reduce attention switches and time needed. It is therefore not clear to which degree the additional cognitive cost in terms of attention switching counter-balances the improvement in understanding. Thus, a study is needed to investigate this effect of business process and rule integration. To this end, we propose an experimental approach to test our hypotheses.

4 Research Method

This study applies an experiment research method to explore differences between linked and separated business process models and rules. In this section, we introduce our experimental design and describe our instruments, experiment settings and participants.

4.1 Experiment Design

The experiment is a single factor experiment. In our experiment, the use of linked rules is the considered factor, with factor levels “present” and “absent”. We used two groups, two factor levels, and two domains in our experiment. Each group was tested with two domains separately, and for each domain, the two groups had different factor levels.

We have three main considerations in our between-subject design. First, our experiment environment only allows us to have one participant to do the experiment at a time. Second, the understanding performance depends on an individual’s cognitive competence and experience. Thus, group imbalance is a challenge for between-subject design. Third, we want to increase the generalization ability of the experiment in terms of domains, while controlling the learning effect.

Under these considerations, we designed our experiment as a balanced single factor experiment with repeated measurement, based on an experiment design from [24] which can increase the power of the experiment given the same number of participants [31]. The overall design is illustrated in Fig. 2. In this design, each participant will be tested for all factor levels and all domains, thus (1) more data will be collected than in a single run experiment, (2) two domains are tested to increase the generalizability of the results. The order of factor levels is reversed between groups, so the factor of order of treatment and learning effect are counterbalanced across groups. Please note that the forms of rule representation are inversed in the two runs. In the first run, Group 1 are given linked rules and Group 2 are given separated rules, while, in the second run, Group 1 are given separated rules and Group 2 are given linked rules.

Fig. 2.
figure 2

Overall experiment approach

Full size image

As illustrated in Fig. 3, when linked rules are present, link buttons (labeled with “R”) will be shown on activities and gateways in a process model, when a link button is clicked, the rules that are connected to the activity/gateway via the link button will be displayed on the “Relevant Rules” area on the right of the screen. When linked rules are absent, no link buttons will be shown in a process model, and all rules will be displayed in the “Relevant Rules” area on the right side of the screen.

Fig. 3.
figure 3

Independent variable illustration

Full size image

4.2 Measurements

To measure the accuracy of understanding we use the percentage of correct answers to comprehension questions. We use the time from the point that a process model is displayed on the screen, to the point that the last question for this process model is answered as the measurement of time efficiency. To measure mental effort we use both an objective measure and a perception measure. We used the eye-fixation duration for each model as the objective measure. Eye-fixation is the maintaining of the visual gaze on a single location. Vision is suppressed during the eye saccade, and new information is acquired only during the fixation. Eye-fixation duration was proved to surpass pupil size as a mental effort measure [32]. As measure of perception of required mental effort, we asked each participant to select the model he or she perceived more difficult.

4.3 Instruments

We briefly describe each part of the experimental instruments below.

Questionnaire.

We have a pre-experiment questionnaire and a post-experiment questionnaire. Questions for which the answers could be affected by participating in the experiment, such as the extent of familiarity with business process models and rules, and the extent of familiarity with the knowledge domains used in the experiment, were included in the pre-experiment questionnaire. To save a participant’s mental effort before the experiment, objective questions which could not be affected by the participation in the experiment, such as a participant’s major and which year he or she is in, were put into a post-experiment questionnaire, together with a question asking participants which model consumed most of their mental effort in the experiment.

Tutorial and examples.

The tutorial covered all BPMN elements and business rule concepts that participants would need to know to perform the tasks, e.g. activity, sequence, activity group, parallel gateway, exclusive gateway, and business rules. Example process models, rules, as well as questions and answers were provided after the tutorial. The instructions direct participants to study the process models, click the rule links, read the rules, and answer the questions. The order of treatments in the tutorial and examples are consistent with the order in the experiment.

Treatment design.

To limit the learning effect, only two process models were used, and only three questions were asked for each model. The information needed from a process model and rules to answer a question are independent from each other thus the information learned from a previous question has little contribution to the current question. We designed process model A based on previous experiments [33, 34], and designed process model B to keep the complexity of the two models as close as possible. The rules and questions of the two process models are designed with the same cognitive load level in mind. The rules covered common rule violations such as time constraints, route selections, and data logic. To assimilate what happens in practice, several rules can control a single activity, and a violation of any of the rules will lead to a breach. We kept a variety of metrics of the two sets of models, rules, and questions the same or as close as possible. A package of the entire experiment is available for download on DropboxFootnote 1.

4.4 Settings

The pre-experiment and post-experiment questionnaires were implemented in QualtricsFootnote 2. The tutorial and experiment were implemented as an Eclipse RCP applicationFootnote 3. The texts and diagrams were proved to be clearly visible from a distance of over 60 cm in the pilot test. As shown in Fig. 4, the screen was divided into three Areas, viz. Process Model Area, Relevant Rules Area and Questions Area. The complete process model and all the rules are displayed without the need of scrolling. No zooming is allowed in the application. All text and diagrams are in black and white so color blindness will not introduce bias to the experiment. We used Tobii Pro TX300, an eye tracker with a 23-inch screen of a resolution of 1920 × 1080 that captures gaze data at 300 HzFootnote 4. The experiment was set in a lab. The lab has no window and the rooftop lights are the only light source. The materials, eye-tracker, and lights had the same settings for all participants.

Fig. 4.
figure 4

Instrument Illustration

Full size image

4.5 Participants

Students at an Australian university participated in this experiment voluntarily. Eight PhD students participated in the pilot tests. Fifty coursework students of an information systems course participated in the main experiment and were randomly assigned to two groups. Our sample size is considerable compared with other comparable experiments, which have sample sizes between 20–30 [32, 35]. All participants were required to have basic knowledge of flowcharts, UML or ER diagrams. We only used the most basic BPMN symbols and easily understandable daily English in the material, which did not require substantial experience from our participants. We did not put a time limit for each student, and all fifty students finished the experiment within an hour. Forty-eight students finished the experiment successfully, and the eye movements of two students in Group 1 failed to be properly recorded by the eye-tracker. Thus, we discarded the two samples in the analysis of eye-movement related data. As an incentive, each student was offered a $30 voucher for participation.

5 Results

For comparing categorical dependent variables between two groups such as answer correctness and the choice of mental effort, we use Chi-squared test, which can be used to compare categorical data [36]. For other numerical dependent variables, we first checked if a dependent variable is normally distributed using Shapiro-Wilk test at a significance level of 0.05 [36]. If data of both groups were normally distributed, we checked whether the data met the assumption of equal variance using dependent Levene’s testFootnote 5 at the significance level of 0.05, and then used the independent-sample t test. If data in any group were not normally distributed, we used the Mann-Whitney U testFootnote 6 across groups. We describe the results for each hypothesis in turn.

For Hypothesis 1, we ran Chi-square tests between the two groups, with the correctness of answers as the dependent variable, for the two models separately. Table 1 shows the Chi-square test results, which show that understanding accuracy was significantly correlated with the form of rule presentation in Model 2 (p = 0.03), but not in Model 1 (p = 0.16), which partially supports Hypothesis 1.

Table 1. Test of Hypotheses 1 – understanding accuracy
Full size table

Conclusion 1:

Linked rules are partially associated with an improved understanding accuracy.

For Hypothesis 2, the time spent of Group 2 in Model 2 was not normally distributed. We ran independent-sample Mann-Whitney tests between Group 1 and Group 2, with the time (from beginning to the end of answering the last question in each run) as the dependent variable. The test result of Hypothesis 2 is shown in Table 2. Table 2 shows that time used in each model is related to the form of rule presentation, supporting Hypothesis 2 at a significance level of 0.05.

Table 2. Test of Hypothesis 2: understanding efficiency
Full size table

Conclusion 2:

Linked rules are associated with increases in understanding efficiency.

For Hypothesis 3, the eye-fixation durations in the two runs were not normally distributed. We therefore ran independent-sample Mann-Whitney tests for the two runs separately. The objective test of Hypothesis 3 is shown in Table 3. From Table 3 we can see that the mental effort is associated with the type of rule presentation, supporting Hypothesis 3 at a significance level of 0.05.

Table 3. Test of Hypothesis 3: objective mental effort
Full size table

The results of the perception of mental effort are shown in Table 4. In Group 1, 0 participants selected Model 1 (linked rules), while 23 participants selected Model 2 (separated rules) as the model requiring more mental effort. Two participants selected ‘equal’ as the answer. In Group 2, 11 participants selected Model 1 (separated rules), while 6 participants selected Model 2 (linked rules) as the model requiring more mental effort. Eight participants selected ‘equal’ as the answer. From Table 4 we can intuitively see that participants indicate that models with separated rules require more mental effort, regardless of model content (model 1 or model 2).

Table 4. Perception of mental effort
Full size table

To statistically compare linked and separated rules, we coded the perception answers as follows: When a model with linked rules was selected as the model that required more mental effort, linked rules were assigned 2 points. When the model with separated rules was selected as the model that required more mental effort, separated rules were assigned 2 points. When a participant selected the two models as equal, both linked rules and separated rules were assigned 1 point. We used a t test for the difference in average mental effort perception between linked and separated rules. Table 5 shows that mental effort in linked rules is significantly smaller than in separated rules.

Table 5. Coded mental effort
Full size table

Conclusion 3:

Linked rules are associated with reduced mental effort required for model understanding.

6 Discussion

Our results support Hypotheses 2 and 3, indicating that linked rules are associated with increases in understanding efficiency and reduced mental effort required for model understanding. While Hypothesis 1 has only partial support. For the results of Hypothesis 1, the p value for Model 1 was greater than 0.05, indicating a lack of statistical significance. To explore this result further, first we compared the two models, and the metrics comparison showed that the two sets of models, rules and questions are the same or close in all the metrics. Second, we investigated answer correctness and time spent of each model. The statistics showed that the two sets of models, rules, and questions had no significant difference (with p = 0.647 and p = 0.822 respectively). Thus, we concluded that there was no bias between Model 1 and Model 2. Finally, we broke down the correctness of answers to each question to explore the lack of statistical significance of the differences between linked and separated rules in Model 1. As shown in Fig. 5, the result of question 1 shows that the group with linked rules had lower understanding accuracy than the group with separated rules, which is against Hypothesis 1, while the correctness of all other 5 questions indicates the support of Hypothesis 1. We assume that one possible reason is that the participants had not learnt how to use linked rules well when they met the first question. Recall that we had to balance time with fatigue and tracking data accuracy and thus had a time constrain in the experiment, so we used a simple illustration of linked rules (See Fig. 3) in the training material, compared with the models and rules in the formal experiment what were much more complex and challenging. Thus, participants may not quickly find how to utilize rule links.

Fig. 5.
figure 5

Answer correctness breakdown to each question

Full size image

Our study is not without limitations. In terms of internal validity, the different layout of screen areas could possibly affect the results. It is possible that the experiment results will be different if we change the location of each area. In terms of construct validity, we operationalized each construct in our study in limited ways. The questions were designed to test the understanding of the effect of business rules on business process models. Following [37], it would have been ideal if we had measured the perceived quality and efficiency of understanding, and asked questions only about a process model itself. Thus, our research results are limited to the treatments, measurements, and questions that we used. Finally, in terms of external validity, we cannot say that the process models, rules, and questions we used faithfully reflect those used in organizations in practice. Organizations may use more complex process models and lager number of rules and the tasks may be more challenging. The use of students as participants could also weaken the generalization ability of the results.

7 Conclusions and Outlook

In this paper, we have studied the relationship between rule integration and business process model understanding. Rules can be integrated into process models in a variety of ways, and in this paper, we report on our findings based on a specific form of rule integration, namely linked rules. We focused on 3 aspects of understanding: understanding accuracy, time efficiency, and mental effort. Our study results presented three conclusions: (1) The association between linked rules and understanding accuracy is partially supported. (2) Linked rules are significantly associated with improved time efficiency. (3) Linked rules are significantly associated with reduced mental effort. Our conclusions are drawn from an experiment design that utilized an eye-tracker. The design of the experiment provides a methodological contribution towards the study of process model understanding. Opportunities exist for future research to perform similar experiments on different rule integration methods such as annotation and diagrammatical integration [38] and investigate the effects on process model understanding.

Business rules have a broad scope, and business rules can be quite varied in many aspects such as change frequency, complexity and governance responsibility [38]. Thus, the best way for each rule to be integrated into a process model can be different. The characteristics of business rules or different rule categories can influence which integration method has the best performance in terms of process model understanding. Quite a few business rule classification frameworks such as [39, 40] exist in literature. Finding the connection between type of rule and the best corresponding integration approach to improve process model understanding will be a valuable topic for future research.