CMMN evaluation: the modelers’ perceptions of the main notation elements

Abstract

Case Management Model and Notation (CMMN) has been introduced as a graphical modeling language targeting the modeling of human-centric processes. Despite its growing reputation since 2016, when the OMG standant was released, the usage and the adoption potential of CMMN is not yet evaluated. The goal of this paper is to evaluate CMMN language and the contribution of its main notation elements to its future adoption, based on the experience of modelers. A CMMN workshop was conducted, where groups of modelers modeled two different human-centric, real-world processes with CMMN. The effectiveness and efficiency of the language and modelers’ usage experience were evaluated. Their perception of the role of the CMMN notation elements to their future adoption CMMN have been recorded through a survey. A multi-criteria decision making method (Analytic Hierarchy Process–AHP) was utilized for analyzing the answers and generating the results. The evaluation results showed that CMMN language could be adopted for modeling non-structural processes and the study participants showed a positive attitude towards adopting CMMN driven by the fact that they overall perceived it as useful. To the best of our knowledge, this is the first attempt to evaluate CMMN language’s usability and prospects of adoption. Moreover, this is the first empirical study that explores the syntax of a process modeling language and its effect on its usage and adoption. Overall, since interest in CMMN is increasing, this work could inspire future researchers and practitioners to further explore the CMMN usage and adoption potential.

1 Introduction

The Case Management Model and Notation (CMMN) modeling language was created by the Object Management Group (OMG) and published in 2014, while its latest version was released in 2016. CMMN is used for capturing work methods that are based on the handling of cases requiring various activities that may be performed in an unpredictable order in response to evolving situations. It is an alternative language to the Business Process Model and Notation (BPMN), also created by OMG, which focuses on control flow to describe business processes. Using an event-centered approach and the concept of a case file, CMMN expands the boundaries of what can be modeled with BPMN, including less structured work efforts and those driven by knowledge workers. Using a combination of BPMN and CMMN allows users to cover a much broader spectrum of work methods. [42]

Once a conceptual modeling language has been proposed, it should be evaluated [27]. Since 2016, when the corresponding standard was released by OMG, CMMN is gathering increasing research interest as indicated by the number of papers published is the last few years. These research effort focus on its application (see for example [4, 24, 25, 50, 65, 69] and analysis (see for example [2, 5, 6, 11, 33, 56]. Furthermore, well-known process management tools, such as Flowable, Trisotech or Signavio invest on integration CMMN in their solution portfolio. However, the evaluation of CMMN language, especially by its users is still an unexplored area. Such an effort may contribute to improve the language usability and provide guidelines for the execution of CMMN models, leading to wider usage of this new standard. Only recently, [7], attempted to explore semiotic clarity of CMMN language’s semantics, however it does not explore any evaluation parameters within the scope of CMMN usage. According to Kleppe et al. [29], a modeling language comprises of three (3) parts: abstract syntax (meta-model), concrete syntax (graphical notation) and semantics, which according to the authors “should not be discarded from the language’s description”.

This research emphasizes on the graphical notation of the language and assesses CMMN language new elements, always taking into account both the language’s meta-model, as well as the rules it sets, and the language’s semantics. The presented evaluation attempt of CMMN is twofold, taking into account mainly two parameters important for graphical notation languages usability, model effectiveness and efficiency and user-satisfaction, as proposed in [55]. First, CMMN models are assessed applying established modeling metrics on complexity and expressiveness to evaluate the usability of the new notation elements and, then, the modelers opinion on its usability, as well as their intention to adopt it, is assessed. Specifically, the modelers participate to a survey that records their usage experience of CMMN language and the contribution of each new CMMN notation element to its future adoption.

During this empirical study, a CMMN workshop was conducted, where participants, divided in groups, modeled two real-world cases with CMMN. The generated models were evaluated utilizing for the first time metrics on complexity and expressiveness for CMMN models introduced by other researchers [9, 34]. Since the participating modelers comprehended the CMMN philosophy and performed adequate modeling, they participated to a survey where they assessed their usage experience of CMMN language, as well as the role of each CMMN notation element to their future adoption and usage of CMMN. A formal, multi-criteria decision making method (Analytic Hierarchy Process–AHP) was utilized to analyze the collected answers and identify the users’ perceptions of the CMMN notation elements.

In brief, the evaluation results showed that CMMN language could be adopted for modeling non-structural processes and the study participants produced valid CMMN models, using the new elements, showed a positive attitude towards adopting CMMN driven by the fact that they overall perceived it as useful. Since the interest in CMMN is increasing, this work could inspire future researchers and practitioners to further explore the CMMN usage and adoption potential in a more systematic fashion. Overall, to the best our knowledge, this is a first attempt to evaluate CMMN language’s usability and adoption based on the modelers’ experience.

The rest of the paper is structured as follows. In Sect. 2, an introduction to the concept of Case Management, as well as the CMMN language takes place. Related works on other modeling languages’ evaluation, together with related works on CMMN language are presented in Sect. 3. In Sect. 4, the empirical evaluation study that takes place in this work is being presented, alongside with details about the steps followed. The first evaluation step of study, the models evaluation, is presented in Sect. 5, as an intermediate validation step for the models, created during the study. In Sect. 6, the user experience evaluation takes place, in order to evaluate the adoption prospect of CMMN. In Sect. 7, the outcome of the evaluation study for CMMN is being discussed. Finally, in Sect. 8, conclusions are drawn, the limitations for this research work are pointed out; future work is also set by the authors in this section.

2 Case management and CMMN overview

2.1 Case management

A Case is not a common business process. It requires knowledge work, namely thinking, skills, expertise, and experience as far as the details of the situation are concerned in order to make all the essential process design appropriately [40]. Case management is a collaborative process that assesses, plans, implements, coordinates, monitors, and evaluates the options and services required to meet the client’s health and human service needs. It is characterized by advocacy, communication, and resource management and promotes quality and cost-effective interventions and outcomes. This type of management refers not only to the coordination of work in one organization that is not routine and unpredictable, and requires human judgment in order to be executed, but it also refers to gathering all of the relevant information in one place, which is called case folder, and acting upon this source of information to fulfill any organizational requests [38].

Applications of Case management include licensing and permitting in government, application and claim processing in insurance, patient care and medical diagnosis in healthcare, mortgage processing in banking, problem resolution in call centers, sales and operations planning, invoice discrepancy handling, maintenance and repair of machines and equipment, and engineering of made-to-order products [42]. Case management as a practice is not something new; references to the term “case management” go back to the 1980s or earlier and the Case Management Society of America was founded back in 1990. Various case management approaches have been proposed to support this flexibility for unstructured processes. As aforementioned, recently OMG published the CMMN as a specification for case management.

Table 1 CMMN elements, notation and description

2.2 CMMN language

The main objective of CMMN language specification [42] is to define a common meta-model and notation for modeling and graphically expressing a Case. It also provides for interoperability guidelines between tools and minimum execution semantics, as stated in [35]. A Case is described as “a proceeding that involves a set of actions that need to be taken regarding a subject in a particular situation in order to achieve the desired outcome”.

Traditional examples are Cases that refer to legal and medical working environments, where a legal Case involves the application of the law to a subject in a certain fact situation, and a medical Case involves the care of a patient in the context of a medical history and current medical problems. The subject of a Case may be a person, a legal action, a business transaction, or some other focal point around which actions are taken to achieve an objective. The situation commonly includes data that inform and drive the actions taken in a Case. There are two phases for each case. Design-time phase, during which, business analysts prepare the case execution by modeling the case. Once a case has started to being executed, the case is in the run-time phase. In this phase, the case workers are working on achieving the case objectives [33].

Fig. 1

CMMN model example [42]

CMMN language provides a plethora of notation for the Case Management elements in order to cover all the aspects of human-centric modeling elements required to represent a Case model. A CMMN model (Fig. 1) primarily comprises the items presented in Table 1. Figure 1 presents the CMMN model of an identical case example, the one of claim management. In CMMN, a Case Plan Item contains all the case elements that are involved. Data objects are projected as Case File items, representing a piece of information of any nature. Tasks are handled as an atomic unit of work and are divided into different types, namely, human tasks (e.g., Change Responsibilities in Fig. 1), process tasks (e.g., Identify Responsibilities in Fig. 1), or other case tasks (e.g., Create Claim in Fig. 1).

Except for these three notation elements, CMMN introduces some new ones, providing modelers with specific new features, making the language more appealing to use. Those are Stage Plan Items, Milestone Plan Items, Event Listeners, Sentries and Discretionary Items. A Stage (e.g., Process Claims) is a logical container of tasks to be performed within the course of a case. It allows structuring a case hierarchically. A Milestone (e.g., Responsibilities Identified in Fig. 1) represents an achievable target, defined to enable evaluation of the progress of the Case. Event Listeners are described as anything that can change the case state and are divided into two different types: timer events and user events. Sentries as a combination of an event and a condition (criterion) are not used as a standalone CMMN element, but it is attached to another element. Sentry criteria are used, categorized into two different types, entry and exit criteria of a sentry (i.e. white rhombuses and black rhombuses in Fig. 1). Finally, a Discretionary Item is an item, of which instances can be planned, to the “discretion” of a case manager. Discretionary Items include Discretionary Stages and Discretionary Tasks.

Regarding CMMN models’ design, two different modeling styles, have been identified [48, 59]: User Driven style [59] referred to as abstractive perspective in [48], and ECA (Event-Condition-Action) style, referred to as analytical perspective in [48]. Either each discrete style may be followed when modeling a case, though most times both of them are combined within a single case model [59]. The main difference between them is the degree of flexibility left to the case worker during run-time, reflected in the abstraction level of the case model–or parts of it–created at design-time. More specifically, following User Driven style, “most choices of what to do when are made by case workers at run-time using their judgment about what needs to happen” [59]. Thus, this style results in using discretionary items and user event listeners to a large extent. On the other hand, the ECA Style, “represents the allowed order of plan items explicitly as much as possible using lifecycle transitions and sentries” [59]. Respectively, this style results in extensive use of sentries and stages. However it is common to have both styles identified within a CMMN model. For instance, in the model projected in Fig. 1, both styles are present. The ECA style is used to define the prerequisite actions needed for the successful creation of a claim. That is done by connecting different process and human tasks (e.g. Identify Responsibilities, Create Claims Notification etc.), which are contained within different stage elements (e.g. Attach Base Information, Identify Responsibilities etc.) and have sentries attached to them. Oppositely, the User-driven style is represented by the use of Discretionary process and human tasks (e.g. Review Documents, Create Letter etc.), alongside with user event listeners, representing that someone is being informed depending on the outcome of the case.

3 Related work

3.1 Modeling languages’ evaluation

Regarding the evaluation of a specification or a modeling language, there are plenty of research works aiming towards that direction. For instance, [19] conducts a comparative evaluation of languages focusing on their underlying grammars, while, [36] evaluates language quality through empirical studies, and [22] adopts an ontological perspective by comparing the primitive concepts of the language to those of foundational or domain ontologies. Along a different path, language utility has been evaluated via experiments, often using student subjects, focusing on language comprehension (e.g., [61]) and/or the users’ ability to carry out meaningful tasks (e.g., [26]). Other evaluations have focused on the effectiveness of a language’s graphical syntax by comparison to standard principles (e.g., [37]), or through applications of the language to realistic examples or case studies (e.g, [17]).

For other OMG modeling languages, like BPMN and Unified Modeling Language (UML) [41], several research attempts can be found in literature. Regarding the latter, in [54] and [62], the applicability of UML as a modeling language is evaluated in practice, while in [58] UML graphical syntax is being analysed and evaluated based on BWW Ontology [64]. For BPMN, the usability of business process models is being evaluated in [46] according to standard ISO 9126, while the appropriateness of BPMN for modeling service choreographies is being evaluated in [10], using an extended quality framework.

On the subject of quality, UML quality is evaluated in [32] through a generic quality framework, while in [57] UML conceptual modeling quality is being measured. There, quality is defined as a three-dimension entity, comprising of three main parameters, i.e. usage, specification and implementation. On the other hand, BPMN quality is being evaluated in [63] through the use of a semiotic quality framework, BPMN models quality is being evaluated in [45] using a set of proposed measures.

Regarding models evaluation, there are many research works that define metrics for OMG specifications, including [20, 21, 23, 30, 66]. For user experience and adoption evaluation, there are works in literature which explore the adoption of a modeling language, like [44], that uses Technology Acceptance Model, in order to evaluate the acceptance of BPMN, as well as [31, 43].

On the assessment of the understandability and comprehension of process models, Dikici et al. in [16], have made a systematic literature review, studying the factors influencing the understandability of process models. Similarly, in [18], the authors have made a systematic literature review regarding the comprehension of visual process models. Finally, in [28] , the authors aim at investigating underlying theories of research into business process model understandability by means of an in-depth analysis of 126 systematically retrieved research articles on the topic.

3.2 CMMN language

Following its release, there is an increasing number of research works, focusing either upon the analysis of the CMMN language [5, 11, 33], or upon extending and improving it [2, 6, 56]. Despite being a relatively new modeling language, the increasing relevancy of CMMN is confirmed by the large number of platforms aiming to support CMMN modeling. All popular Business Process Management platforms have added CMMN in their portfolio and support in the integrated usage of BMMN and CMMN to model business activities more efficient. Camunda BPM Platform, which supports design of CMMN models through a modeling tool named as Camunda Modeler consists of such an example. Furthermore, there are modeling environments, such as Trisotech, with its modeling suite tool, Trisotech Case Modeler, and Flowable, with its Case Management Tool, are examples of modeling software providers that invested on building CMMN execution engines.

As far as the exploration of CMMN language usage is concerned, several applications of Case Management, can be identified, that use CMMN, combined with BPMN, in real-world scenarios [24, 65]. In a similar fashion, CMMN has been utilized by the authors in [50] for the needs of a real-world project, applied for the design and the implementation of a collaborative process by the project team. Moreover, there are also works like [69] and [25] that compare the usage of CMMN compared to BPMN based on a specific case study.

Finally, regarding CMMN models, recent works attempt to identify metrics for models evaluation, including [34] where complexity metrics are introduced, and [9], where workflow patterns first introduced in [15] are being identified in CMMN models, in order to measure models’ expressiveness.

In conclusion, some research works attempt to extend or improve CMMN, while others are exploring the application of CMMN to a specific domain or real-world scenario. Other works, are identifying differences between CMMN and other modeling languages.

As the interest in CMMN is increasing and modeling platforms began to invest on CMMN engines, the evaluation of CMMN language by users becomes a necessity [1]. To fill that research gap, this study focuses on two aspects of CMMN language that could be evaluated: its usability and future adoption from the modelers. The evaluation of CMMN is based on its usability based on the models produced by modelers in terms of complexity and expressiveness, as well as its potential to be adopted by modelers based on this experience. To the best of our knowledge, this is the first attempt to assess both the usability and the adoption prospect of CMMN in terms of its syntax, specifically the new notation elements that CMMN introduces. Both the produced models and the modeling experience of modelers are evaluated. To evaluate the usage of the new notation elements in CMMN models, metrics introduced by other researchers for CMMN models [9, 34] are applied in practice for the first time. To evaluate user preferences on the new notation elements existing well-known approaches are utilized [52]. The proposed evaluation approach is presented in the following section.

4 CMMN evaluation study

4.1 Evaluation study design

To evaluate the CMMN language, we relaid on modelers experience and their perception of the new notation elements introduced in CMMN. These elements are either new as for example sentries or existing ones redefined in CMMN context, as milestones. As CMMN is not widely used, it is not common to encounter experienced modelers or process engineers in it. Thus, we conducted a workshop introducing CMMN language to process engineers with modeling experience mainly with BPMN. There, participants designed non-structural processes with CMMN and the language evaluation was monitored based on the resulting process models and modeler’s experience. Such workshops aim to make modelers familiar with the modeling language, especially a new one, ensuring that they will understand its philosophy and modeling principles [3]. Essentially, we have adopted an one-shot case study design [60, 67, 68] to acquire empirical knowledge on the modelers’ experience with CMMN in terms of usability. Although there are limitations of the validity of the results of such an approach, this empirical study provided the opportunity to obtain an indication of the new CMMN elements potential.

Fig. 2

Evaluation study design

Figure 2 depicts the steps we followed to conduct our study. First, we conducted the workshop and collected the CMMN models that the case study participants created. Then, we evaluated them in terms of model efficiency and effectiveness using proposed metrics for CMMN models. Secondly, we acquired the case study participants’ experience during applying the CMMN language via a survey and we examined the modelers’ perceptions with a formal, multi-criteria decision making method (Analytic Hierarchy Process - AHP). The evaluation protocol applied, consisting of the aforementioned steps of the empirical evaluation study, the method and tools utilized to conduct each of them, in addition to the corresponding data, generated during each step, are publicly available¹.

CMMN workshop. The workshop started with an introduction to the language to ensure that the participants have understood the modeling language’s philosophy and basic principles. To that end, we detailed the syntax and the notation of the language, as well as their use; and any unique characteristics of the language were highlighted. Then, we (the workshop instructors) modeled a test case with the collaboration of the participants, in order to help them to get familiar with the modeling language’s use. During the example case modeling, we discussed any questions about the language’s use, alongside with any misunderstandings they had during modeling. Further, the participants were divided in teams of two or three people and attempted to model (free of instructors support) few identical business cases, which we presented to them. We worked on team-level for encouraging collaboration and sharing ideas on CMMN modeling. All teams used the same modeling tool and no collaboration between teams was allowed for avoiding influenced results (e.g., influenced opinions). By the end of the workshop, each team had designed a model per business case. After the workshop, the models’ evaluation phase began.

Models evaluation. The first part of the evaluation focuses on the created models, as their evaluation indicates whether there was a good use of CMMN language’s syntax and proper understanding of its principles. CMMN models evaluation is also considered as an intermediate validation step for the created models, with the view that well-designed models indicate good usage and understanding of the language, leading to trustful user experience results.

User experience evaluation. Finally, after model evaluation, user experience is also evaluated. At this stage, the case study participants, present their modeling approaches for the use cases, answering questions set by the instructors. The questions refer to the participants’ overall experience using the modeling language being evaluated, addressing factors that characterize the language’s usage (i.e., usability, easiness, etc.), as well as its adoption prospect. Regarding the latter, focus is cast on the modeling language’s syntax, namely, the elements that its notation consist of, as a factor of influence for the language’s adoption prospect. To record participants’ experience and draw conclusions about both the modeling language usage and its adoption prospect, a survey including a questionnaire is conducted. As aforementioned, AHP is used, as a formal method, for the analysis of questionnaire’s data, as it facilitates pairwise comparison of new CMMN notation elements.

4.2 Evaluation approach application

4.2.1 CMMN workshop

As aforementioned, for the purpose of the study, the authors set up a workshop about CMMN that begun with an introductory presentation, and lasted four (4) weeks in total. Participants to the workshop were 24 process modelers. They were chosen, mainly because they were familiar with business processes, and their modeling experience would be useful for CMMN evaluation. During the introductory presentation, an introduction to CMMN language took place, presenting its main characteristics, target and scope. During that, great importance was given to the five (5) new notation entities that CMMN introduces (e.g., Stages, Milestones, Sentries, Discretionary Items and Event Listeners) and what advantages the case designers gain using them. Then, an example case was modeled as a CMMN tutorial, aiming to educate participants about Case Management principles and the modeling techniques required to model a case. Any arising questions relating CMMN modeling philosophy, difficulties in modeling and any possible misunderstandings were discussed at this stage, so as to guarantee the proper usage of CMMN during modeling.

Prior to the cases modeling procedure, the participants were first divided in eight (8) modeling teams of three members each. After this step, two (2) real-world cases were presented to the participants, by people that could be considered as experts, considering the fact they have been active contributors to these cases. The evaluation study’s participants had the chance to interview those people and could contact them at any time, for questions regarding any misunderstandings during modeling. That way, it was reassured that the participants were well-aware of the two cases’ nature.

Cases modeled. The first case referred to patient treatment, a classic case example. There, different hospital departments personnel is involved in order to decide a patient hospitalization status and perform her potential medical treatment. Patient Treatment is a challenging issue, since it largely depends on human decision often taken in an ad-hoc manner, a fact that makes it highly dynamic. It fits into the domain of Healthcare, which, as it was mentioned above, is a domain where the work procedure needs human worker involvement as the work that has to be done is highly variable [49]. The second case referred to the work-flow of a product exchange platform, ReWeee Platform. [47, 48]. With this platform users and households can donate or exchange unused Electronic or Electric Equipment (EEE) in order to prevent the creation of EEE waste. This work-flow can be identified as a case, considering the fact that its success lies within the social communication between volunteers and their collaboration of all interested parties in order to achieve the best possible result. The analytic descriptions of the scenarios, identified as cases, that the participants modeled are narrated in “Appendix A”.

After cases presentation, each team should model both examples, in order to avoid the issue that some modeling teams could gain advantage over the others, due to prior knowledge of a domain as the latter “might create substantial difficulties in an experimental study” [3, 19]. The modeling teams were given a period of three weeks in order to design the cases. Any required clarifications were given by the instructors, providing the modeling teams with continuous feedback whenever required.

4.2.2 Models evaluation

For the evaluation of CMMN models, designed by the study’s participants, two (2) metrics, already existing in literature, were used. The first one was the complexity of the designed CMMN models, as this is introduced in [34], while the second one was the expressiveness of CMMN models, measured in [9] through the identification of workflow patterns. These two models evaluation metrics were also used for models validation. Models evaluation procedure is analytically presented in Sect. 5.

4.2.3 User experience evaluation

Having completed the modeling of the assigned Cases, teams presented their models on a specified day, describing their approaches, their modeling perspectives, as well as any challenges they faced. After models’ presentation, participants took part on a survey about their experience about modeling with CMMN. It included a few questions regarding CMMN language’s usability, and a questionnaire about the contribution of the new CMMN notation elements to the adoption of this new modeling language. Discussion and collaboration with each other was not permitted, as the goal was to capture the opinion of each participant individually. Participants’ experience evaluation procedure, with the use of AHP, is analytically presented in Sect. 6.

5 Models evaluation

In order to evaluate the effectiveness of CMMN usage, one should evaluate CMMN as far as its syntax use and more specifically the modeling results it produces. For this purpose, the models designed by the case study’s participants were evaluated in order to examine two factors. Firstly whether the generated models conform with the nature of the case studies these represent, and secondly, whether there was a good usage of CMMN syntax in order to do so. To this end, it was decided to explore metrics for the evaluation of CMMN models that would characterize the modeling philosophy behind them as well as their completeness. For each metric, some variability measures were assessed, namely the mean, the range and the standard deviation, in order to explore the correlations between the results.

5.1 Measuring CMMN models complexity

5.1.1 Method

As aforementioned, the first evaluation parameter for CMMN models that was measured, was the complexity of CMMN models, as it is described in [34]. There, three (3) discrete complexity metrics are being introduced: Size (CS), Length (CL), and Complexity (CC). More specifically:

• Size (CS) of a model is defined as the sum of the CMMN elements that exist in a CMMN model. These elements can be a selection of case, stage and discretionary stage, fragment, case file item, task and discretionary task, event listener, milestone and connector elements (see Table 1).

• Length (CL) is defined as the maximum nested depth of a CMMN model. What is measured is the number of nested stages or fragments. For example, a model that is comprised of a case plan item that contains a stage plan item that contains a task plan item has length equal to three.

• Complexity (CC) is defined as the sum of the weights, that each CMMN element has. Weights for CMMN elements are defined in [34], and are projected in Table 2.

Table 2 CMMN elements weights as appear in [34]

Fig. 3

Model example for product exchange platform case

Example

To make clearer how these metrics were calculated during the models evaluation procedure, the following example is presented. Figure 3 presents a model for the case of product exchange platform. For that model, Size (CS), Length (CL) and Complexity (CC) are calculated as following:

CS: number of Case plan items + number of stages + tasks (of any type) + number of discretionary items + number of case file items + number of event listeners + number of milestones + number of connectors = 33.

CL: maximum nested depth of model. Here we have a case plan item that contains stage plan items, containing task plan items. Thus CL = 3.

CC: (number of case plan items * corresponding weight) + (number of stage plan items * corresponding weight) + (number of discretionary stage plan items * corresponding weight) + (number of plan fragments * corresponding weight) + (number of case file items * corresponding weight) + (number of task plan items * corresponding weight) + (number of discretionary task plan items * corresponding weight) + (number of event listeners * corresponding weight) + (number of milestones * corresponding weight) + (number of entry criteria * corresponding weight) + (number of exit criteria * corresponding weight) = 31.

5.1.2 Results

Table 3 Patient treatment complexity results

Table 4 Product exchange complexity results

As far as the complexity metrics of the created models are concerned, Tables 3 and 4 project the results for Patient Treatment and Product Exchange cases respectively. Great deviation is observed between the models Complexity (CC) for both cases. This deviation, is also visible to the range between the CC values of the created models. Obviously, there were modeling groups that modeled the same case with different design philosophy. An identical example, are modeling teams M2 and M6, which for both cases, the one (M2) had the lowest, and the other (M6) highest size, length and complexity values. One main reason behind that, is the use of discretionary items in the CMMN model.

Another observation, is that all models, except for those of modeling team 6 (M6), for both cases had similar values of length CL. CL as was described in Sect. 5.1.1, measures the depth of a CMMN model. This arises the assumption that all modeling teams, more or less, had described their cases in several layers. More specifically, all teams utilized nested Stage elements that grouped tasks and case data.

In general, greater size and greater length, usually led to greater complexity. Regarding the two case studies, it seems that, the Patient Treatment case has been modeled in a more complex manner that the Product Exchange case. This assumption arises from the fact that for the majority of teams, Patient Treatment case models had greater complexity than those of Product Exchange.

5.2 Analyzing CMMN models expressiveness

5.2.1 Method

Table 5 Patient treatment expressiveness results

Table 6 Product exchange expressiveness results

The second evaluation parameter for CMMN models that was measured, was the one of models’ expressiveness. In [9], the expressiveness of CMMN is analysed by identifying patterns in CMMN models. These patterns are suggested for CMMN and consist of an adjusted subset of the extended workflow patterns, introduced by van der Aalst in [15] to cover all process types. Additionally, in [9], the authors divide the workflow patterns into three main categories: (i) patterns that are handled by CMMN basic constructs (fully supported), (ii) patterns that rely on CMMN’s engine capabilities (partially or conditionally supported) and (iii) patterns that cannot be handled by CMMN language notation at its current state. In this work, we narrowed the pattern selection criteria to the patterns suggested by [9] that are be fully supported by CMMN. From these patterns we focused on those that could cover the entire range of the studied CMMN notation elements. For this purpose, we chose the most specific ones from the four categories identified in [9]. From the first category (i.e., “Control Flow”), we chose Sequence, Merge and Split, Advanced Branching, Milestone and Cancellation patterns. From the third category (i.e., “Data”), we chose the whole category, as in CMMN notation there is no differentiation between data types, and from the fourth category (i.e., “Exception Handling”), we chose the External Trigger pattern. Note that, from the second category (i.e., “Resource”), we could not choose any patterns as these are not currently fully supported by CMMN.

Following we provide a brief description for each one of the selected patterns:

• Sequence: This pattern establishes that one task can only be enabled after the completion of another task. This pattern uses entry sentries.

• Merge and Split: The two split patterns in this category are the parallel split and the exclusive choice. On the other hand, the merge patterns are synchronization and simple merge. All that patterns use entry sentries to describe conditions.

• Advanced Branching: Some advanced patterns can be represented with CMMN, the multi-choice and multi-merge patterns, that use entry sentries to describe conditions that will determine which branch would be enabled.

• Milestone: The milestone pattern requires the ability of enabling a task based on such achievable targets.

• Cancellation: Cancellation and force completion patterns can also be expressed in CMMN, relying on the Exit Criterion Sentry that can be associated to a task, a stage or even the whole case.

• Data: Using the Case File Item as a precondition to a task, it is possible to represent all of the task precondition patterns.

• External trigger This pattern signals the occurrence of a user event that impacts an item and that requires some form of handling. This kind of signal, in CMMN, can be represented as a user listener.

Example

In order to make clearer, how expressiveness of calculated during the models evaluation procedure, the following example is presented. Again, Fig. 3 is being utilized. For that model, the number of times, each one of the aforementioned patterns appears in it, is counted as follows:

Sequence: 5, Merge and Split: 2, Advanced Branching: 1, Milestone: 0, Cancellation: 2, Data: 1, External Trigger: 0.

5.2.2 Results

The results of the CMMN models expressiveness measurement are projected in Tables 5 and 6. Regarding the expressiveness of the generated models by the participating teams, the general observation is, that all the examined patterns, more or less were identified in the models. To be more specific, for both use cases, all the modeling teams, used at least the majority if not all of the examined CMMN patterns. This is translated to the assumption that all modeling teams designed expressive models, using CMMN standard up to a satisfying level.

In a second level of analysis for the models expressiveness, one could observe that the pattern that was identified most, in the models, was Sequence. In terms of CMMN elements, that practically means that there was an extensive use of Sentries, which is a prerequisite in order to represent sequence between successive tasks. However, that was not a fact for all modeling teams. In both Patient Treatment and Product Exchange use cases, there were teams that used sequence patterns excessively, while others did not. An identical example, are modeling teams M3 and M7. This observation enhances the assumption that there were different modeling perspectives between the modeling teams, not only on one of the use cases, but also between them.

Comparing the two use cases, Patient Treatment was modeled by the participants in a more expressive manner, judging by the mean of each pattern, while, the Product Exchange use case had greater deviation and range between its values.

6 User experience evaluation

As aforementioned, after models’ presentation, the participants took part to a survey evaluating their experience of using CMMN language. The survey’s purpose was to identify the CMMN prospect of being accepted and adopted by modelers as a non-structural process modeling language. To that end, factors related to the usability of a system were adopted. More specifically, Usefulness, Ease of Use and Attitude toward Using were selected. Ease of Use refers to the degree someone believes that a system is easy to use, Usefulness refers to the degree someone believes that a system is useful for her, while Attitude towards Using refers to someone’s desire to use it.

The survey participants recorded their perspective on the usability of CMMN language, as well as their view on the contribution of the new CMMN notation elements, towards the adoption of the language. The survey instrument (questionnaire) included also items that relate the new syntax elements of with the aforementioned usability factors, addressing also the usability of each notation element separately.

6.1 Evaluation procedure

Evaluation questions. The survey questionnaire assessing the users’ experience included the following six (6) questions:

• Q1: How useful was CMMN language to achieve your modeling goal?

• Q2: How easy was it for you to use CMMN language?

• Q3: How much your experience of modeling with CMMN would drive you to use it?

• Q4: Which one of the CMMN elements would contribute most in your Intention to Adopt CMMN?

• Q5: Which usability factor affects most the contribution of each notation element towards your Intention to Adopt CMMN?

• Q6: Which usability factor, when related to the language, could affect your intention to adopt CMMN?

Usability evaluation. The first three questions referred to the perspective of the study’ participants regarding the usability of CMMN language. More specifically, each of these first three questions referred to each one of the usability factors, i.e. the first question (Q1) refers to the Usefulness of CMMN language, the second question refers to the CMMN language’s Ease of Use and the third question refers to the participants Attitude towards Using CMMN.

Evaluation method. The survey items were answered with a scale from 1 to 5 (1 - Not at all, 2 - Not so much, 3 - Somewhat, 4 - Very Much, 5 - Absolutely). Then, descriptive statistics were performed to assess the contribution of each usability factor. The results are presented in Sect. 6.2.1, projected in Fig. 4.

CMMN adoption evaluation. The last three questions referred to the adoption of CMMN language, and the contribution of the new CMMN notation elements (e.g. Stages, Sentries, Milestones, Discretionary Items and Event Listeners), to it, taking also under consideration, how the usability factors of each element could affect their importance to the adoption of CMMN language as well. These questions were answered by the participants within a questionnaire, comparing the new CMMN notation elements to each other, regarding their importance to the adoption of the language. A comparison of usability factors to each other was done within the scope of the questionnaire as well. As the processing of questionnaire answers required comparison of data, the multi-criteria decision making method Analytic Hierarchy Process (AHP) [52] was used (see “Appendix B”), on a web-based tool, previously created by some of the authors². AHP is commonly used in evaluation problems especially in cases where there is a limited number of participants like the one described in this work [13, 14]. The application of AHP to the questionnaire answers as well as the results it produced are described in Sect. 6.2.2.

Evaluation method: analytic hierarchy process. For the processing of questionnaire answers AHP was applied, as it has as a fundamental part the pairwise comparisons, according to which the participants compare the various characteristics or elements in pairs instead of assigning their weights in a single step. This reduces the influence of subjective points of views, associated with eliciting the weights directly [13, 14].

Fig. 4

Participants perspective on CMMN language usability

6.2 Evaluation results

6.2.1 Usability evaluation results

What one could observe in the study’s participants answers, was that for each one of them the majority of the participants had a positive response. None of them, found CMMN specification completely useless or difficult to use, while only a small minority was negative to the prospect of using again CMMN in the future. More specifically, as shown in Fig. 4, for the first question regarding the Usefulness of CMMN, 57 percent of the participants answered positively, while only eight percent of the participants answered that it was not useful for them. For the second question regarding the Ease of Use of CMMN, only 15 percent of the participants answered that the CMMN specification was difficult to use. For the third question regarding the participants’ Attitude towards Using CMMN, only four percent of the participants answered that they would not use CMMN in the future based on their experience. Generally, the answers showed a positive attitude by the participants towards CMMN, as well as a good adoption prospect.

However, despite the good prospect of CMMN adoption, there was no clear evidence about the participants’ attitude towards the specific syntax of CMMN, whether it was considered useful or not, easy or difficult to use and how this contributes to the language’s adoption. For this purpose, the participants were asked to complete a questionnaire that explores how the new CMMN elements contribute to the adoption of this new modeling syntax, also in relation with the aforementioned usability factors for CMMN. Thus, for each CMMN element, one could understand how it contributes to the adoption of the language. For instance, it could be identified whether a CMMN element would be used because it is considered easy to use, because it is considered useful or because people tend to like using it.

6.2.2 CMMN adoption evaluation results

The hierarchy levels of AHP are presented in Fig. 5. The first level deals with the definition of the objective of evaluation, which is to empirically explore the modelers Intention to Adopt CMMN. In the second level, the elements of CMMN, upon which the evaluation will be based, are identified. E1 stands for Sentries, E2 for Milestones, E3 for Stages, E4 for Discretionary Items and E5 stands for Event Listeners. Usability factors F1, F2, F3 which represent Usefulness, Ease of Use and Attitude toward Using respectively, consist the third level of the hierarchy.

Fig. 5

AHP hierarchical model

Table 7 Intention to Adopt - Combination table for questionnaire data

The results of AHP application, calculated with the Eqs. 1, 2 and 3, are projected on Table 7. The column “Intention to Adopt” shows the results of the application of AHP in the first level of the hierarchy (Fig. 5). In that level, the importance of each one of the five evaluated CMMN elements is being identified, through the calculation of relative CMMN elements weights towards Intention to Adopt of CMMN. With the comparison of CMMN Elements importance towards the Intention to Adopt, the forth evaluation question (Q4) is being answered. What is observed is that, two CMMN elements are considered by the participants as the most important towards Intention to Adopt; Stages and Discretionary Items, followed by Sentries. On the other hand Milestones and Event Listeners were not considered so much important.

On the second level of the hierarchy schema (Fig. 5), we relate the studied usability factors with the five new CMMN elements. The results of this interrelation are shown in the central column of Table 7, giving us the big picture about the participants’ answers to the fifth evaluation question (Q5). Regarding the usability of CMMN elements, all of them has been considered useful. This observation is based on the fact that the Usefulness factor had greater score than the average for all the elements. Secondly, despite the fact that some of the CMMN elements were not considered so easy to use, e.g. Stages, than others, e.g. Milestones, the participants Attitude towards Using them was greater. The same observation can be made regarding Discretionary Items and Event Listeners. Note that, again the greatest Attitude towards Using is observed for Stages, Discretionary Items and Sentries. On the contrary, Milestones and Event Listeners have the lowest scores for Attitude towards Using.

Finally, the last row of Table 7, shows the results of the pairwise comparison of usability factors towards the participants Intention to Adopt CMMN, reflecting the answer the last evaluation question (Q6). The results of the usability factors were calculated according to the Eq. 3, described in B.1, which takes into account their scores of each usability factor for each CMMN element. What one could comment is that participants Intention to Adopt CMMN comes mainly from the fact that CMMN is considered useful. Ease of Use has been given lower importance score by the participants, while, in general, participants have a good Attitude towards Using CMMN.

Results validation. In order to validate the results regarding the importance of CMMN elements towards Intention to Adopt, the validation procedure described in B.2 and projected in Fig. 6 took place. As an observation, there was no change in the importance ranking of Stages and Discretionary Items in relation to the other CMMN elements, as well as no overlap between them. Additionally, there was no significant change in the ranking of the other CMMN elements despite of the last two, Milestones and Event Listeners, a fact considered unimportant in this case. In general, this validation procedure, enhanced the observation that for the participants the most important features of CMMN were the Stages and Discretionary Items elements. For the AHP results regarding the interrelation of usability factors with the CMMN notation elements, no verification procedure was applied, as there was no direct interrelation with the evaluation objective of intention to adopt CMMN. In addition to that, to calculate the verification of results for the usability factors, the relative scores projected in the central column of Table 7 were taken in account in the calculation procedure as it is mentioned in B.1.

Fig. 6

Verification of CMMN elements weights towards Intention to Adopt

Fig. 7

PDFs of usability factors final scores

Finally, in order to validate the reliability of the final ranking of the usability factors importance towards intention to adopt, given the level of uncertainties involved by carrying out a sensitivity analysis, the verification procedure described in section B.2 is being followed. In Fig. 7, we show the probability density functions (PDFs) of the final scores of the factors for intention to adopt, which were calculated using $10^5$ Monte Carlo iterations. Figure 7 suggests that the PDFs of the final scores have a small overlap for s = 0.2. This in turn implies that the ranking of the final scores of the factors have a small probability to change (probability of rank reversal). Moreover, the figure indicates that the PDFs of the scores are centered around the value estimated using the AHP, as depicted in Table 7, and exhibit a Gaussian-like behavior.

7 Discussion

First of all, to the best our knowledge, this is the first attempt to evaluate CMMN language’s adoption based on its users experience. As aforementioned in Sect. 3.1, related works can be found in literature for BPMN modeling language, however, for CMMN, there is not such a research attempt. Additionally, it is the first research attempt that takes under serious consideration the syntax, i.e. the notation elements, of the modeling language, for its evaluation, identifying how it contributes to the language’s adoption prospect. Moreover, this study is the first attempt to utilize a multi-criteria decision making method (AHP) for assessing a process modeling language.

Moreover, regarding the models created by the participating teams, the models’ evaluation shows that there was a variety in the modeling styles adopted by the modeling groups. For instance, some modeling groups have adopted two completely different modeling philosophies between each other. More specifically, modeling group M8 seem to have adopted the User-driven style [59], designing models with few sequence and branching patterns identified (see Tables 5 and 6), while modeling group M3 seem to have adopted the ECA style [59], with much more sequence and branching patterns identified, like modeling group M3 (see Tables 5 and 6). However, there were also modeling groups that were adopting a style of modeling according to the case that they were working on. For instance, modeling groups M1 and M5 adopted both User-driven and ECA styles, different each time, for the two different cases.

Concerning the fact that CMMN provides the case designers with the freedom to adopt various styles of modeling [59], according to their perception of the nature of each case, the aforementioned variety of adopted modeling styles was considered an interesting outcome. It shows that the models were not designed with the same philosophy between the participating groups, despite the fact they modeled the same cases. The latter shows that the modeling style each group of participants adopted, as well as their opinion on the studied CMMN notation elements, was not influenced by the nature of the cases modeled. It also reassures that there was no communication between the participating groups during the modeling procedure.

Regarding the syntax of CMMN language, the evaluation showed that according the study’s participants opinion, the newly introduced elements of CMMN, contribute most to the adoption of CMMN. More specifically:

• The elements that were considered the most important ones regarding CMMN adoption, were Stages and Discretionary Items. As one can notice, they had got the highest importance scores, 0.2984 and 0.2704 respectively, regarding which element could contribute most in CMMN adoption prospect (see column “Intention to Adopt” of Table 7). Note that, also from the models evaluation in Sect. 5, what was observed was an increased use of Stages and Discretionary Items, a factor that increased the complexity of the models (see Tables 3 and 4 about models’ complexity), showing a positive attitude towards using them by the participants. It also shows that a fair amount of the participants were fond of modeling in a User-driven style

• Sentries were considered adequately important for CMMN language’s adoption with the aforementioned two elements, getting an importance score of 0.2244, regarding which element could contribute most in CMMN adoption prospect (see column “Intention to Adopt” of Table 7). Again, during models evaluation, an increased use of sentries was observed, as there was a large number of sequence patterns identified, making the majority of models quite expressive (see Tables 5 and 6 about models’ expressiveness).

• On the contrary, Milestones and Event Listeners, were not considered that important. The have got scores of 0.1156 and 0.0931 respectively, regarding their contribution to CMMN language’s adoption, despite the fact that these elements were considered easy to use, getting the highest scores, i.e. 0.4134 and 0.3618, in column “Ease of Use” in Table 7. Such an observation was also made during models evaluation, as milestone and external trigger patterns, which required the use of Milestones and Event Listeners, were not identified that much (see Tables 5 and 6 about models’ expressiveness).

Overall, the results of both models evaluation and user experience evaluation lead to the conclusion that CMMN could be adopted by process engineers for modeling non-structural processes.The study participants found CMMN a quite useful modeling language; “Ease of Use” usability factor got the lowest score regarding its prospect of the participants intention to adopt CMMN, while “Usefulness” got the highest score.

Towards that direction, the understandability of CMMN notation elements could be improved, making the language easier and more appealing to use. Another important issue is the support that CMMN has, regarding its models’ execution. As already identified in [50], CMMN models are being executed, by the majority of existing platforms and tools, over a BPMN engine, lacking full execution support for some notation elements. Future researchers and practitioners could focus on building effective CMMN engines that would provide the optimal execution support for the language’s notation, increasing the execution capabilities for the CMMN models.

8 Conclusions and future work

This research focused on a still unexplored issue, namely the usability and future adoption of CMMN language and the role that the CMMN notation elements could play. After an introductory workshop about CMMN, process engineers modeled, using solely CMMN, two different real-world business cases. The created models were first evaluated, using pre-existing metrics for CMMN models, in order to examine whether there was a good usage of CMMN for modeling purposes. The model evaluation results showed that, in general, the participants performed a good utilization of CMMN language as the generated models’ design philosophy was aligned with the nature of the modeled cases, making the models adequately complex and expressive. Then, the workshop participants’ experience was recorded with a survey regarding the usability of the language, as well as their opinion on the CMMN new notation elements. The analysis of the questionnaire results was performed using the multi-criteria decision making method AHP. The results showed that CMMN has a good prospect of being adopted for human-centric process modeling purposes, with useful new elements that drive the modeler to have a positive attitude towards using CMMN language.

Regarding the limitations of this research work, first of all, the complexity metrics, introduced by Marin in [34], require further theoretical validation, a limitation that should be addressed by future researchers of CMMN. Secondly, we performed one case study to explore CMMN usage and potential for adoption, which provides few evidence on which to generalize. There was a small number of participants, which also narrowed the number of cases for modeling. As we used two specific contexts, Patient Treatment and Product Exchange, we cannot generalize the results beyond those two different domains. The latter urges future researchers to test whether the results of this work could also be identified in different domains. As CMMN is being supported by well established modeling tools, future researchers could evaluate CMMN language’s usage, applying the evaluation approach proposed in this work to other domains, such as legal or banking.

As a next step, we intend to use the results of this study towards a more rigorous investigation of CMMN usage and adoption utilizing, e.g. multiple case study design, which will provide evidence from many sources and involve a larger audience, thus making it feasible to generalize the conclusions drawn. More participants and more domain experts will be the target audience and different domains.

Furthermore, we aim to explore the subject of execution of CMMN models, which is important aspect for a new modeling language such as CMMN. To this end, the degree of automation of making CMMN models executable in practice should be explored, through the definition of execution requirements for language elements and models.

The main issue regarding CMMN models execution, is that currently, the majority of modeling tools and platforms, supporting CMMN, provide for the execution of CMMN models over a BPMN engine. There are established modeling environments, such as Flowable and Trisotech investing on CMMN execution engines. Though, the degree of automation for the overall execution process remains an issue and should be balanced with flexibility provided during execution.

Appendices

Appendix A: Cases description

1.1 A.1 Patient treatment

To begin with, the patient is admitted to a hospital’s Medicine Clinic if he/she needs to be hospitalized, a decision that is taken at the Emergency Department. The Emergency Department personnel provide the physicians of the clinic with information regarding the clinical status of the patient, such as medical history and any examinations that have been done or scheduled.

Based on this initial information, the physicians of the Medicine Clinic start the treatment of the patient. They specify a diagnosis for the patient and prescribe the medication accordingly. Such information is registered into the patient’s file. During treatment, a clinical examination takes place every morning by the physicians, aiming at monitoring the patient’s clinical course. To this end, laboratory and/or imaging examinations may be scheduled. The results, which are also registered into the patient’s file, are evaluated by the physicians and if necessary the diagnosis and medication are revised. There may be cases that the physicians will need to consult a specialist in order to conclude about the patient’s health problem or about the way the patient should be treated.

The nursing personnel aids in the treatment process through operations regarding, for example, the preparation and administration of the specified medication, blood drawing and measurements of vital signs. Medication administration and measurements are performed at the times specified by the physician. The measured values are written in the patient’s chart. Moreover, the nursing personnel keeps notes of anything remarkable regarding the patient, for example, a sign they observed, as well as of any action they performed by their own initiative, for example any ad-hoc medication they may have administered to the patient.

During treatment, several unexpected situations may arise, which may lead to ad hoc clinical, laboratory or imaging examinations, as well as to reconsideration of the medication administered or even of the diagnosis specified so far. The patient may need to be transferred to the Intensive Care Unit or to undergo an urgent surgery.

The need for a patient to remain hospitalized is daily examined after the morning clinical examination based on the data gathered up to that point. If it is decided that the patient does not need further hospitalization, the treatment process ends and the patient is discharged.

1.2 A.2 Product exchange

First of all, there are two types of users, i.e., guest and registered users that differentiate themselves in the permissions that they get granted regarding the use of the platform.

More specifically, when someone visits the web platform for the first time, he gets prompted to register, by creating a user account. This account can be created either by signing up via an email and a password or via a social network account. After a successful registration, the, from now on, registered platform user, is able to submit an advertisement donating or exchanging an item, to declare interest for an existing EE product and propose an offer to acquire it, as well as to communicate with any other user who owns a desirable electric device.

Moreover, a registered user is not only able to search a product based on some conditions, namely, filters like item categories, item state, donating-user region, but also to either suggest changes regarding the item’s category for which he/she is searching, or even to comment in an advertisement that he/she had expressed interest for. That way, the appropriate users will be notified for either the category change proposal or the commenting in an advertisement.

Finally, registered users have a profile in which they are able to be notified for any recycling actions taken via a news-feed as well as being informed for general topics regarding recycling and its benefits. Within each user’s profile, a calendar exists via which a user can be informed for any recycling events taking place.

Appendix B: Analytic hierarchy process

1.1 B.1 Methodology description

The AHP adopts a hierarchical form using three conceptual levels, as it is projected in Fig. 5. In the first level, the objective of the decision making process is defined. In the next level, the number of elements on which the evaluation will be based is identified. The various elements are denoted by $E_{k}$, where k is an integer with $1 \le k \le N$, and N is the total number of elements. In this paper we consider the CMMN elements. In the last level of the hierarchy we have the factors, denoted by $F_{i}$, where i is an integer with $1 \le i \le J$ and J the total number of factors. [12].

According to the AHP, in order to explore which of the CMMN elements contributes most towards the intention to adopt CMMN, then one must perform pairwise comparisons and evaluate the weights of importance of each CMMN element. In the same context, we have to explore the scores of each factor $F_i$ for serving $E_k$ Pairwise comparisons is a fundamental part of the AHP, according to which the participants compare the elements/factors in pairs instead of assigning their weights in a single step. This reduces the influence of subjective points of views, associated with eliciting the weights directly.

Each participant m ($1 \le m \le M$, where M is the group of participants), compares all possible combinations of CMMN elements by filling out the N x N pairwise comparison matrix $\mathbf{P} ^{( m )}$=$[P_{ij}^{(m)}]$, the elements of which signify the importance of a CMMN element $E_{i}$ compared to another CMMN element $E_{j}$ towards Intention to Adopt CMMN, assigning values from the nine-level scale[12]. The participants need to complete only the upper triangular elements since PWC is a reciprocal matrix. The weights $w_{k}^{(m)}$ of a CMMN element $E_{k}$ according to participant m are calculated by solving the eigenvalue problem, according to which the eigenvalues of $\mathbf{P} ^{( m )}$ are calculated and the eigenvector $\mathbf{x} _1^{( m )}$ = $[ x_{1k}^{( m )} ]$ associated with the largest eigenvalue $\lambda _{max}^{( m )}$ is determined [52]. The weight $w_k^{( m )}$ are obtained normalizing the sum of the eigenvectors $\mathbf{x} _1^{( m )}$ of the matrix to unity,

$$\begin{aligned} w_k^{(m)}=x_{1k}^{(m)}\left[ \sum _{l=1}^{N}x_{1l}^{(m)}\right] ^{-1} \end{aligned}$$

(1)

After all the comparisons have been completed, the average weight $w_{k}$ for each element $E_{k}$ is calculated by averaging out the weights $w_{k}^{(m)}$ obtained by the M participants.

$$\begin{aligned} w_{k}=1/M\sum _{m=1}^{M}w_{k}^{(m)} \end{aligned}$$

(2)

Care should be taken so that the pairwise comparison matrices produced by the participants are as consistent as possible in terms of proportionality and transitivity [8]. The PWC matrix $\mathbf{P} ^{(m)}$ is said to be perfectly consistent if all its elements are of the form $P_{ij}^{(m)} = q_i^{(m)}/q_j^{(m)}$, where $q_i^{(m)},q_j^{m}$ are positive real numbers.The consistency ratio (C.R.) is one measure for consistency can be readily obtained from the pairwise comparison matrices as described in [8]. In our case, the C.R. values were less than 0.1 (ranged between 0.015 to 0.076) which is considered acceptable [39].

The same procedure is followed for the factors of the second level of hierarchy. Towards this end, the factors are pairwise compared with respect to each element and for each factor $F_{i}$ one obtains the relative scores $S_{ik}$ under element $E_{k}$, depicting the score of $F_i$ for serving $E_k$. Finally, one can evaluate to what extent each acceptance factor contribute to the Intention to Adopt CMMN, by multiplying the relative scores $S_{ik}$ by the weight $w_{k}$ of the corresponding element and estimate the overall weight $R_i$.

$$\begin{aligned} R_{i}=\sum _{k=1}^{N}S_{ik}w_{k} \end{aligned}$$

(3)

1.2 B.2 AHP results verification

Changes of the group of participants. The AHP decision process, performed for the quantitative evaluation, involves pairwise comparisons commonly used in evaluation problems with a limited number of participants, since by augmenting the size of the participants beyond 15 there is no significant change in the final outcome [13]. As in our case there are 24 participants, it is very interesting to investigate the impact of modifying the group size of participants, from 24 to 15, on the AHP results. More specifically, it would be interesting to investigate the consistency of AHP results regarding the importance of elements towards the modelers intention to adopt CMMN, namely the results calculated with Eqs. 1 and 2. Such a verification would be very important as it is related with the objective of the evaluation, as it is projected in Fig. 5. In this context, we perform Monte Carlo simulations [51] of $N_{MC}=10^5$ iterations. For each iteration z ($z\le N_{MC}$) we randomly ignore a group of 9 participants and estimate the average weights of elements $W_k^{(z)}$ for the new group of $M=15$ participants. Finally, we estimate the average weights from all iterations $W_k$.

$$\begin{aligned} w_k= \frac{1}{N_{MC}}\sum _{z=1}^{N_{MC}}W_k^{(z)} \end{aligned}$$

(4)

Changes of elements and factors weights. In order to further validate the reliability of the final ranking of the factors for intention to adopt, given the level of uncertainties involved by carrying out a sensitivity analysis, Monte Carlo simulations are performed by simultaneously changing more than one parameters. The weights of all elements and the relative scores of factors are perturbed from $w_k$, $S_{ik}$ to $w_{k}(1+\varDelta w_k)$, $S_{ik}(1+\varDelta S_{ik})$, respectively, where the perturbations $\varDelta w_k$, $\varDelta S_{ik}$ are assumed zero mean, identically distributed, independent random variables uniformly distributed inside [$-s s$] [12]. Such random perturbation may be due to inconsistencies of the pairwise comparison matrices [53].