Keywords

1 Introduction

Process mining is a technique that is designed to discover, monitor and improve actual processes (i.e. not assumed processes) by extracting knowledge from event logs commonly available in today’s information systems [1]. Process mining is used to improve performances of business processes and analyze compliance to business rules [1] and to achieve digital transformation in organizations [2, 3]. This technique is becoming more popular [4] and the process mining market is growing fast. It is estimated that there are approximately 35 process mining vendors offering process mining products and services. The process mining market for new product license and maintenance revenue is valued at $550 million in 2021, a 70% annual growth compared to 2020 [5]. Despite the market growth of process mining, limited research exists on the effectiveness of application of process mining in organizations [4, 6]. The majority of studies on process mining focus on technical aspects of process mining, e.g. developing process mining techniques and improving algorithms [7,8,9]. There is a clear imbalance between the amount of research conducted on process mining applications on the one hand, and adoption in organizations and the increasing popularity and market growth of process mining on the other hand [4, 10, 11]. Moreover, limited research exists on the application of process mining project methodologies within organizations. These methodologies are important as they give practitioners guidance and support in applying process mining in organizations, stimulate the adoption of process mining in organizations, aid in sharing best practices and prevent reinventing the wheel [12, 13]. Yet, to the best of our knowledge, existing process mining methodologies have only been scarcely validated in one or just a few case studies [12,13,14]. As a result, it is difficult to assess to what extent existing process mining methodologies effectively support the application of process mining in organizations. Clearly, there is a need for a broader validation of current process mining methodologies.

In this research the following research question has been developed; “What are the differences and similarities between current process mining methodologies and process mining practices at local governments and consultancy firms in the Netherlands?”. To answer this research question, four existing process mining methodologies have been critically reviewed and compared to process mining project elements derived from practitioner experiences with process mining. In total, 27 interviews have been conducted with process mining experts of consultancy firms and professionals at local governments. This allows for a broader perspective than the current limited amount of case studies available in literature to validate existing process mining methodologies [12,13,14]. Based on this comparison, we identify gaps between existing process mining methodologies and practitioners. Recommendations for improvements to (future) process mining methodologies are suggested with the aim to support the enhanced adoption and usage of process mining in practice. The remainder of this paper is structured as follows. In Sect. 2 various process optimization methodologies, process mining methodologies and process mining challenges and enablers are discussed. The used methodology is described in Sect. 3. Section 4 includes the results and Sect. 5 includes the conclusion, limitations and suggestions for future research.

2 Theoretical Background

This section provides an overview of related work on process optimization methodologies, process mining methodologies and enablers and challenges of adoption of process mining in organizations. Given the vast amount of literature on business process optimization and management, we do not strive for a comprehensive overview. Rather, we present a selection of key studies that represent the main lines of research in these areas. We carried out an extensive search for process mining methodologies and enablers and challenges, resulting in only a limited number of studies available in the literature. This literature survey approach served our aim of illustrating and positioning process mining methodologies and evaluating their use and validation in practice.

2.1 Process Optimization Methodologies

Business Process Reengineering (BPR) can be defined as the fundamental rethinking and radical redesign of business processes to achieve dramatic improvements in critical, contemporary measures of performance such as cost, quality, service and speed [15]. Based on a review on existing BPR methodologies, a BPR methodology for executing BPR in organizations is developed [16]. The methodology analyses the as-is process to identify bottlenecks in the current process, the design of a to-be process, and implementation of the reengineered process. It delivers continuous improvement by initiating ongoing improvement measures. But the method lacks validation by organizations and practitioners. Before starting a process optimization project, organizational readiness and commitment is crucial. A significant need for the process to be reengineered is vital [16]. Also, egalitarian leadership, collaborative working environment, top management commitment, supportive management, and use of information technology are positive indicators when assessing organizational readiness for BPR [17].

Business Process Modelling (BPM) is the discipline that combines approaches for the design, execution, control, measurement, and optimization of business processes. There is a trend to increase focus on process monitoring, adjustment and process diagnosis [1], simplicity, predictions, more extensive cooperation in organizations, anticipating on customer needs and optimizing processes using design-by-doing and optimization iterations [18]. BPM has distinct disadvantages. Factual process data is not always used in redesigning processes and process related decisions. Various stages of BPM are not supported in a systematic and continuous matter and only severe process problems will trigger another iteration in the BPM life cycle when designing or improving the process [1].

Data mining techniques aim to analyze large datasets to find unexpected relationships, and summarize data in an understandable way [19]. The CRoss Industry Standard Process for Data Mining (CRISP-DM) and Sample, Explore, Modify, Model, and Assess (SEMMA) are two widely used data mining methodologies [13]. Both methodologies have limitations regarding the length of the process, selection of data, and needed knowledge on data mining when executing a data mining project [20] and missing guidelines for organizations on how to conduct deployment in data mining projects [21]. SEMMA is considered highly technical and there is a lack of clarity on how to apply new knowledge obtained by data mining. Both methodologies are very high-level, highly complicated [22, 23] and provide little support for process mining methodologies [24]. The focus lies on modelling by using elements such as Petri Nets and analysts are reluctant to use them as they are discouraged by the method complexity, the work needed for preparation of the mathematical model and the difficulty in comprehending and interpreting the results [25, 26].

2.2 Challenges and Enables of Process Mining in Organizations

In order to compare the process mining methodologies identified above with experiences and best practices, we describe process mining challenges and enablers of process mining in organizations. The identified challenges focus mainly on event log and data quality issues [10, 27, 28], selecting appropriate process mining processes [4, 29] and vendors [11], business case calculation and implementation of improvement actions, and making process mining a continuous effort in organizations [4]. These challenges have also been mentioned in the process mining manifesto [30] which also includes improving understandability and usability by non-experts as key challenges. Identified enables of process mining methodologies are actionable insights, confidence in process mining, perceived benefits, and training and development [10], managerial support, project management availability, resource availability, process mining expertise, and data and event log quality [31]. It has to be noted that these process mining challenges and enablers have not been widely empirically validated. Expert views, case studies, surveys or field tests on process mining enablers and challenges are scarce and not systematically studies.

2.3 Process Mining Methodologies

As data mining projects were not tailored towards process mining projects, the L*life cycle model [1] was coined as one of the first process mining methodologies. This model focuses on process mining projects of structured ‘lasagna’ processes. The Process Mining Project Methodology (PMPM) and the accompanying process mining life-cycle model [12] was designed in response to the lack of process mining methodologies that provide guidance how to apply process mining in practice. The authors of the PM2 methodology [13] state that the main bottleneck of previous process mining methodologies was the lack of iterative analysis which the authors considered vital. The process mining project proposal [14] was developed as a response to [1, 12, 13]. Previously developed process mining methodological approaches developed provide mostly generic guidelines, but do not define the specific steps and tactics for the challenges that a practitioner must go through when facing a process redesign project through process mining [14]. The L* life-cycle model stages neither reach the necessary level of detail, nor define the specific steps to be followed when it comes to developing a process mining project [14]. The PMPM methodology [12] does not deepen into key aspects such as project planning and data preparation and extraction from the different information systems. The difference of the process mining project proposal compared to previously mentioned methodologies [1, 12] is that it was developed using an engineering design science methodology. Moreover, it was evaluated in three case studies and emphasizes data preparation from different data sources. In Table 1 we present an initial comparison of the methodologies discussed in this section based on their structure, validation, and limitations mentioned in the paper of the respective authors.

Table 1. Initial comparison process mining methodologies.
Full size table

3 Methodology

3.1 Practitioner and Expert Interviews

In order to gain in-depth insights in practitioner experiences in process mining projects, semi-structured interviews have been conducted. Semi-structured interviews are suited for gathering independent thoughts, allow for follow-up questions on unclear or interesting answers and aid in examining uncharted territory with unknown possible direction of answer given [32]. As there is limited research and validation conducted on the adoption of process mining and process mining methodologies in organizations [2, 4, 11, 13, 14] semi-structured interviews provide the flexibility to explore this relatively under-researched research topic and aid to the exploratory nature of this research. In total 19 interviews with professionals at 15 local governments are conducted. Using simple random sampling on all Dutch local governments, 9 local governments are selected. The remaining 6 local governments are selected based on the personal network of the researchers. Also, 8 experienced process mining experts working at consultancy firms are interviewed. In total 5 of these interviewees were selected by conducting a Google search on keywords ‘Process mining AND experts’, and ‘process mining AND consultancy’. The remaining 3 interviewees were selected based on the personal network of the researchers.

Using the identified stakeholders of the process mining methodology of [13] and the processes that were analyzed in previous research as identified in Table 1, interviewees were selected based on having knowledge of process steps identified in Table 1, or were analysts responsible for analyzing processes in their organization. As a result, the interviewees have a variety of roles, such as data-analysis, BI analyst, project manager, financial controller, process manager, innovation consultant and process consultant. The interviewees work in organizations of different sizes, ranging from large (500+ FTE), to medium-sized (<500–500 FTE and smalls (<50 FTE). The interviews took place between March 2021 and October 2021. Detailed information on the interviews is available upon request. As only a few studies area conducted into the adoption and usage of process mining [4, 6] and process mining adoption and process mining methodologies can still be considered in its infancy, the interview questions were of explorative nature. Therefore, the interview questions focused on process mining familiarity, desired process insights, process optimization- and mining bottlenecks, involved stakeholders and the steps followed when executing process mining projects. All interviews were transcribed and summarized. Every interview transcript is given an anonymous abbreviation. For the local governments this ranges from LG1 until LG 19, the used abbreviations for the expert interviews are E1 until E8. Parts of the interview transcripts were labeled using the interview topics as described above. Next, similarities and differences were identified between the various labeled interviewee transcripts, also taking into account the role of the interviewees and organizational size.

3.2 Gap Analysis

A gap analysis is a tool or process to identify gaps, or differences between the organization’s current situation and expectation, or “what ought to be in place”. Gap analysis indicates areas where managers should take action to narrow the gaps between current situation and expectation, hence improving organizational effectiveness [33]. Gap analysis consists of identifying an organization’s needs, highlighting the gaps and implement plans to fill the gaps. In this paper, organizational needs are identified by conducting semi-structured interviews, and are deducted from the theoretical framework on current process mining methodologies and its limitations, and process mining enablers and challenges. Using this input relevant elements/criteria of a process mining methodology are identified. These elements/criteria are clustered based on the stages of a process mining project. Using these criteria, a gap analysis has been created in which the four process mining methodologies mentioned in Sect. 2.3 are compared against. Actual implementation of plans lies outside the scope of this paper. The gap analysis aids in systematically assessing the extent to which current methodologies reflect process mining practice in organizations, and which steps or alterations can be made to (future) process mining methodologies to improve connection with process mining practices in organizations.

4 Results

In the following the key outcomes of the semi-structured interviews are gap analysis are included.

4.1 Local Government Professionals

Familiarity with Process Mining

Familiarity and experience with process mining projects is most often seen at large-sized local governments. Process mining projects often initiate out of personal interest in BI- or data analysis, driven by a passion for data and the need and urgency felt to digitalize. Process mining is often at an explorative phase, resulting in isolated efforts only known at BI departments. Process mining pilots often run for years mainly due to data quality issues, lack of trust in process mining and no felt urgency of management in process mining and process optimization. “It turned out to be very difficult to make a solid business case for process mining, as management only notices the investment, and not the added value and revenue that could be generated. All in all, it took approximately 2 years before we could start” (LG3). Interviewees working at small-sized local governments often wondered if their organization was ready for process mining. “The first impressions is that process mining requires a kind of maturity that not all local governments have” (LG8).

Desired Process Insights and Process Optimization Bottlenecks

The interviewees unanimously mentioned a need for more insights in actual process steps, throughput time and reducing this throughput time to save cost and meet goals in reducing throughput time. Compliance related matters such as segregation of duty and execution of authorized activities were identified as valuable process insights. “Are people doing the right thing, or are they violating their authorizations? This is important, because we have rules for this in the municipality” (LG9). Experienced process mining users mostly mined the financial processes because of data availability, expected cost savings and compliance violations, not meeting process KPIs, understandability of the process, and no involvement of many departments. Having knowledge about the process before conducting a process mining project, management support, communicating quick wins and improvement actions, and cooperation between stakeholders is considered crucial and triggers analyses interpretations and improvement actions. The main identified process mining bottlenecks are the time-consuming formatting of the data for process mining, and missing- or incorrect insertion of data in the data source. “People must be made aware of incorrect data registration and the consequences of not registering the data properly must be communicated to the involved employees” (LG2). Experienced process miners mention that the tools used only allow for the creation of a single process mining map, while dashboarding with own selected KPIs and continuous monitoring is preferred. “We want to monitor improvements in lead time or failure to follow the process in a dashboard, where findings are preferably expressed in time or money savings” (LG12). The interviewees mentioned that process optimization is done ad hoc, not at all, based on gut-feeling. Organizational bottlenecks for process mining found are convincing the management board on the added value of process mining, the complex IT landscape with many data sources, the unfamiliarity of process mining in the organization, unclarity about responsibilities, unclarity about the actual process steps due to lack of process step documentation, and finally the lack of organizational urgency to digitalize the processes were mentioned “It is unknown how these processes are currently monitored, that is hardly done. Not every team is in control. However, we feel pressure to digitize processes and to improve process documentation” (LG7).

Stakeholder Involvement and Process Mining Methodologies

Involved stakeholders, or desired to be involved stakeholders mentioned in process optimization- or process mining are the BI analysts who make the process map and conduct process analysis, the management board which approves the investment in process mining resources, the IT department for advising on and implementation of applications in the municipality application landscape and finally the process owners which analyze results and steers improvement actions for their processes Remarkably, besides one local government, the process mining efforts and analysis still remained at the BI department, and a thorough analysis, implementation and setting up improvement actions together with the identified stakeholders did not take place. Process mining methodologies were not specifically followed.

4.2 Process Mining Experts

The financial processes were mostly mined because of understandability of these processes, urgency in the organization to improve the process, Lean Six Sigma projects and the availability of data. “The starting point is always the company goals, and whether data is available. In reality it turns out that 99% of the times this is the purchasing process” (E2). Projects often focused on process discovery to gain insights in actual process steps and throughput time. Reduction of throughput time and compliance-related insights were mentioned, such as authorization of activities and segregation of duty. Enablers of process mining in organizations found are the ability to make a dashboard with own selected KPIs, an affordable purchasing price of tools, simplicity of the tool (no programming) and the ability to execute process mining on a continuous basis. Also mentioned were using correct data, involvement of process owners to being able to interpret the analysis and make changes in the organization, a data-driven culture, support of top-managers, start small to gain trust and support, focus and dedication to the project, link to business goals and communication of results in the organization to create enthusiasm and familiarity with process mining. Using a workshop on (the added value of) process mining to help clients interpret and read a process mining map was considered vital, as often process stakeholders were not familiar with process mining and maps were not considered easy to read and interpret. The bottleneck that was mentioned by all experts was the data quality relating to incorrect data formats and the incomplete insertion of data in systems by employees. “It was difficult to get the data out of the systems, resulting in less trust in having good quality available data. The replies were often “we have tried that before but it did not work” (E5). Sometimes process mining projects were terminated due to unavailability of data. Other bottlenecks encountered in the execution of a process mining project are the unfamiliarity with process mining hence showing the added value of process mining, identifying process owners, working with fragmented systems and various data sources, and not-well documented process descriptions.

Process mining projects frequently started with an explanation of process mining and its added value to the various stakeholders involved. Research question and KPIs were developed and sometimes linked to business goals. Most time-consuming was the data extraction and cleaning. For the analysis often Disco or ProM was used, but also UiPath and Minit were mentioned. The interviewees favored the ease of use of Disco and the ability to make dashboards with UiPath and Minit, but were less satisfied with some tools’ inability to include own selected KPIs in the dashboard, the high purchasing price and the limited perceived ability to execute process mining on a continuous basis. The analysis phase often started with the creation a rough sketch of a first dashboards, followed by designing more detailed dashboards and discussing the outcome with the customer and interpret the analyses. Bottlenecks were listed, and it was determined together with the process owner which improvement actions would yield most result (in time- and cost saving) and took least time to implement. After selection the improvement actions, the improvements were implemented and monitored. At various stages of the project, analysts, process owners and management was involved. What stands out is that several iterative customer validation rounds took place at various project stages. Validation was related to whether customers could relate to the data, to verify if the dashboard answered the customer question and was understandable, and regarding the conclusions drawn from process mining analyses. Validation took place multiple times and has a iterative nature “We do not know the process as well as the process owners. At the beginning we draw conclusions that were not recognized by the process owners. Together with the process owner we validate whether they recognize our findings. This is an ongoing iterative process” (E8).

4.3 Elements of Process Mining Methodologies

Combining the findings in the literature reported on in Sect. 2 and the interviews, the elements that can be considered relevant in process mining projects can be derived (see Table 2). The elements in italic were only identified in the interviews, the remaining elements were identified in both the interviews and the theoretical section of this paper.

Table 2. Relevant elements in a process mining project
Full size table

4.4 Comparison Table Gap Analysis

To identify differences and similarities between existing process mining methodologies and process mining practitioner experiences, the identified elements of Table 2 are used to compare the four identified process mining methodologies of the theoretical framework on. Table 3 includes an overview on the differences and similarities of the identified process mining methodology elements against the existing process mining methodologies.

Table 3. Comparison framework process mining methodologies
Full size table

4.5 Gap Analysis

All methodologies include determining a project goal, formulating a problem definition and research questions, and data extraction and preparation to be suitable for process mining. Analyzing results and defining improvement actions is present in all methodologies. Not all methodologies focus explicitly on the involvement of stakeholders in process mining projects and defining improvement actions. [1] focuses mainly on stakeholder involvement in the phase where the project goals and questions are derived and [12] includes theoretical scenarios of stakeholder involvement and mentions involving an analysis, project leader and manager in the evaluation phase. Involvement of analysts, process owners, managers and IT specialists is more included in [13, 14]. Not all methodologies focus on KPI specification and interpretation and drawing conclusions. Only [14] strongly focus on the quantification and selection of improvement actions. Findings in the theoretical framework [16, 17] and the interviews indicate that organizational willingness and data availability are crucial. Lack of these elements can even lead to the termination or not setting up of process mining projects. It therefore is questionable if and to what extent every organization is suitable for process mining projects. Also, hardly any attention is paid to process vendor- tool- and process selection while these are identified process mining challenges in the theoretical framework [4, 11] and in the interview results. Often Disco or ProM was used as a tool, but the choice of vendor and tool selection lacks argumentation. And this is remarkable, as there are approximately 35 process mining vendors with associated tools [5]. Establishing familiarity with process mining and efforts to increase the familiarity of process mining was only mentioned at [13], while the theoretical framework [10, 30, 31] and the interview findings indicate the challenge regarding the low level of familiarity with process mining in organizations. Finally, the results indicated that creating a process mining dashboard with the flexibility to include self-selected chosen KPIs and process mining projects as a continuous effort in the organization important is important, but is not part of the process mining methodologies. The methodologies of [1, 12] share least similarities with the process mining project elements. These methodologies are relatively theoretical, do not explicitly identify various stakeholders in various stages of process mining projects, do not focus much on vender- and process selection, quantification and selection of improvement actions and especially [1] focus least on iteration, validation and process mining as a continuous effort. Because of the iterative nature, stakeholder involvement, various validation efforts and focus on quantification and selection of improvement actions, the methodologies of [13, 14] have most similarities with process mining practitioner’s experience. But none of the methodologies include all relevant elements as identified in this research.

5 Conclusion and Discussion

This study identifies the differences and similarities between existing process mining methodologies and process mining practitioner experiences. Similar elements identified at both the practitioner experiences and the methodologies are goal determination, problem definition and research questions, data extraction and preparation, analyzing results and defining improvement actions. However, none of the existing process mining methodologies include all process mining project elements as identified during this research. These are among others tool-, vendor- and process selection, organizational willingness, communication of quick wins, and quantification and selection of improvement actions. This research contributes to understanding the gap between process mining methodologies and practitioner experiences. The first limitation of this study concerns the generalizability. The interviews were held with professionals and process mining experts at local government agencies and consultancy firms. Hence, we provided only a partial view on process mining initiatives. More research on process mining experiences is needed adding to the completeness of process mining experiences and relevancy process mining methodologies. In addition, general recommendations were made to process mining methodologies indicating that there is one process mining methodology, while different type of organizations might benefit from different process mining methodologies. Therefore we recommend the in the future to be developed process mining methodologies to be validated with more case studies in various sectors in order to increase the generalizability of process mining project elements and methodologies identified in this research. It is expected that including these elements derived from practitioner experiences will aid in giving practitioners guidance and support in applying process mining in organizations, stimulate the adoption of process mining in organizations, provide support in overcoming currently identified process mining challenges and prevent reinventing the wheel.