Process Mining of Periodic Rating Scale Survey Data Using Analytic Hierarchy Process

Abstract

The main purpose of our research is to propose original algorithm to evaluate the dynamic behavior of processes from the survey data collected with the help of periodically repeated surveys based on Likert scale questions. This approach supposes the usage of AHP (Analytic Hierarchy Process) for assessment the factors influencing the process behavior. Our idea is to use the aggregated periodic rating scale data as alternatives inputs for AHP evaluation. The practical usefulness of proposed process quality evaluation technique was proved by examining particular Polish rehabilitation hospital service quality changes over time frame from 2008 to 2017.

1 Introduction

To explore the run of some processes that usually are dependent from behavior, sentiments and satisfaction of participating members, we apply the questionnaires, surveys to learn the participants’ opinion about the process they are involved. More often this situation appears when evaluating the quality of service provided by social infrastructure, healthcare level secured by some hospital, knowledge level achieved by educational institution, satisfaction from used service or purchased goods. To examine such process behavior usually is used rating scale data obtained from public opinion poll or survey with Likert scale questions. A Likert scale let us to measure attitudes or opinion of respondents and rate the answers on a level of agreement from ‘strongly agree’ to strongly disagree [Joshi et al. 2015; Murray 2013].

There are a lot of methods to analyze the rating scale data. Recommendations for analysis, interpretation, literature review and reporting of scores derived from Likert-type scales are presented in recent work of [Robert Warmbrod 2014]. The [Jamieson 2004] outlines some common pitfalls seen in practice when using Likert scales. The article of [Harpe and Pharm 2015] explains the situations, when parametric and nonparametric analytical techniques is more appropriate for rating scale data.

The Likert scale data is easy to summarize but, hard to interpret. You can compare the responses with results from previous similar survey or find some in advance known process behavior description. Usually such comparison is hard realized. So the task of constructive interpreting the Likert scale responses leaves to researcher. Usually they use simple statistical location and variation characteristics like mean, median, mode, standard deviation, percentile, its confidence intervals and so on.

In this paper we analyze more complicated situation. We study the process changes over time by the help of repeated surveys. We want to find the way for analyzing and comparing the data retrieved from multiply Likert scale based surveys. The proposed research method is described in Sect. 2. The practical implementation of introduced testing idea on the data of rehabilitation hospital service quality changes over time is done in Sect. 3. The paper is finished with main findings and final conclusions of research done.

2 Research Method

The main task of the research is to propose the algorithm for process mining of periodically collected numeric rating scale data by using AHP (Analytic Hierarchy Process).

Let us assume we investigate behaviour of some process. It may be quality of healthcare in some hospital, educational achievement for given school or university, service level of some society and so on. The general approach of exploring patterns of process changes is based on selecting some factors influencing this process by different weights (see general framework in Fig. 1).

Fig. 1.

Framework for process investigation

To investigate the factors, we can employ the survey with rating-scale (Likert scale) questions.

The main problem is how to interpret the answers and get the meaningful assessment of factor importance. Other challenge arises in case we collect survey answers periodically and look for process changes along the specific time frame. These problems make the core of our research in this article.

Let the $\left\{ {F_{i} } \right\}_{i = 1}^{n}$ denotes the set of factors. The survey rating-scale questions we assigned to most appropriate factors. The set of $Q_{1}^{i} ,Q_{2}^{i} , \ldots ,Q_{ki}^{i}$ denotes questions assigned to $F_{i}$, having any quantity of ordered response levels. For the research of the process evaluation task we employ AHP method (Saaty 1980; Ishizaka and Labib 2011; Kriksciuniene and Sakalauskas 2017). Application of AHP means that we exercise four major steps:

• 1. Problem modeling in the hierarchical form;

• 2. Valuation of factor weights by pairwise comparisons;

• 3. Weight aggregation into the priority vector by using the eigenvalue method;

• 4. Ranking the decision alternatives.

The factors of quality evaluation of the hospital were arranged into two level hierarchical tree (Fig. 1). The experts have been employed for the group decision of the consistent factors weights. During AHP evaluation procedure they use the pairwise comparison of factor importance and merge joint decision as a priority vector (Saaty 1980). The $W = \left\{ {p_{i} } \right\}_{i = 1}^{n}$ denotes the generalised and consistent vector of weights of the factors forming first level of the decision tree.

The last step of the AHP method consists of selecting and ranking the decision alternatives which, in general, are defined as entities or objects characterized by the weighted hierarchical tree of factors. For this stage different types of decision alternatives can be discussed. It highly depends on the process we try to investigate; the factors we select for analysis or type of survey.

In our case the decision alternatives have be defined as time periods to be evaluated and ranked according the AHP algorithm. This viewpoint is innovative for mining data of numeric rating scale survey.

Application of time period as a decision alternative enables to find the best time periods, see process changes over time and identify the factors with the highest impact on the process changes.

To rank the periods alternatives, we need to calculate the questions reliability from survey results according to selected period.

The reliability calculation of chosen ordered rating-scale questions is not easy task. Firstly, we should agree how to interpret a single rating-scale answers. Assume we have R respondents answers to s-level rating scale question. The frequency of answers is presented in Table 1, where M₁+ M₂+ … + Ms = R.

Table 1. Frequency table of answers

The aggregation the answers to single indicator can be done by many different ways. We can use top-box or top two box scoring, net top box, customer experience index, Z-score to percentile rank and other raw responses aggregation to single indicator [Measuring U 2018; Joshi 2015; Murray 2013; Subedi 2016].

For answers aggregation to single indicator we introduce new method, which takes count not only on content of answers, but also consider the response rate of respondents. This let us get more reliable rating-scale questions evaluation.

As we have ordered categories, accepting its degree of arbitrariness, we might give scores to the categories from 1-‘strongly disagree’ to s-‘strongly agree’, and then produce an aggregated reliability index using formula:

$$ ARI_{s} = \frac{{M_{1} \cdot 1 + M_{2} \cdot 2 + \ldots + M_{s} \cdot s}}{{(M_{1} + M_{2} + \ldots + M_{s} ) \cdot s}} $$

(1)

$ARI_{s}$ is positive and equals to 1 only in case all respondents choose ‘strongly agree’ answer. So the bigger is $ARI_{s}$ the more respondents agree with our statement.

In other hand, we need to assure sufficiently active participation in survey. The number of respondents directly affect the significance of the research. Consequently, we advise to supplement the aggregated reliability index with the response rate of respondents. Let U-stands for the number not-responded participants to given question. Then for selected alternative we can define generalized respondents aggregated reliability index (RARI), which takes account on the response rate of respondents:

$$ RARI_{s} = ARI_{s} \cdot \frac{R}{R + U} $$

(2)

This index takes a value equal to 1 only in case we have the answers from all respondents, and they all choose ‘strongly agree’ answer.

These calculations can be done for every alternative-periods of time. So, using this procedure we can rank all the time periods from best to worst and try to identify the sources allowing to reach the best results.

3 Experimental Research of the Method

The data for our experimental research was given by one rehabilitation hospital in Poland. The administration of this hospital from 2008 to 2017 have distributed to patients the survey about the quality of treatment, personnel helpfulness, infrastructure and admission conditions. The main purpose of such questioning was the intention to find the positive and negative factors of rehabilitation process, assess the influence of specific actions and find the quality improvement patterns over the time. The survey answers were collected as the numeric rating scale data, so direct comparison the quality and efficiency of rehabilitation in specific time frame was not easy task. The following research will show possible solution of this problem.

We’ll try to get not only the estimation of the rehabilitation hospital service quality changes over time frame from 2008 to 2017. The survey consists of 20 Likert-scale questions (Table 2) with different number of rating-scale levels –from 2 to 5.

Table 2. The questions of survey

The answers of all possible respondents to specific question were collected and presented as a frequency table. An example of such frequency table (answers to question no. 11) is presented in Table 3.

Table 3. A part of frequency table (answers to question no. 11)

As we see from Table 3, the data was collected every quarter of the year and distributed according the patient’s department: A1, REH, AR, GOŚ, DZIENNY. This let us make the quality investigation not only by time periods but also evaluate the efficiency of rehabilitation by departments.

According the research framework described in previous section we need to select the set of factors mostly influencing the rehabilitation quality and distribute the questions to groups related to these factors. For this task we have involved seven hospital specialists. They have selected 4 factors: Process of admission to hospital; Personnel helpfulness; Quality of treatment; Infrastructure and food. The questions related to these factors we can see in Table 4.

Table 4. Questions related to specific factor

The next step is to evaluate the factors weights. We have asked the hospital specialists to assess the consistent factors weights using the pairwise comparison of factors importance. The results of factors evaluation by 7 experts are shown in Table 5.

Table 5. Factors weights along the group of expert decision

From the last column of the table, we see calculated consistency ratio (CR) of weight distribution. From practice of AHP application, we know that CR couldn’t exceed 10%. Our experts’ decision follows this rule.

Now we can choose the alternatives and calculate its priority vectors for every factor. As alternatives we selected the annual time period from 2008 to 2017 and estimate its value for specific questions. Using formulas 1–2 for calculation alternatives priority vector and taking into account determined factor weights we got following result about the rehabilitation hospital service quality changes over 2008–2017 (Fig. 2):

Fig. 2.

Annual rehabilitation quality changes

General trend (Fig. 2) shows some points of rapid change and we have asked the hospital authorities to justify trend variation. The sharp increase of satisfaction between years 2008–2009 correlates with the significant rise of the level of payment for healthcare services by Polish national payer. The 2012 drop was related to the personnel crisis. In the region a new rehabilitation centre was established and some staff was proposed there a leading position and leave the hospital. The rise in 2016 and 2017 may be due to the government law about incremental long-term pay rise for all healthcare employees.

On the next figure (Fig. 3) we presented the annual service quality changes over all departments.

Fig. 3.

Annual rehabilitation quality changes over apartments

According the hospital authorities the 2008 was the year where the “Gos” department was under construction and temporarily was located in less comfortable facility. This also explains the 2009 upswing when the department return to the renewed facility. The 2013 peak may collocate with giving the “Gos” department additional new rooms. The somehow continuous decline of satisfaction in “AR” department is due to increasing number of patients without sufficient employment growth and facility expansion. Both “Dzienny” and “REH” departments are operating in stable environment, thus the fluctuation of patients’ satisfaction is not high.

The last figure (Fig. 4) shows the annual pattern of selected factors influencing the patients’ satisfaction in rehabilitation process.

Fig. 4.

Annual rehabilitation quality changes along the selected factors

Going further into details concerning particular fields of quality it is worthy to notice that the improvement of healthcare financing and employment of more staff in 2009 is significantly reflected in three dimensions where the personnel is the critical factor, i.e. personnel helpfulness, quality of treatment and process of admission to hospital. Similarly echoed is the staff crisis in 2012. Persistent progress of satisfaction of the process of admission to hospital from 2012 may be related to greater involvement of ICT solutions allowing better interaction between hospital and patients’ parents. The results about the patients’ satisfaction in rehabilitation process highly correlates with the sources of changes disclosed by hospital authorities. So, the method of analysis the periodic rating scale data using AHP give us meaningful results and can be used in practice.

4 Conclusions and Main Results

The investigation of applying AHP for tracking temporal change of compound quality indicator enabled to draw the following conclusions:

• 1. Widespread application of Likert scale for customer surveying provides data which could be used for enhanced interpretation of its results in time range. However existing methods for processing results, measured by Likert scale do not enable continuous comparative evaluation among different surveying events.

• 2. The article proposes original approach for processing Likert scale–based survey data for estimating compound characteristics of time periods and further performing time series analysis by applying concept of evaluation of alternatives by AHP framework.

• 3. The proposed approach is employed for design of compound indicator for quality evaluation in the application domain of person centered healthcare.

• 4. Experimental evaluation was performed by processing longitudinal survey data, collected at one rehabilitation Hospital in Poland. Application of quality change by proposed AHP-based approach is correlated to implementation of strategic management decisions and programs.