Keywords

1 Introduction

The roots of process quality may be traced back to the 1900s from the Industrial Engineering discipline when Henry Ford managed to build cars at a significantly reduced price by changing his manufacturing process [1]. The focus on process quality led to a major shift in quality control where processes were measured with statistical techniques. The quality control movement led to the development of Total Quality Management (TQM) principles in the 1970s, Six Sigma in the 1980s and Lean techniques are being used more recently [2]. The quality control movement entered the software engineering discipline with the development of the Capability Maturity Model (CMM) in the 1990s where process quality was measured in maturity levels of process capability [3]. The successor of CMM, the CMM Integration (CMMI) was progressively made abstract to cover development, management and acquisition aspects beyond software and into the areas of product, service and overall business processes. However, CMMI maturity levels that determine process capability are the only key representation of process quality characteristics in process assessments.

The initial standard for process assessment ISO/IEC 15504, also termed Software Process Improvement and Capability Determination (SPICE) was also based on, inter alia, CMM. The ISO/IEC 15504 standard series were initially focused on software development processes but it had been expanded in other business areas including management, engineering and service operations. The reference models based on ISO/IEC 15504 defined the capability aspect as the only process quality characteristic. While the scope of processes has expanded in terms of its types (i.e. development, management, governance, and so forth) and its application (i.e. software, IT service management, automotive, space, medical devices, and so forth), the quality characteristic of processes is limited to process capability. There is a need for a common vocabulary and conceptual framework to recognize and categorize other process quality characteristics for assessment.

It is understandable why process capability is the widely adopted measurable aspect of process quality. According to ISO/IEC 33001, process quality is defined by the “ability of a process to satisfy stated and implied stakeholder needs when used in a specific context”. When a process is described, the stakeholder needs of a process are often listed as outcomes that the process needs to achieve to meet its purpose. It then implies that process quality is the ability of the process to meet its purpose – which is defined as process capability. Based on this rationale, one might incorrectly conclude that process quality is process capability. However, if one reviews the definition of process quality, there are two caveats:

  1. (a)

    How can one be certain that “all” stakeholder needs of a process are listed as outcomes to achieve?

  2. (b)

    How can the “implied” stakeholder needs, and the “context of use” considered for assessment?

Process capability determines the ability of a process to meet business goals [4]. Since meeting process goals is the major quality check for a process, there is no doubt that process capability is the major process quality characteristic. However, the two questions raised above introduce the need for other quality aspects of a process during assessments. Currently the scope of process capability is limited, so it does not determine the overall process quality. The ISO/IEC 33000 standards series released in 2015 recognized this challenge and used the generic term “process quality characteristics” to develop generic process measurement frameworks for assessment [5].

Specific examples of process quality characteristics beyond process capability are provided, such as process security, process agility and process safety [5]. However, there is only a single process measurement framework for assessment of process capability published as ISO/IEC 33020 [4], paving a way for other process measurement frameworks to be built. Recent studies on the adoption of ISO/IEC 33000 assessment framework still relate to process capability as the sole process quality characteristic, e.g. [6]. Other process quality characteristics have been proposed, e.g. for safety [7] and sustainability [8], however a holistic list of constructs (theoretical concepts) for process quality characteristics have not been proposed for assessment.

In this paper, a comprehensive view of process quality is undertaken, by focusing on the intrinsic and extrinsic quality attributes associated with a process. This focus is used to propose a taxonomy of process quality characteristics. The exemplar studies where the proposed process quality characteristics have been used for the determination of process quality are also included.

The purpose of this taxonomy is twofold: (1) to provide a framework for representing and combining process quality characteristics; and (2) to ultimately enable process assessment using multiple process quality characteristics. Both purposes are critical, given the importance of process assessments to understand quality attributes of a process internally (intrinsic factors) as well as quality surrounding the process environment that are influenced by extrinsic factors.

2 A Proposal of Characteristics as Intrinsic and Extrinsic Quality Attributes

Table 1 identifies the proposed aspects of the taxonomy and defines, for each of these aspects, whether it is something that the process can control (intrinsic quality attribute) or the process cannot control (extrinsic quality attribute) or both. It is important to recognize that the taxonomy should not be considered exhaustive or a final list. In this first instance, all possible aspects of process quality characteristics have not been considered and the taxonomy itself is subject to continuous revisions. The elements that classify process quality continue to evolve due to context-dependent scenarios and implications surrounding process execution, management and environment during assessment.

Table 1. Aspects for process quality characteristics
Full size table

In determining the aspects that characterize process quality for assessments, two simple heuristics were followed. The first was to review the extant literature to determine the aspects and its application in process assessments across different disciplines, mainly software engineering and business process management. For example, the process attributes in system and software quality models from ISO/IEC 25010 [9] and BPM principles [10] were considered to determine initial aspects for process quality characteristics.

The second heuristic was to put the aspects in a simple sentence of the form: “<aspect> is what the process can or cannot control”. If an aspect can be controlled by the process, i.e. it is mainly related to process activities and outcomes, it is classified as an intrinsic quality attribute. By intrinsic quality, it refers to “quality something has in itself, apart from its relations to other things” [11]. For example, because one can say that “process must meet its purpose by fulfilling its outcomes”, the “effectiveness” is an intrinsic quality attribute. In a similar way the aspects of “culture” and “security” are classified as extrinsic quality attributes, i.e. these are quality aspects outside of process control but still belong to the environment where the process is executed or managed.

Note that being an intrinsic quality attribute and being an extrinsic quality attribute are not mutually exclusive. For example, the aspect “reliability” described in Sect. 3.3.1, is employed as an intrinsic quality attribute because it is something a process can improve by making changes within its activities, but also an extrinsic quality attribute since there are other environmental and contextual factors to consider reliability of a process (e.g. availability of technology to support process execution). All aspects are discussed in detail in Sect. 3.

3 A Taxonomy for Process Quality Characteristics

Beyond a process’s core focus on its activities, outcomes and resources, it is apparent that the process is affected by its relationship with its stakeholders and other processes; operating environment; and management environment. This paper will discuss each of the aspects in Table 1 under the following four logical themes: core attributes (activities and resources of a process); relationship attributes (association of a process with stakeholders, other processes and reference models); operating environment (operational context for a process); and management environment (management context for a process).

Figure 1 illustrates these themes and the aspects that each contains. However, it should be noted that this represents only one of a number of ways that process quality characteristics can be categorized. The themes and their aspects are discussed in detail in the following subsections.

Fig. 1.
figure 1

Themes & aspects of process quality characteristics

Full size image

3.1 Core Attributes

The first logical theme in the taxonomy addresses the core attributes of process quality. Aspects discussed in this grouping describe the intrinsic quality features of a process – its ability to meet the stated goals and the usage of resources.

3.1.1 Effectiveness

Process effectiveness, also referred to as efficacy, defines the quality feature of a process to meet its purpose by fulfilling all stated outcomes. The major constituent of a process is a series of activities; therefore, it is important to measure that the activities are performed as intended. When one considers process assessment, they are primarily interested to find out the effectiveness of a process. Consequently, it is the most widely accepted process quality characteristic. This aspect is primarily defined as the metric of “process capability” and it has been used since early days of maturity models for processes. A formalization of Process Effectiveness was included in the Maturity Model proposed by Humphrey [12]. The process measurement framework for assessment of process capability published in ISO/IEC 33020 [4] also provides the metric for process effectiveness.

3.1.2 Efficiency

The second most important process quality characteristic deals with resource utilization, primarily in terms of time and cost involved. Efficiency determines that the process makes optimal use of the resources available to it while performing its activities effectively. Effectiveness and efficiency are often contradictory since highly effective processes typically require costly resources. Nevertheless, a balance between these two quality attributes is needed so that the process productivity is promoted, while deadlines are achieved and costs are reduced [3]. A typical example of process efficiency is the metric of “process cycle time” to measure the duration of a process. Since optimal resource utilization is a core objective of a process, this quality attribute may be listed as a key outcome for a process and measured in terms of overall process capability.

3.2 Relationship Attributes

The second logical theme in the taxonomy addresses the relationship of a process with its stakeholders. Aspects included in this theme describe a process’s relationship with its customers, process team members in the role of managers or performers, and with other processes and process reference models. Since these aspects focus on the relationship of a “process” with other stakeholders, both intrinsic and extrinsic quality attributes can be relevant for process assessment.

3.2.1 Satisfaction

Every process has at least one customer – internal or external. Process satisfaction defines the relationship of a process with its customers. Customer satisfaction may not be defined at the process level, however once the relationship of a process with its immediate customer(s) is determined, the usefulness, value, trust and service level of a process can be ascertained based on the customer satisfaction indicator.

Consumer satisfaction can be a process metric describing customer emotions resulting from process assessments (including perceived performance of a process) based on their experiences dealing with the process as a stakeholder external to the process [13]. The value perceived by a customer is usually determined by the utility and warranty of the underpinning service [14]. The utility and warranty parameters of a process are typical candidates of process outcomes, therefore achieving the outcomes of a process, i.e. process effectiveness may cover this aspect. However, process satisfaction considers value and usefulness from the eyes of the customer. One useful metric for this aspect is “service level”, which enables customers to report their degree of satisfaction (or dissatisfaction) within the agreed service levels (also referred to as service level agreements or SLA) [15].

3.2.2 Usability

While satisfaction represents quality characteristics in terms of a process’s relationship with its customers, the quality characteristic of usability portrays its relationship with the process team members – typically in the roles of process owner, process manager and process performer. Usability is about user experience same as satisfaction for customer experience.

Since activities and involvement of process team members vary widely, process usability is challenging to monitor [16]. Process usability can be determined from the assessment of the appropriateness of the process in terms of its ease of use, accessibility and operability. Accessibility and usability are closely related, as they both enhance user experience. Operability can be measured in terms of users’ perceived difficulty of performing process activities. A useful metric for process usability could be related to its learnability measured in terms of the metric of “learning time” needed by users to understand and train to use the process, as being undertaken in a study for user requirements elicitation [17].

3.2.3 Compatibility/Variability

A process rarely executes in isolation. For any process, it may depend on other processes or there could be other processes that depend on it (inter-dependencies). It is also possible for a process to co-exist with other processes in parallel. Therefore, the process in use must be “compatible” with its reference models that explains the relationship with other processes. The aspect of compatibility (or variability as the opposite measure) of a process refers to its relationship with other processes and process reference models.

Compatibility with process models also determines the quality attribute of maintainability and testability of a process. By compatibility, a process must be a good fit with a process model so that the model can be used for assessments, estimations and testing to determine the quality state of the process itself [18]. Another quality attribute that is useful to check is interoperability – typically highly capable processes contributes towards better process interoperability across enterprises [19]. A practical metric for process compatibility could be evaluating “process tailoring guidelines”. While process variations may be necessary, such variations are typically managed using tailoring guidelines [20]. Therefore, a review of tailoring guidelines can help to determine process compatibility (or lack thereof – i.e. process variability).

3.3 Operating Environment

This logical theme concerns the operating environment where the process executes. It should be noted the aspects in this grouping can relate to both intrinsic and extrinsic quality attributes since these process quality characteristics can be improved by actions within the process parameters and also other operational factors beyond the process.

3.3.1 Reliability

Under realistic operating environment, a process cannot be expected to be perfectly capable, i.e. there cannot exist a process with 100% process capability, that means a process is directly affected by its reliability [21]. A reliable process is typically characterized by its availability. A highly capable process that is not available when it is needed is of no use. Therefore, process reliability is a very important aspect of process quality that depends on the operating environment of the process.

Process assessments determine process quality at a specific point of time. Therefore, measuring reliability is challenging during process assessments because process reliability is a dynamic aspect that requires active monitoring [22]. Therefore, real-time process reliability assessment may not be possible unless the process is fully automated and support real-time decision support, for example, online sales process. For other processes, historical process performance data can be used to ascertain reliability. For example, it may be possible to undertake a historical trend analysis from active monitoring systems to assess reliability. A review of other proactive measures that ensure high availability of a process can also demonstrate a strong process reliability. A process that can regularly fulfil its intended outcomes is one that is considered “reliable”. Therefore, a useful intrinsic quality metric to test process reliability is check historical data of its “failure rate”, i.e. how much a process has failed per unit time in the current operating environment. An example of extrinsic quality metric for process reliability is identifying the “knowledge level” of operating environment for a process, i.e. the number of inventive problem-solving knowledge for executing the process. Using knowledge-based methodology to develop new systems and solutions to resolve process problems during its operations has been proposed to improve process reliability, for example, using the principles of Theory of Inventive Problem Solution (TRIZ) to check existing knowledge if the problem has been solved already [23].

3.3.2 Flexibility/Agility

Processes must be able to accommodate changes in the environment in which they operate. To determine this attribute of a process, two closely related aspects of flexibility and agility are useful. Flexibility relates to adaptability of a process to respond to changes; while agility focuses on the speed of response to the changes (how quickly can a process change) in the process operating environment. Process flexibility and agility are determined by intrinsic quality attributes as well as context-dependent operating environment of the process.

Internally, process flexibility can be determined by evaluating “process tailoring guidelines” against the capability of a process to meet its outcomes; i.e. how capable is a process given the number of adaptations. Likewise, process agility can be measured using tailoring guidelines against the time efficiency of a process; i.e. how quickly can a process change given the number of adaptations. A number of quantitative and qualitative metrics to determine process flexibility and agility use the aspects of process effectiveness and efficiency along with the measure of tailoring guidelines or actual process changes [24].

Four process flexibility configurations as extrinsic quality metrics to improve the process operating environment have been proposed in the area of business process management [25]: flexibility by design (handling anticipated changes with defined supporting strategies); flexibility by deviation (handling simple occasional unanticipated changes); flexibility by under-specification (handling anticipated changes where supporting strategies are not defined); and flexibility by change (handling complex but occasional or permanent unanticipated changes). When the process flexibility metrics are compared against the speed of response to changes, they provide useful metrics for operating environment that support process agility.

3.4 Management Environment

During process assessments, the overall management environment under which the process operates plays a critical role in determining process quality and performance. In ISO/IEC 33020 [4], the proposed process measurement framework for process capability recognizes the importance of management environment for quality levels beyond level 1, i.e. regarding process management, standardization, control and innovation. The progression from capability levels 2 to 5 demonstrate maturity of the management systems under which individual processes or process areas operate. Since key aspects of management activities affecting process quality are covered by the process measurement framework for process capability, the focus of this theme is on the management areas where processes operate.

This logical theme describes three key management areas as aspects for process quality characteristics. This list is not exhaustive as a large number of management areas can be relevant for different processes based on their context of use and implied objectives. These are extrinsic quality attributes as processes have little to no influence towards these aspects. However, a process is significantly dependent and affected by these management environment aspects.

3.4.1 Sustainability

Sustainable growth and environmental impact of human activities are significant areas of research in all areas. The evolving green ICT initiatives are an indication of the recognition of process sustainability as a quality metric.

Research by Lami et al. [26] have presented sustainability aspect in software processes by evaluating the culture of green IT in software organisations. This research discussed process sustainability and initially related the concept of sustainability with process capability so that sustainability can be measured as part of process capability. This is only feasible when sustainability goals are explicitly included in the expected outcomes of a process. Given the broader implication and extrinsic nature of sustainability beyond processes, the researchers proposed a new measurement framework for process sustainability assessment [8] that comply with ISO/IEC 33000 series. A practical metric for process sustainability is “carbon emissions and energy costs”. While it appears sustainability is only significant for manufacturing industries, one must realise ICT carbon footprint in terms of energy use by data centers and by ICT consumers. Therefore, recognizing sustainability as a process quality characteristic will encourage promoting a sustainable culture and activities at a process level.

3.4.2 Security

Process security as a quality characteristic for assessment can be undertaken from various perspectives: information security relating to confidentiality, integrity, authenticity and non-repudiation of data associated with process work; and safety and risks associated with process environment. This aspect is related to extrinsic quality attribute since most of control activities fall beyond the boundaries of a typical process.

While security is important and has its own set of processes for maintaining information safekeeping, what is important is the content of security measures undertaken during process work [27]. Since processes are information-intensive and increasingly prone to automation in the digital era, evaluating information security at a process level is critical so as to determine data access requirements and data integrity. A useful metric to evaluate security environment for a process is “number of information security breaches” in relevant process environments.

Beyond information security, security can be viewed from the perspective of process risks. An integrated risk management process model has been proposed to operate within IT settings based on the foundations of ISO standards on risk management and process assessment [28]. While the researchers provided a useful process model and harmonized with a focus on process assessment, there is an opportunity to extend this research so that process risk determination perspectives can be a foundation towards process security as a process quality characteristic. In a similar vein, process safety has been proposed as a potential process quality characteristic [7]. In this research, safety integrity levels have been proposed to determine process dependability that is measured in terms of reliability, maintainability and availability – some of these aspects are already covered earlier in this paper.

3.4.3 Culture

Process culture is an extrinsic quality attribute that is proposed as a single aspect in this paper but it is determined by multiple organizational factors. Some key factors that may facilitate process culture are: leadership buy-in, governance of process actions, continuous improvement, communication support, knowledge management, documentation, IT architecture and innovation. Process culture elements are adopted from the management environment at an organisational level.

There is a large body of research on process culture in the discipline of business process management as culture is considered a key element in BPM practice [29]. Cultural assessment in terms of process quality has been undertaken at an organisational level in areas of customer service, organisational structure, continuous improvement, commitment, innovation and accountability [30]. Current BPM researchers and practitioners treat culture as a manageable enabler of process initiatives rather than a barrier. In software engineering discipline, use of technology to improve process culture in software development teams have been researched [31]. In this light, process culture can be used as a process quality characteristic to monitor culture environment conducive for process activities. A relevant process metric for process culture can be “number of improvement actions” for a process.

4 Discussion

Section 3 presented a discussion on the potential process quality characteristics for process assessments. Table 2 outlines the proposed taxonomy of process quality characteristics (represented as “process aspects”) with example process quality metrics and exemplar research references on the relevant process quality.

Table 2. Taxonomy of process quality characteristics
Full size table

Figure 2 illustrates the ten process quality characteristics represented as “process aspects” based on the four themes to provide a framework for process assessment.

Fig. 2.
figure 2

A framework for assessment areas based on process quality characteristics

Full size image

5 Conclusion

In this paper a taxonomy of process quality attributes is proposed based on ten aspects mapped to intrinsic and extrinsic quality attributes. The discussion of these aspects subdivides them into four logical themes: core attributes, relationship attributes, operating environment and management environment. While assessment areas have expanded in different areas, the process quality metric is limited to process capability, even with the recent movements towards automation to determine process quality during assessments, for e.g. [32, 33]. The proposed taxonomy can be used to evaluate processes with a wider view based on different contexts and implications during process assessments. There is currently no discussion of theoretical underpinnings and limited justification for the proposed process quality characteristics. In the future the taxonomy can be used as a platform to justify broader aspects of process quality measurement. Consequently, this research serves as a foundation to develop process measurement frameworks and ultimately to evaluate different process quality aspects during process assessments.