1 Introduction

1.1 Motivation

Usability and accessibility are currently considered as quality references in software development [1, 2]. In general, most companies take into account quality attributes related to usability and accessibility to develop software products. However, quality concerning user-centered processes is less common [3], and we have detected a lack of proposals focused on improving usability and accessibility processes, which would reduce errors, revisions and implementation time, allowing at the same time the systematic construction of usable and accessible software products.

Development process assessments help to corroborate the desired level of usability and accessibility in the systematic construction of software products, increasing the level of satisfaction and psychological wellness when the user interacts with the software produced by a given process.

A common method to measure the quality of a software process is based on the utilization of capability maturity models, which provide a framework for continuous process improvement based on the maturity level of the organization’s software development processes. Each level has a number of process areas that must be achieved. The achievement of these areas or stages is detected through the satisfaction or dissatisfaction of several clear and quantifiable goals. With the exception of the first level, each of the remaining maturity levels is composed of a certain number of Key Process Areas (KPA) identifying a set of interrelated activities and practices that, when carried out collectively, allow reaching the fundamental goals of the process.

However, although there are different capability maturity models, most existing approaches are focused on the development point of view rather than on a user-centered vision. There are few capability maturity models centered on user and usability, and they lack a joint conception of usability and accessibility practices, which heavily affects the quality of the products resulting from such processes as accessibility and usability together represent essential characteristics to ensure quality. In addition, existing user-centered capability maturity models lack a set of prescribed activities, work products (used and generated by such activities), process attributes and management practices to guide, measure and improve the capability of the processes.

1.2 Research hypotheses

According to the aforementioned motivation, our paper is based on the following conceptual hypotheses used to conduct our research:

H1

We detect a lack of previous works focusing on the evaluation of the quality of processes featuring usability and accessibility together. Existing capability maturity models lack a user-centered vision, which heavily influences the quality of the products, as accessibility and usability represent essential characteristics to ensure quality. In addition, existing maturity models lack prescribed and detailed activities, work products, process attributes and management practices to guide in the achievement of quality through the capability maturity evaluation of the development process.

H2

It is possible to propose a model for the evaluation of capability maturity in usability and accessibility centered processes, which may help organizations ensure and manage a user-centric approach throughout the development life cycle, integrating usability and accessibility characteristics to improve quality during the software development overall.

H3

The solution proposed can deal with process evaluation, being useful for audits and process improvement overall.

1.3 Methods

Our research has been inspired by the scientific method. This way, and starting with the motivation and problem description, we have proposed the initial aforementioned hypotheses to conduct our research. Such hypotheses will be corroborated throughout the paper as follows:

  1. 1.

    In order to corroborate H1, we have carried out a Systematic Mapping Study [4]. Results obtained demonstrated that there is lack of existing work based on measuring usability and accessibility in development processes.

  2. 2.

    To corroborate H2, we have developed MODECUA, the Spanish acronym for MOdelo para la DEterminación de la Capacidad de mejora de procesos centrados en la Usabilidad y la Accesibilidad. MODECUA comprises a model to determine the capability maturity of processes specifically focused on usability and accessibility. The main objective of MODECUA is to help companies carry out a guided and systematized user-centric approach for the development of usable and accessible software products, allowing to determine and improve the quality of their processes. MODECUA has been created based on the improvement and integration of different international quality standards. On the one hand, it is based on the seven Human-Centered Design (HCD) processes [5] established in ISO/TR 18529 [6], and the standards ISO 9241-210 [7] and ISO 9241-20 [8] for usability and accessibility, respectively. On the other hand, a contributed capability scale to determine the process’s maturity based on the standard ISO/IEC 15504 [9], and denominated U + A SPICE, is provided. This way, the main benefits from the application of our model are the following:

    • Determine the process’s capability maturity focused on usability and accessibility, working out an optimal quality level in user-centered development processes;

    • Ensure that software products are usable and accessible to end users, thus increasing user satisfaction;

    • Guarantee and promote a user-centric approach in the organization, facilitating audits on user-centered issues;

    • Minimize costs, time and effort in software development processes.

  3. 3.

    Finally, in order to corroborate H3, a case study, based on a real company, demonstrate the application of MEDECUA. Results obtained helped to corroborate that our approach is suitable for the intended purpose.

This paper is structured as follows. Section 2 includes related work. Section 3 presents MODECUA in detail. Section 4 presents the case study. Finally, Sect. 5 reports on conclusions and future work.

2 Related work

2.1 Existing capability maturity approaches

There exist different proposals intended for measuring the capability maturity in development processes. However, most of them are little or not directly related to usability and accessibility, or they consider both quality characteristics separately.

CMM [10] is a useful tool to help organizations improve their processes. It defines a set of priorities to solve and improve software development problems, providing a conceptual framework to improve the development in a systematic way. However, it has several drawbacks, as the implementation process involves a long time and cost, and the adoption of this model requires a great implication of all stakeholders. On the other hand, it is not specifically focused on usability and accessibility. Similarly, CMMI [11], a later version of CMM, integrates different processes in the organization, also reducing the number of defects in the software by detecting them in the early stages of the lifecycle. In addition, it supports agile processes and reduces maintenance and support times, offering, in contrast to CMM, a systematic guide for the improvement. However, it is not specifically focused on usability and accessibility either.

A more human-related approach can be found in People CMM [11], which is easier to integrate in organizations having a previous adopted maturity process. It provides a guide to solve problems of certain work practices, and it helps to develop staff skills and increase the retention of talent in the organization. In general, it promotes an organizational culture, improving internal and external communication. As for drawbacks, People CMM requires the organization to previously have a certain level of maturity and, as well as similar approaches, it is not focused on usability and accessibility.

ISO/IEC 15504 (known as SPICE) [9] is another popular approach that allows organizations to increase the maturity of their software development processes through continuous improvement. This model is widely spread and adopted by software development companies with formalized processes, being widely compatible and easy to integrate with other process models. ISO/IEC 15504 is the largest model that has been tested and agreed, presenting two independent dimensions for processes and capability. Nevertheless, it is not specifically focused on usability and accessibility. This standard has been revised by the new ISO/IEC 33000 [12].

2.2 Systematic mapping study

In order to look for more related approaches, we have carried out a Systematic Mapping Study to find research works related to the measurement of usability and accessibility processes.

A Systematic Mapping Study comprises a structured mechanism for searching bibliographical resources and counting contributions that match a set of topics. The methodology used was the following:

  1. 1.

    First, a comprehensive bibliographical search was carried out, using different online libraries that provide research contributions according to a specific set of (search) topics;

  2. 2.

    Second, a screening phase was carried out, removing duplicated papers;

  3. 3.

    Third, a refinement phase was accomplished. This way, inclusion and exclusion criteria are applied in order to obtain a more reduced set of papers, closer to the topics pursued;

  4. 4.

    The reduced set of papers was analyzed in detail, in order to investigate the contributions according to the topics researched and create a state of the art.

This way, we have utilized the following bibliographical resources:

  1. 1.

    ACM Digital Library;

  2. 2.

    IEEE Xplore;

  3. 3.

    SCOPUS;

  4. 4.

    Google Scholar.

Also, we used a search string including the following topics:

(“usability” or “usable “ or “user-centred” or “user-centered”) and

(“accessibility” or “accessible”) and

(“maturity” or “capability”)

However, we did not obtain results for the query proposed, i.e., there were not significant results of research works including usability and accessibility maturity proposals. This way, to analyze the problem, we split up the query in order to investigate the terms in isolations, which helped to obtain papers related to maturity and capability, some of them related to usability or accessibility apart. These results are summarized in Table 1.

Table 1 Results obtained by splitting up the initial search string and considering the aforementioned bibliographical resources
Full size table

In order to further refine the results and study similar approaches apart based on usability and accessibility capability maturity, we proceeded to the screening phase, removing duplicated papers and applying the following inclusion criteria:

  1. 1.

    Works selected must be based on a capability or maturity proposal for usability measurement, preferably in form of a descriptive or prescriptive approach for measuring usability processes;

  2. 2.

    Works selected must be based on a capability or maturity proposal for accessibility measurement, preferably in form of a descriptive or prescriptive approach for measuring accessibility processes.

Once applying 1 or 2, we found a few works related to usability or accessibility maturity, lacking the capability feature. This is the case for Corporate Usability Maturity [13], specifically focused on usability. It provides clear indicators at each stage, which facilitates the identification of maturity levels. However, this model presents some specific drawbacks. To cite a few, it does not describe in detail the related activities to reach maturity and it requires a great deal of time, effort and resources for the organization to get the highest level of maturity. Besides, it lacks formality and detailed description to be widely adopted, and it does not consider accessibility. Similarly, the Usability Maturity Model [5] is another approach focused on usability. It is suitable for incremental-iterative software development processes, allowing to integrate usability practices in the model. It is focused on quality and continuous improvement of the processes, gradually describing the integration and usage of the model by specifying processes, activities, input and output products. This model is based on the standards ISO 13407 [14] and ISO/IEC 15504, providing also assessment tools. As for drawbacks, it is not specifically focused on accessibility, also lacking a quantitative assessment and the detail of the usability techniques to apply. In addition, both the structure and the execution of the processes are not sufficiently detailed.

Other recent approaches provide recommendations for improvement, though presenting low detail to be considered as integral models. This is the case for the Corporate UX Maturity Model [15], which is focused on usability. This model is easy and simple to understand, clearly specifying the stages to be followed by the organization in the adoption of the user experience. In addition, it describes key indicators that help organizations achieve the necessary level of maturity. However, this approach is not very formalized and widespread, and does not specify the detail of the activities to reach the pursued maturity and the practices to be included in each stage, lacking a mechanism for the systematic measurement.

On the other hand, Keikendo Maturity Model [16] is another approach also focused on user experience. It provides recommendations and suggestions for improvement, and advice on how to reach the desired maturity level. It is also easy to understand, providing an online test to find out the level of maturity in the organization. However, as with previous approaches, the Keikendo Maturity Model is not very formalized and widespread, and does not specify the techniques and practices needed, the method to quantify the user experience practices and the metrics for the evaluation.

As shown so far, the few existing approaches are principally based on usability. However, accessibility is an important concern today that should be addressed [17, 18]. In this sense, the Accessibility Maturity Model [19] is an approach principally focused on accessibility. It features an online interactive tool to determine the maturity level of the organization. However, this approach is not very formalized and widespread, and does not provide enough detail of the activities to achieve for reaching the desired level of maturity. In addition, it does not describe specific guidelines, metrics or evaluation techniques, which is an important drawback to reach a user-centered conception [20].

Results obtained from the study of the related work helped to corroborate H1. In a nutshell, there is a lack of formal models to deal with the capability maturity assessment of process related to both usability and accessibility, and most related approaches are not very formalized or limited in terms of the specific practices and techniques needed. Besides, most of existing user-centered maturity approaches lack systematic methods and do not provide sufficient support to be practically applied [3].

3 MODECUA

MODECUA is inspired by the necessity of having a formalized framework to measure the capability maturity in processes intended to develop usable and accessible software. This way, we are based on well-known standards in order to keep up the way capability maturity is assessed according to the normative standard.

Specifically, MODECUA is based on the seven HCD processes appearing in ISO/TR 18529. This model has been selected due the following strengths that it provides:

  1. 1.

    It is focused on the integration of usability practices in software development processes;

  2. 2.

    It is focused on an incremental-iterative software development vision, allowing to include usability practices during all stages of the process, since usability must be addressed iteratively and incrementally to ensure that the final system ensures an optimal level of usability;

  3. 3.

    It is focused on quality and continuous improvement of software development processes;

  4. 4.

    It is very descriptive, providing a step-by-step guide for the integration and utilization of the model;

  5. 5.

    It describes in detail the activities that must be followed to reach the desired maturity of the processes;

  6. 6.

    It is the most formalized model of maturity focused on usability, since it is based on the most widespread and adopted regulations, such as ISO/IEC 15504 and ISO/IEC 13407 (now ISO 9241-210);

  7. 7.

    It provides an evaluation tool, offering examples of compliance with the human-centered process.

However, ISO/TR 18529 presents some drawbacks such as the ones listed down below that will be improved in our approach:

  1. 1.

    Although being focused on usability, it does not include accessibility issues;

  2. 2.

    It does not provide a quantitative evaluation of the proposed practices;

  3. 3.

    The definition of the model structure is not clear enough for a practical application;

  4. 4.

    It is focused on evaluating the compliance of the practices, overlooking the evaluation of the process quality and the capability vision;

  5. 5.

    Although it is focused on usability practices, it does not detail the techniques and methods to be used;

  6. 6.

    It does not evaluate specific attributes of usability.

In addition, we have considered ISO 9241-210 and ISO 9241-20 standards in order to provide usability and accessibility activities and work products (used and generated by such activities) in order to complement the information of the aforementioned HCD processes. The capability maturity evaluation is carried through a customized version of ISO/IEC 15504 that we have called U + A SPICE (Usability and Accessibility SPICE), used to measure the maturity of the capability levels of a user-centered process featuring usability and accessibility together.

Figure 1 shows a general overview of MODECUA. The seven processes appearing in the original ISO/TR 18529 have been improved with 28 new contributions based on ISO 9241-210 and ISO 9241-20, which corresponds to eight new activities, 11 adapted activities and processes, and nine renamed activities. In addition, we have integrated 127 new work products used and generated by these new usability and accessibility activities.

Fig. 1
figure 1

General overview of MODECUA along with the standards and contributions made

Full size image

Besides, the proposed U + A SPICE scale comprises six capability levels including attributes based on usability and accessibility practices specified in the model, as well as the integration of ten management practices for achieving process attributes. These elements can be measured to find out the maturity level. It is worth mentioning that the proposed scale and measurement mechanisms can be integrated into the procedure described in ISO/IEC 15504. This provides standardization in the assessment to be easily adapted and used by organizations.

Figure 2 shows the correspondence between the aforementioned components in MODECUA, that is, the improved HCD processes, including new usability and accessibility activities and products, and U + A SPICE consisting of six capability levels including the corresponding process attributes, practices and practice requirements to measure the capability maturity.

Fig. 2
figure 2

Correspondences between HCD processes and the capability maturity in MODECUA

Full size image

The utilization of MODECUA is twofold. On the one hand, the seven HCD improved processes can be used as a software development method to assure usability and accessibility by following the proposed activities and generating the contributed work products. On the other hand, a concrete development process can be evaluated against such HCD process to assess the capability and maturity (even individually) and thus obtain evidence of the quality degree reached, with the idea of carrying out further process improvements.

3.1 HCD processes

As explained, we have included new activities in ISO/TR 18529 corresponding not only to usability concerns, but also to accessibility activities that need to be considered in order to have a complete user-centered vision.

ISO/TR 1829 includes seven main processes identified with HCD (Human-Centered Design) and numbered from 1 to 7. Those contain different sub-process (activities), and can be iteratively executed to create user-centered software products. These processes can be defined as follows:

  • HCD.1 Ensure HCD content in the system strategy. This process is intended to set the focus on aspects related to stakeholders and users, which may involve all the organization’s concerns such as marketing, planning, system conception and enterprise and so on;

  • HCD.2 Plan and manage the HCD process. This process is aimed at the specification of the human-centered activities to be integrated in the whole lifecycle;

  • HCD.3 Specify the stakeholder and organizational requirements. The purpose of this process is to set organizational and stakeholder requirements, taking into account needs, competencies, working environment, etc.

  • HCD.4 Understand and specify the context of use: This process is aimed at understanding users, their tasks and the environmental conditions in which the system will operate;

  • HCD.5 Produce design solutions. This process is intended to create design solutions according to the information previously inferred about users, stakeholders and contextual analysis;

  • HCD.6 Evaluate designs against requirements: The aim of this process is to obtain information from users and stakeholders about the design solutions;

  • HCD.7 Introduce and operate the system: This process is intended to operate the system according to the human aspects considered.

In general, we have considered the same process order but switching HCD.3 by HCD.4, as according to usability and accessibility standards (ISO 9241-210), the contextual analysis should be considered first, before specifying user, stakeholder and organizational requirements. This information is shown in Table 1, where the seven HCD processes (in bold) and activities are presented, together with the type and ISO source to indicate the new additions and modifications proposed, intended to consider usability and accessibility practices extracted from ISO 9241-210 and ISO 9241-20, respectively. As shown in Table 2, the type for each process and activity can be:

  • N, when a new activity is proposed. In this case, the ISO source column corresponds to the reference from where the activity has been extracted;

  • A, when an activity or process has been adapted taking into account the ISO source column. Adaptations are motivated for the necessity of improving and completing some of the activities to consider usability and accessibility practices. On the other hand, processes 3 and 4 have been altered in order to follow ISO 9241-210 process, which first identifies the context and then the requirements;

  • R, when an activity has been renamed in order to better match usability and accessibility specifications;

  • E, when an activity remains unchanged with respect to the ISO/TR 18529 standard.

Table 2 HCD processes and activities including new contributions
Full size table

We have included the following new activities:

  1. 1.

    HCD.1.6 Based on the ISO/IEC 15504-7 about specifying the importance of ensuring effective communication regarding the performance of the processes, through a clear assignment of responsibilities to the parties involved;

  2. 2.

    HCD.4.1 Based on the ISO 9241-210 about identifying the needs of users and stakeholders;

  3. 3.

    HCD.4.2 Based on the ISO 9241-20 about identifying and specifying user accessibility needs;

  4. 4.

    HCD.4.9 Based on the ISO 9231-210 about ensuring the quality of user requirements specification;

  5. 5.

    HCD.6.7 Based on the ISO 9241-210 about modifying design solutions based on user-centered evaluations and feedback;

  6. 6.

    HCD.6.8 Based on the ISO 9241-210 about communicating the design solution to the persons responsible for the implementation;

  7. 7.

    HCD.7.7 Based on the ISO 9241-210 about enabling long-term follow-up;

  8. 8.

    HCD.7.8 Based on the ISO 9241-210 about promoting sustainability and user-centered design.

Moreover, we have adapted the following activities:

  1. 1.

    HCD.2.1: Based on the original HCD.2 but adapted to ISO 9241-210, which specifies that the participation of users and others involved in the development of software offers a valuable source of knowledge, so it must be active in all its stages;

  2. 2.

    HCD.2.3: Based on the original HCD 2.3 but adapted to the activity included in ISO 9241-20 about producing design solutions paying special attention to accessibility considerations;

  3. 3.

    HCD.3: Based on the original HCD 4 but adapted to ISO 9241-210 about understanding and specifying the context of use;

  4. 4.

    HCD.3.2: Based on the original HCD 4.2 but adapted to ISO 9241-20 about understanding and specifying the context of use, paying special attention to the variety of characteristics of users and the impact of tasks, equipment and the characteristics of the environment that affect the accessibility;

  5. 5.

    HCD.4: Based on the original HCD 3 but adapted to ISO 9241-210 about specifying stakeholder and organizational requirements;

  6. 6.

    HCD.4.6: Based on the original HCD 3.4 but adapted to ISO 9241-20 about defining the system usage, paying special attention to the variety of characteristics of users and the impact of tasks, equipment and the characteristics of the environment that affect the accessibility;

  7. 7.

    HCD.4.7: Based on the original HCD 3.5 but adapted to ISO 9231-210 about starting from the user requirements;

  8. 8.

    HCD.5.4: Based on the original HCD 5.4 but adapted to ISO 9241-20 about producing design solutions, paying special attention to accessibility considerations;

  9. 9.

    HCD.5.5: Based on the original HCD.5.5 but adapted to ISO 9241-20 about producing design solutions, paying special attention to accessibility considerations, and also to ISO 9241-210 about designing user tasks, user-system interaction and user interface to meet the user needs, taking into account all the user experience;

  10. 10.

    HCD.5.6: Based on the original HCD 5.6 but adapted to ISO 9241-210 about making more concrete design solutions;

  11. 11.

    HCD.6.3: Based on the original HCD 6.3 but adapted to ISO 9241-20 about evaluating the accessible design solutions with users that represent the target group.

In addition, we have renamed the following activities in order to be better adapted to the standards utilized: HCD.4.3, HCD.4.4, HCD.4.5, HCD.4.8, HCD.5.2, HCD.5.3, HCD.7.1, HCD.7.3 and HCD.7.5.

Together with the activities specified in Table 2, a total of 127 input and output work products have been defined. Such products are evaluated by practices in order to check whether practice requirements are satisfied and thus determine the capability maturity level using U + A SPICE (see Fig. 2). The list of work products is shown in Table 3. Those have been identified as IWP in the case of input work products and as OWP in the case of output work products. The enumeration of each work product is represented as X.Y.Z, where X.Y corresponds to the activity where the work product is requested or generated, and Z corresponds to the enumeration of the corresponding work product. For instance, IWP.1.1.1 is necessary for the execution of activity HCD.1.1, and OWP.1.1.1 is generated through the execution of activity HCD.1.1.

Table 3 Input and output work products for each activity represented in Table 2
Full size table

3.2 U+A SPICE

As previously specified, MODECUA features, along with the proposed processes, a sub-model aimed at measuring the capability maturity level (of the aforementioned processes). More specifically, we have based on ISO/IEC 15504 to create U + A SPICE, a customized version of the standard though featuring new attributes and practices oriented to evaluate usability and accessibility processes.

U + A SPICE comprises six levels of capability inspired by ISO/IEC 15504. Those integrate five new process attributes related to usability and accessibility, as well as ten new practices containing specific requirements that have to be fulfilled to satisfy the goal of the practices.

As specified in Fig. 2, capability levels are achieved through the evaluation of the process attributes, which are rated based on the compliance of the practices, which evaluate work products of diverse nature such as specifications, usability and accessibility guidelines, methods and evaluation techniques, and so on. Practice requirements specify how practices must be accomplished, that is, what elements must meet practices and products to reach the required quality.

We have modified the original ISO/IEC 15504 to improve process attributes, integrating new ones together with the corresponding practices and requirements focusing on practical usability and accessibility. It is worth mentioning that the MODECUA model keeps up compliance with the capability scale defined in ISO/IEC 15504 due to its generality and applicability, preserving the standard application as in other software processes.

Table 4 shows the structure of U + A SPICE featuring capability levels (CL), which are enumerated from 0 (lowest capability or incomplete process) to 5 (highest capability or optimizing process):

  • CL.0 Incomplete Process: At this level, the process is not carried out or its purpose is not achieved. There is no attribute at this level; however, it is recommend the implementation of a training and awareness plan for the integration of the user-centered approach in the process and move forward to the next level of capability;

  • CL.1 Performed Process: At this level, all practices of the human-centered process (HCD) are carried out, and they are adopted as base activities;

  • CL.2 Managed Process: At this level, the practices of the human-centered design process (HCD) are managed; standards and techniques focused on usability and accessibility are applied, documented and monitored, in order to achieve the objectives defined;

  • CL.3 Established Process: At this level, human-centered practices are an important part of the organization’s strategy as usability and accessibility are formally established and considered as indispensable quality attributes in the development process. The organization has a multidisciplinary team of user experience responsible for applying the human-centered design process to all projects;

  • CL.4 Predictable Process: At this level, the performance of the process practices becomes clear, using metrics that control and measure their performance;

  • CL.5 Optimizing Process: At this level, the performance of the human-centered design process is continuously improved in a controlled and measured way. The multidisciplinary team of user experience is strategic and part of the organizational culture, participating actively and constantly in the conception and development of innovative products. The strategic area of user experience determines what types of projects should be financed as the benefits of the user-centered process are perceived.

Table 4 U + A SPICE structure including new additions
Full size table

For each capability level in Table 4, process attributes (PA), practices (P) and practice requirements (PR) are defined. In addition, the element type and the ISO source are indicated. The column “Type” can have two different values: N, when the row represents a new contributed element, and E when the row represents an existing one. The ISO source column also indicates the corresponding references.

As shown in Table 4, in order to ensure usability and accessibility, practices are extracted from standards like WCAG 2.0 [23] and other product quality standards such as ISO/IEC 25000 [22], own contributed work products, referenced proposals and so on.

3.3 Capability maturity assessment

Along with the capability levels, it is possible to measure the maturity of the organization’s processes to obtain a capability maturity approach according to five defined levels:

  • Level 1—Initial: At this level, the organization does not have a stable environment for user-centered software development and maintenance. Even when correct engineering techniques are used, efforts are undermined due to lack of planning. The success of the projects is mostly based on personal effort, featuring failures, delays and cost overruns. The user-centered results of the projects are unpredictable;

  • Level 2—Repeatable: At this level, the organization has institutionalized user-centered project management practices. There are basic metrics and a reasonable monitoring of quality. The relationship with users and stakeholders is systematically managed;

  • Level 3—Defined: At this level, the organization features a good management of user-centered projects, including correct coordination procedures between groups, staff training, more detailed engineering techniques and a more advanced level of metrics in the processes;

  • Level 4—Managed: The organizations has a set of significant quality and productivity metrics related to the user-centered approach, which are used in a systematic way for decision making and risk management. The resulting software is of high quality under a user-centered conception;

  • Level 5—Optimizing: The entire organization is focused on the continuous improvement of user-centered processes. The metrics are intensively used and the innovation process is managed.

The maturity level assessment in U + A SPICE is carried through the fulfillment of the process attributes (PA). The process attributes fulfillment can be obtained from each practice (P), through the fulfillment of each specified practice requirement (PR). To evaluate the maturity level, and in order to match the ISO/IEC 15504 ratings, process attributes are rated using the following values: L = Largely achievement or F = Fully achievement.

Table 5 shows the five measurable maturity levels in order to evaluate the maturity of usability and accessibility processes. Although the standard ISO/IEC 15504 does not specify the way of numerically measuring L and F, we proposed the following calculation procedure (see Fig. 3) as a systematic way to measure quantitative values for maturity according to the capability of the process attributes defined in Table 4.

Table 5 Assessment of maturity level in U + A SPICE
Full size table
Fig. 3
figure 3

Algorithm to evaluate the capability maturity in MODECUA

Full size image

As depicted in Fig. 3, in order to evaluate a certain capability level (CL) in terms of its representative process attribute (PA), it is necessary to validate the corresponding activities and practices (P). To do that, process activities and attributes are evaluated by scoring the degree of the fulfillment (in a scale ranging from 0 to 3) with each practice requirement (PR) in terms of the corresponding working products generated by each process activity. Ratings are processed in a summative way, using the following general scale:

  • 0 (Not achieved): No significant evidence of compliance with the activity or practice is found;

  • 1 (Partially achieved): Evidence with the activity or practice exists, but it is unpredictable or not directly related work products exist;

  • 2 (Largely achieved): There is significant evidence that the activity or practice is performed using and producing some of the work products related to the process. However, not all the products are strictly generated or some related weakness exists;

  • 3 (Fully achieved): There is sufficient and significant evidence that the activity or practice is performed using all the required work products of the process. No significant weaknesses exist.

As depicted in Fig. 3, in order to evaluate a process, it is necessary to assess the maturity of a capability level in terms of its representative activities and process attribute. This way, an average capability maturity value is calculated: \(\overline{{CM}_{n}}\), where n represents the capability level. This way, when \(\overline{{CM}_{n}}\) ≥ 2.55, then the desired capability level n is fully reached (F score). As specified, only L and F scores are considered for the maturity evaluation (see Table 5), though all scores can be useful for reporting purpose. This way, assessment values can be defined as follows:

  • From 0 to 0.45: Not achieved (N);

  • From > 0.45 to 1.5: Partially achieved (P);

  • From > 1.5 to 2.55: Largely achieved (L);

  • From > 2.55 to 3.0: Fully achieved (F).

These values are calculated according to the ISO/IEC 15504 recommendations, where N represents a 0–15% achievement, P represents > 15–50% achievement, L represents > 50–85% achievement and F represents > 85–100% achievement.

It is worth noting that the achievement of a certain maturity level implies the achievement of all capability levels involved. For instance, to reach maturity level 3, it is necessary to achieve capability levels 1, 2 and 3, and thus obtain F values for process attributes PA1.1 and PA2.1, and obtain L or F value for PA3.1. This way, the algorithm depicted in Fig. 3 has to be executed n times for each process attribute (PA) appearing in a same maturity level. This provides a quantitative value for the evaluation of the capability maturity according to the information depicted in Table 5.

MODECUA and U + A SPICE helps to corroborate H2, stating that it is possible to propose a model for the evaluation of the capability maturity in usability and accessibility centered processes.

4 Validation

In order to validate our approach, we have applied MODECUA to a real software development process used by a Mexican company that will remain anonymous to guarantee confidentiality. This company has been working for more than 20 years in the technology sector, featuring development and integration of software solutions. It has over 50 employees including developers, project managers, and persons responsible for maintenance, support and quality. Currently, the company utilizes RUP (Rational Unified Process) [26] as software development process. The RUP methodology comprises four phases, i.e., Inception, Elaboration, Construction and Transition, which are executed in an iterative-incremental way.

In order to apply our approach, it was necessary to have all the information about activities and work products generated by the company’s RUP development process. This information has been evaluated according to the U + A SPICE capacity scale in order to determine the level of maturity. Practices have been analyzed in detail, as well as the work products to determine the capability maturity level of the development process studied. Table 6 shows a summary of the activities and work products provided by the company that will be used in the evaluation with U + A SPICE.

Table 6 Number of activities and work products resulting from the development process of the company to evaluate
Full size table

Using the information of the company’s development process, we want to evaluate the process HCD.1 (Ensure HCD content in the system strategy) against capability level 1 (CL.1—Performed Process) and maturity level 1 (Initial). This implies, according to the information shown in Table 4, to satisfy the process attribute PA.1.1—Integration of the user-centered approach and thus fulfill practice P.1.1.1—Ensure the user participation in the system strategy as well as all stakeholders along the process lifecycle, and practice P.1.1.2—Ensure the integration of the user and organization requirements during the process. This entails fulfilling five and ten practice requirements for A.P.1.1.1 and A.P.1.1.2, respectively, which implies to consider the work products provided by the company that will be evaluated according to the practices in order to see whether the corresponding practice requirements are satisfied.

To carry out this task, it is necessary to achieve to following steps:

  1. 1.

    It is necessary to carry out an inspection to corroborate that the process HCD.1 and corresponding activities (HCD1.1–HCD1.6) included in MODECUA are implemented through the development process of the company studied;

  2. 2.

    It is also necessary to carry out an inspection to corroborate that all activities of the company’s development process generate the output work products corresponding to the activities of process HCD.1 (OWP.1.1.1–OWP.1.6.2) specified in MODECUA. To validate the compliance, an inspection of the project plan’s documents was carried out in order to obtain detailed information about activities, users, stakeholders, deliverables, resources and all the information about development;

  3. 3.

    Using the method specified in Sect. 3.3 and, more specifically, the algorithm shown in Fig. 3, the different activities of the process to be evaluated (HCD.1 in this case) have to be rated from 0 (not achieved) to 3 (completely achieved). In addition, a rating for each attribute process (AP.1.1.1 and AP.1.1.2) is calculated;

    1. a.

      A value between 0 (not Achieved) and 3 (fully Achieved) is assigned to each practice requirement, considering the work products, according to the following criteria:

      1. i.

        0 (Not achieved): No significant evidence is found that demonstrates the inclusion of the practice in the development process of the company studied;

      2. ii.

        1 (Partially achieved): There is evidence that the practice is carried out in the company’s development process. However, this practice does not generate any work product;

      3. iii.

        2 (Largely accomplished): There is significant evidence that the practice is carried out using and generating some of the work products specified by MODECUA. However, not all the work product are used or generated, and the achievement of the process is not as specified either;

      4. iv.

        3 (Completely achieved): There is sufficient and significant evidence that the practice is carried out using and generating all the work products specified by the MODECUA, and the process is achieved as specified;

    2. b.

      In the case of practice P.1.1.1, it will be rated with 3 if all practice requirements (PR.1.1.1.1–PR.1.1.1.5) are fulfilled. However, in this case, practice requirements PR.1.1.1.4 and PR.1.1.1.5 are not satisfied. This way, only 3 out of 5 requirements are satisfied, being the rating for practice P.1.1.1: (3 × 3)/5 = 1.8;

    3. c.

      In the case of practice P.2.1.1, it will be rated with 3 if all practice requirements (PR.2.1.1.1–PR.2.1.1.10) are fulfilled. In this case, all practice requirements are satisfied. This way, 10 out of 10 requirements are satisfied, being the rating for practice: (3 × 10)/10 = 3.

According to the results obtained (see Table 7), the final rating for this capability level 1 is: (1 + 2 + 1.8 + 3)/8 = 0.98, which corresponds to a P (partially achieved) value (0.45 < 0.98 < 1.5). This indicates that the maximum capability level for this process (HCD.1) is 1 (performed process) and the maturity level reached is 0, as a L or F rating for PA.1.1 is necessary in order to reach a maturity level 1 (initial). As the development process of the studied company did not reach maturity level 1 for process HCD.1, the evaluation through the next level (level 2) was not carried on. This indicates that the company lacks of an integration of human-centered design practices, and it has to improve process HCD.1 according to the scores obtained for each activity and practice. A detailed improvements report can be developed in order to research de desired capability maturity in the process to ensure a human-centered design content in the system strategy.

Table 7 Ratings for capability level 1 (CL.1—performed process) and maturity level 1 (PA.1.1—integration of the user-centered approach) for process HCD.1 (Ensure HCD content in the system strategy)
Full size table

Following the same steps described above, we calculated the rating (capability maturity level 1) for the rest of the processes. Results for all processes are detailed in Table 8.

Table 8 Evaluation results for all processes against capability maturity level 1
Full size table

As shown in Table 8, only processes HCD.3 (Understand and specify the context of use) and HCD.5 (Produce design solutions) reached capability maturity level 1. The rest of them do not even reach the initial level of capability maturity. As both HCD.3 and HCD.5 processes obtained a rating of L (largely achieved), the evaluation through the next level (level 2—Repeatable) was not accomplished, since it is necessary to obtain a rating of F (fully achieved) in PA.1.1 to evaluate PA.2.1 and thus reach a capability maturity level 2 (see Table 4).

This case study helped to corroborate H3, stating that it is possible to deal with process evaluation in order to improve it and be useful for audits and process improvement overall.

5 Conclusions

While usability and accessibility are important quality characteristics in software development today [27,28,29,30], there is a reduced number of capability maturity models focused on optimizing the user-centered development process. Admittedly, there are different capability maturity models for software processes, and some of them are centered in usability. However, there is a lack of approaches including usability and accessibility quality characteristics together, which heavily affects the quality of the products resulting from such processes, as well as the satisfaction and psychological wellness when the user interacts with such products.

This paper presents MODECUA, a model to determine the capability maturity in processes specifically focused on usability and accessibility. The main objective of MODECUA is to help companies carry out a guided and systematized user-centered approach for the development of usable and accessible software products, allowing to improve and determine the quality of their processes. MODECUA comprises seven improved processes from ISO/TR 18529 that we have contributed with eight new activities based on ISO 9241-210 and ISO 9241-20, 11 adapted activities and processes, and nine renamed activities. In addition, we have integrated 127 new work products used and generated by such new activities. In addition, a capability maturity assessment model, called U + A SPICE has been proposed. Inspired by ISO/IEC 15504, U + A SPICE allows to quantitatively measure the capability maturity against the activities and work products proposed in MODECUA. This capability maturity model is compatible with ISO/IEC 15504, as it uses the same maturity assessment mechanisms to be easily integrated in organizational quality models.

MODECUA, together with the proposed model to assess the capability maturity—U + A SPICE, have been applied to a real development process of an IT company in order to be validated, obtaining evidence corroborating that the proposed contribution can be used for reporting useful indicators for process improvement in terms of usability and accessibility.

We are currently applying MODECUA to evaluate other processes. This helps us refine and validate both processes and capability maturity assessment. In addition, we are planning to develop a supporting tool to systematize the capability maturity assessment in order to easily obtain and manage reports for audit purposes and control the evolution.