Keywords

1 Introduction

The Information Technology (IT) industry represents a significant sector of global economic activity. Although there are no consensuses about numbers, according to a study conducted by the International Data Corporation (IDC) in partnership with ABES (Brazilian Association of Software Companies), the global market for this segment is associated with a gross annual revenue on the order of USD 2,23 trillion. Software and Services represent around 50% of this amount. This study is conducted every year based on data collected by the IDC with their 50 offices around the world as well as through interviews with software suppliers and exporters [1].

Concurrently, according to a study by SOFTEX Software and Services Observatory initially published in 2009 and updated in 2012 [2], the net revenue of the software and services industry of Brazil increased more than the national GDP for the same period. The study also demonstrates that the fraction associated with the IT services companies had an important role in the increase in these numbers, considering that it grew more significantly than the software sector as a whole.

Understanding the relevance of the IT services sector worldwide and specially in the country and recognizing that the improvement of the productivity and quality of this segment are relevant factors for its continued growth; in 2011, the Software Engineering Research Group of PUCPR developed a maturity model for the management of IT services [3]. The aim of this project was to propose a process improvement model for IT services with evolutionary maturity levels, allowing the gradual implementation of their improvement – especially by small and medium-sized companies – and the possibility of being subject to assessment by an independent third party.

As the result of a close partnership between the university and the productive sector, in the following year (2012) this model was transformed into the MPS Reference Model for Services (MR-MPS-SV) of the Program for the Brazilian Software Processes Improvement (MPS.BR), becoming part of the family of maturity models disseminated by SOFTEX [4]. By that time, the MPS Reference Model for Software (MR-MPS-SW), that was released in 2013, was already well consolidated, with the official assessment of almost 400 companies and with a well-established environment that promoted improvement: consultants, lead appraisers, auditors, and an individual and cooperative business model, as well as the availability of public and private financing. From 2012, when the service model was launched, to late 2018, MR-MPS-SV was gaining ground in this environment, as demonstrated by the indicators: 63 companies officially assessed from different regions of the country; 1 foreign company officially assessed (Uruguay); 267 professionals qualified through official training; and, 52 professionals who passed the certification exams to perform the roles of implementer (consultant) and appraiser of maturity in IT services.

Although the initial numbers are encouraging, our previous studies indicate that process improvement programs face many difficulties in maintaining their activity in the organizations. According to [5], in a study focused on software process improvement programs, many factors are responsible for the maintenance or dismissal of improvement programs. The study grouped these factors into four categories: human factors, factors associated with the improvement project, organizational factors, and factors associated with the process itself. In turn, according to [6] and [7], companies that use agile software development methods mature in a way that is different from companies that use traditional maturity models. These studies focused on companies that were assessed by software maturity models; however, it is conjectured that the same factors, or very similar ones, are observed in IT service companies.

By 2015, the MR-MPS-SV model entered the second phase of its lifecycle, given that the first assessments – which are valid for three years – have started to expire. The assessment of the first company, conducted during late 2012, expired in late 2015 and was not renewed. The same happened with some other companies, whose assessment expired in 2016 and 2017. This leads to questions about the benefits perceived by these service companies regarding the use of this model that make them willing or unwilling to invest in this direction.

Therefore, the central question of this study is: What results were obtained by those companies officially assessed according to MR-MPS-SV reference model? Additionally, the study intended to identify the difficulties that were faced by these companies in the implementation and maintenance of the adopted practices and to explore how the various participants in the service process improvement environment perceive these difficulties and these benefits.

The answers to these questions could direct improvements in the MR-MPS-SV model, reformulations of the public policies for supporting the IT services sector, and reflections about the need for additional training and qualification as well as the establishment of new strategies to support the growth of this sector.

The general objective of the present study was therefore to investigate the results obtained and the difficulties faced by companies that have implemented the MR-MPS-SV model and whose official assessment was successful, from the perspective of the various participants involved in this environment.

2 Literature Review

2.1 IT Service Management (ITSM)

The main IT Service Management (ITSM) models used by the industry are the Information Technology Infrastructure Library (ITIL®) [8], the ISO/IEC 20000 Standard [9], CMMI-SVC® [10] and, in Brazil, MR-MPS-SV [4], as described next.

The mapping of the relationship among ITIL®, CMMI-SVC®, ISO/IEC 20000, and COBIT processes can be found in [3]. Additionally, a comparative study between ITIL® and ISO/IEC 20000 is provided in [11].

ITIL® – Information Technology Infrastructure Library

ITIL® was designed as a framework containing the best IT service management practices. According to [8], it “(…) provides guidance to service providers on the provision of quality IT services, and on the processes, functions and other capabilities needed to support them.” Its most recent version, from 2011, separates the IT service processes into five dimensions: Strategy, Design, Transition, Operation, and Continuous Improvement. The model does not present an adoption strategy based on maturity levels, and it is up to the organization interested in following its practices to determine the best route for implementation.

Throughout the years, ITIL® has become the standard currently used by the industry for IT service management. Companies worldwide are inspired by its good practices for structuring their IT service processes, adopting those that they consider the most useful or necessary for their daily routine.

ITIL® does not have an organizational certification process; i.e., it does not have an official way to communicate to the world that a given company follows its recommended practices. ITIL® certification is for the professional solely.

ISO/IEC 20000 International Standard

ISO/IEC 20000 (Information Technology – Service Management) is the first international standard created by ISO to discuss IT service quality management [9]. The standard is divided into 12 parts, some of which are still under design and have not been published. Like ITIL® framework, ISO/IEC 20000 is not organized in maturity levels but in processes. Some are general processes of the service management system, and the others are related to the categories of design and transition of new or modified services, service delivery processes, transition processes, relationship processes, and control processes.

Unlike ITIL®, the standard can certify; i.e., it is possible for an organization to undergo a formal evaluation process and obtain the ISO/IEC 20000 Certification. For such, the organization must have implemented all the requirements of the standard. The evaluation is performed by certified representatives. Currently, more than 600 organizations have been certified worldwide, of which 10 are in Brazil.

CMMI SVC – CMMI® for Services

The CMMI-SVC model (Capability Maturity Model Integration for Services) was originally conceived by the Software Engineering Institute (SEI) and is currently part of the set of products and services offered by the CMMI Institute [10]. Its structure is the same as the CMMI-DEV model, and it is widely used as a reference in the improvement and assessment of software processes. This structure contains 24 process areas organized into four maturity levels. A new version of CMMI suite was recently released.

As with the ISO/IEC 20000 Standard, a company can undergo an official evaluation process based on the CMMI-SVC. The difference is that the CMMI-SVC does not require the implementation of the whole model, so it is possible to evaluate whether the company complies with a specific maturity level (step-by-step approach) or the implementation level per process area (continuous approach).

The assessment is performed by application of the SCAMPI method, conducted by assessors certified by the CMMI Institute. It is valid for three years. After this period, a new assessment should be performed. Currently, approximately 800 companies in the world have been assessed using the CMMI SVC, of which 21 are in Brazil. Of these, only 15 are active.

MR-MPS-SV

The previously mentioned MR-MPS-SV model was developed at PUCPR [3]. Its structure also follows the format of maturity levels, as in the MR-MPS-SW and CMMI-DEV software models and in the CMMI-SVC service model.

The main difference between the MPS and the CMMI models is that the MPS processes are divided into seven maturity levels, where Level G is the initial level and Level A is the highest level. This allows the organization to perform smaller improvement steps within each level, which is particularly appropriate for small and medium-sized companies.

The first level is Level G, which processes are: Requirements Management; Work Management; Incident Management; Service Level Management; Service Delivery. Level F processes are: Measurement; Quality Assurance; Work Portfolio Management; Configuration Management; Acquisition; Problem Management. Level E processes are: Change Management; Human Resources Management; Organizational Process Definition; Organizational Process Assessment and Improvement. Level D processes are: Budget and Service Accountancy; Development of the Service System Level C processes are: Service Continuity and Availability Management; Information Security Management; Capacity Management; Decision Management; Release Management; Risk Management; Service Reports. Levels A and B have no new processes.

A company is assessed by appraisers authorized by SOFTEX and associated to an Assessment Institution. As with the other models and standards, this assessment is valid for three years. Through 8/30/2016, 63 assessments had been performed, 28 of which were in Brazil and 1 in Uruguay. As expected, because the MR-MPS-SV is still relatively new, most companies are at maturity Level G, which is the first level. Although it is more recent than the CMMI-SVC, the MR-MPS-SV has assessed more companies in Brazil than has the CMMI-SVC.

2.2 Studies on the Efficiency of Maturity Models

Authors in [5] studied the critical success factors for the maintenance of improvement programs in companies that implemented the MPS-SW model and found four categories of factors: human, organizational, technical, and improvement project. The study was based on (i) a systematic literature review, mapped with the support of grounded theory procedures and the Atlas.ti tool; (ii) a survey conducted with 21 experts (MPS SW assessors and implementers); and (iii) a case study in four assessed companies (levels G, F, and C) using semi-structured interviews and content analysis.

Periodically SOFTEX, in partnership with researchers from COPPE UFRJ, conducts a study on the improvements obtained by the companies that apply the MR-MPS-SW. According to the most recent study [12], companies assessed at higher maturity levels have more clients outside the country, have more hired personnel, develop larger projects, have larger production capacity, deliver products closer to the estimated delivery date, and deliver them with higher quality (fewer bugs). This most recent study was based on a database with 181 respondents from 148 different companies, extracted from a database with 500 assessed companies.

Although the results are related to companies that have implemented the software maturity model, it is conjectured that most benefits and difficulties are also to be found in companies that implement the service model. These studies supported the identification of related questions for companies that used the MR-MPS-SV, as presented later.

3 Research Method

For the selection of the research method, the characteristics of the research itself were initially considered. Regarding the research objectives, this study could be classified as a descriptive research. In relation to data collection procedures, this study is considered multiple sources, because it is based on the analysis of documents (from the assessments) and in a survey. Finally, this study is considered a quali-quantitative study. The research method selected was the survey, with data collection performed through questionnaires distributed to the participants in the ecosystem of the IT service process improvement. Survey was selected because it is: systematic, impartial, representative, theory-based, quantitative, and replicable. For such, the seven-phase process proposed by [13] was applied.

4 Results

  • Phase 1: Identify research objectives: The main objective of the study was to investigate the results obtained and the difficulties faced by companies that have implemented the MR-MPS-SV model and whose official assessment was successful, from the perspective of the various participants involved in this environment.

    To achieve this objective, the following specific objectives were identified:

    • To identify the factors that motivated the implementation of the MR-MPS-SV;

    • To identify the benefits obtained with the use of the MR-MPS-SV;

    • To identify the difficulties faced in implementing the MR-MPS-SV;

    • To identify the satisfaction levels of the companies in relation to the MR-MPS-SV.

  • Phase 2: Identify and characterize the audience: The study was performed with a focus on the identification of the perception of the companies officially assessed, using the MR-MPS-SV model as a reference, and on the implementers that were responsible for providing consultation on the implementation (when they are present). Initial focus on the companies officially assessed using the MR-MPS-SV until the cut-off point of the research.

  • Phase 3: Design the sampling plan: According to [13], for populations with fewer than 200 elements, such as in this study, the survey starts to be called a census and should cover the complete population, i.e., all companies assessed. In analyzing the information about the assessments, it was found that four of them had used a more recent version of the model (version 2015). The main difference between the two versions (2012 and 2015) is associated with the simplification that occurred in the process for a few of the initial levels. Specific considerations will be presented in the analyses of the results.

  • Phase 4: Design and write the questionnaire: The questionnaire was structured according to the guidelines by [13] to ensure coverage of all of the objectives in the design. The first block relates to questions about the characterizations of the company, the respondent, the implementation, and the MR-MPS-SV assessment. The second block had the statements to be analyzed, related to the expectations (motivators), benefits obtained, and difficulties with the implementation (which originated from the study by [5]). The third block, consisting of one part with 12 statements regarding the current use of the processes of the MPS SV model, complements the analysis of the difficulties and is based on the expected results for each process of levels G and F (the highest level implemented by the companies to this point) according to [4]. The fourth block, consisting of one part with six questions, is for the assessment of the company’s satisfaction with the MR-MPS-SV model. Due to space limitations for this paper, it is not possible to list the questions, but they can be obtained directly from the authors.

  • Phase 5: Run a pilot with the questionnaire: The decision was made not to run a pilot test but rather to conduct a peer review of the questionnaire, which was done by two experts in software engineering, one with 20 years of experience in the software and IT services process improvement initiatives in industry and the other with extensive experience in software engineering and in the scientific research in the area. Their suggestions allowed the questionnaire to be refined before it was distributed.

  • Phase 6: Distribute the questionnaire: The strategy for distributing the questionnaire was to do it through the institutions that provides consultancy on the implementation of the model because of the ease of contact and of following the completion process. The Qualtrics link was sent by e-mail.

  • Phase 7: Analyze the results and write the report: The collected data were imported to Microsoft Excel for filtering and analysis. The partially completed questionnaires were eliminated, and only those that were fully completed were considered for the analysis, as presented in the next section.

5 Results Analysis

5.1 Characterization of Respondents and Companies

In total, 25 individuals from 18 different companies answered the questionnaire. Among the respondents, 10 were sponsors, 13 were internal professionals who were responsible for the deployment or were part of the process team, and 2 were members of the process execution team (technical team). The time the respondents had been with the companies averaged 9.5 years. Among the companies, only one offers services outside the country, covering the United States. All of the others provide services only in Brazil. Two of them are MPS Implementing Institutions (II) and Assessment Institutions (AI). Most of them are small (45%) and large (33%) companies; micro and medium-sized, 11% each. Regarding the services provided by these companies, a varied portfolio was observed, including software development and maintenance, software testing, telephony and internet, and IT service monitoring, among others. There were companies that provide services to a small group of eight clients, and there were others with up to seven thousand clients served. The average implementation time for the model in the Level G companies was 13.6 months; in the Level G companies with additional processes at Level F, 13.5 months; and in the Level F companies, 11.7 months. As can be seen, the inclusion of processes in addition to those at Level G did not change the average time. Regarding the implementation costs, because this question was optional it was not possible to obtain conclusive data about those that implemented Level F or those that implemented Level G with Level F processes. Out of the 13 respondent companies that had implemented Level G, only 5 reported the costs, which averaged US 3,000. Out of all respondent companies, only four had covered 100% of the implementation costs with their own resources. The others had used their own resources and resources from a government agency, in the ratios of 20/80, 50/50, and 40/60.

5.2 Motivations for Implementation

Ten factors that motivated the implementation of the MR-MPS-SV model were presented:

  1. 1.

    The market was demanding certification

  2. 2.

    Competitors were implementing the MR-MPS-SV or similar model

  3. 3.

    We wanted to standardize the processes

  4. 4.

    We wanted to improve company productivity

  5. 5.

    We wanted to improve the quality of the services provided by the company

  6. 6.

    We wanted to improve the company’s profitability

  7. 7.

    We wanted to improve our image to the company’s customers

  8. 8.

    We wanted to reduce the number of company employees

  9. 9.

    We wanted to increase the number of customers

  10. 10.

    We wanted to increase the number of services offered.

Each of such factors should be evaluated using the scale 0 (I don’t know), 1 (it was not a relevant factor), 2 (this factor was slightly relevant), 3 (it was a relevant factor), and 4 (it was a very relevant factor). There was also one open question for the inclusion of other reasons. The results were grouped according to the role of the participants in the organization: sponsors (7 respondents), II and AI sponsors (3), professionals responsible for the process (13), and technicians and process executors (2).

As can be seen in Fig. 1, the most relevant motivating factors for the sponsors were standardization of processes, improvement of productivity, improvement of service quality, improvement of profitability, and improvement of the company image. Although they reported lower values, the professionals responsible for the process demonstrated much the same perception as the sponsors, as can be seen in the lines of the graph in Fig. 1. This is consistent with the perceived improvements.

Fig. 1.
figure 1

Adoption motivators factors.

Full size image

As also can be seen, for the II and AI sponsors the factors are different, and items such as the improvement of productivity, the improvement of profitability, and alignment to the actions of competitors were not relevant factors. This is understandable considering that these companies target a very specific market niche and because they were pioneers in this type of certification. Other reasons were mentioned in the open question, such as to improve control and management, to improve the meeting of deadlines, and to make good use of the free pilot project for the evaluation.

5.3 Perceived Benefits

In this part of the research, seven statements regarding the perceived benefits were presented: (1) Productivity; (2) Quality; (3) Profitability; (4) Image; (5) Employee Reduction; (6) Number of Clients; and, (7) Number of Services.

The respondents used the following scale: 0 (I don’t know), 1 (the benefit was not perceived), 2 (the benefit was partially perceived), 3 (the benefit was perceived), and 4 (the benefit was strongly perceived). There was also a space to report the estimated percentage of the improvement obtained and two open questions at the end of this phase for the respondent to report additional benefits and make comments. As can be seen in Fig. 2, once again the most distinct answer profile is that of the II and AI sponsors. For them, the greatest benefit achieved was the improvement of the quality of service provided (20% to 100%), whereas the other benefits were not perceived. This is consistent with what was indicated as the motivation for this group.

Fig. 2.
figure 2

Types of perceived benefits.

Full size image

In turn, the sponsors for the other company types perceived improvements that are primarily related to the following attributes: productivity, quality of the service provided, company image, and profitability, as can be seen in Fig. 2. The difference between the perception of the sponsors and that of the professionals responsible for the process is notable in that the latter are slightly more pessimistic than the former. Although they were fewer in number (only two respondents), the technicians mentioned the reduction in staff members, which was not confirmed by the other respondents at the same company.

As presented in Fig. 3, all sponsors reported an increase in productivity (average of 32.9%) and quality (average of 39.3%). Similarly, most of them reported an improvement of the company image (average of 40.0%) and of profitability (average of 9.3%). The latter is associated with a lower intensity. The comparison of the improvements perceived by the sponsors with the motivations for implementing the model indicates a relationship between them, which helps to explain the high level of satisfaction with the model. Most professionals responsible for the process reported an increase in productivity (average of 35.7%), quality (average of 37.5%), image (average of 48.0%), and profitability (average of 17.4%). By comparing the improvements perceived by the sponsors and those perceived by the professionals responsible for the process, it is evident that the average perceptions are similar, with the exception of profitability.

Fig. 3.
figure 3

Perceived benefits.

Full size image

In the open question, as additional perceived benefits, the following were mentioned: smaller workload, fewer bugs, better communication among team members, better workload planning, predictability of deliveries, stronger involvement of the areas favoring customer satisfaction, better management, greater quantity of data for decision making, better team rapport, greater process control, and reduction of stress levels among staff members.

5.4 Difficulties Faced

The difficulties faced by the companies were mapped using the 48 factors identified by [5]. It was found a clear difference in the behavior of the II and AI companies in relation to the others. For the latter, only seven types of difficulties were mentioned, of which the most relevant was associated with the high cost of the process, followed by the lack of resources (human and financial), workload, deadline pressure, and lack of standardization. In the other companies, three items stand out for being cited as the most relevant difficulties for the respondents: lack of human resources in terms of time, lack of human resources in terms of number of people, and workload. These are followed by the high cost of the process, complexity of the model, extensive documentation, and lack of standardization. In turn, it seems that there were no difficulties in relation to support from higher management, reduction in creativity, or lack of infrastructure.

5.5 Satisfaction with the MR-MPS-SV Model

In this part of the research, the satisfaction level of the respondent with the MR-MPS-SV model was queried. The possible answers were 0 (I don’t know), 1 (very unsatisfied), 2 (unsatisfied), 3 (partially satisfied), and 4 (completely satisfied).

Out of the seven respondent sponsors, six considered themselves completely satisfied with the model and one considered himself partially satisfied; however, the reason was that “they still need to mature more in the model” which suggests that the dissatisfaction related mainly to the current maturity level rather than to the model itself. Among the II and AI sponsors, all considered themselves completely satisfied. Of the 13 professionals responsible for the process, 10 considered themselves completely satisfied and 3 partially satisfied. The reasons cited were the high cost of implementation without subsidies, in the case of micro and small companies; the fact that some processes had not yet achieved successful implementation; and the perception that some requirements of the model were not appropriate for the organization.

In this part, there was also a question regarding what the company intended to do regarding to the improvement program. Out of the seven sponsors, five intended to renew the assessment at a higher maturity level, one intended to renew the assessment at the same maturity level, and one did not intend to renew it but did intend to continue using the defined processes. The reasons for the last response were not given. When queried whether they would recommend the model to a friend, 76% (19 respondents) indicated with no restrictions, and 24% (6 respondents) indicated with some restrictions. None of the respondents said that they would not recommend the model or that it would be with many restrictions.

6 Conclusions

The results indicated a high level of satisfaction with the MR-MPS-SV model as well as significant improvements in the productivity, quality, image, and profitability of the companies that had applied it, which is completely aligned to the Manifesto [14] values of promoting business value and success. Expressions that justified such satisfaction included “because it works”, “because I trust the model”, “because of the benefits achieved”, and “because it is excellent for the company” among others. The main difficulties in the adoption were also mapped and analyzed. It was concluded that the model defined by the academy demonstrates evidence of success in its implementation in industry, reinforcing the importance of research applied to the market.

The second step of this research is in progress and aims for the completion of the questionnaires by implementers that provided consultancy to the companies. The addition of a new perspective is expected, particularly regarding difficulties faced during the implementation. After the second step is concluded, a new study will be adapted for use on the companies assessed by the CMMI-SVC, to establish a comparison between the difficulties and the benefits perceived by the users of the two models. Because of the structural differences between these models, different sets of benefits and difficulties are expected to be found.