Keywords

1 Introduction

1.1 Assessment and Self-assessment

The Software Process Improvement (SPI) methodology, built on the Total Quality Management (TQM) principle, is a robust methodology defining a sequence of tasks, tools, and techniques to plan and implement improvement activities of software development processes, to achieve specific goals such as improved product quality and reduced costs [1]. The idea is that a good process will result in stable and robust outcomes. Continuously improving the process according to external and internal needs and analysing whether the process is executed in the way it is defined is the cornerstone of any process and maturity assessment. Most companies have a clear definition of how their processes need to be executed, through their process model. However, during the actual execution, the process model may not always be followed closely [2]. SPI is a process maturity approach, which gives an indication of how close a developing process is to being complete and capable of continual improvement through qualitative measures and feedback. Thus, for a process to be mature, it must be useful, automated as much as possible, reliable, and continuously improving.

Maturity has been used to assess growth in Information Technology (IT) organisations, with Capability Maturity Models, such as Capability Maturity Model Integrated (CMMI), SPICE (ISO/IEC 15504) and Bootstrap. “Empirical observation suggests their suitability as a set of benchmarks for strategic and tactical direction, as rationale and justification for projects, risk reduction through more consistent and tangible planning, measures for review, measures for control and reporting purposes and a mutually supportive learning cycle resembling continuous improvement [3]. These models are compatible frameworks providing a roadmap for Software Process Improvement (SPI). Maturity scales tend to be outcome-based and therefore suited for control and direction setting.

Self-assessment can provide insights into the comprehension of SPI efforts and show their compliance to the different process models [4]. The usefulness of a self-assessment tool is that organisations applying SPI can use the tool themselves, when suitable/convenient for them, without incurring any extra costs for appraisal of the degree to which SPI Manifesto principles, practices and processes are applied.

The STEEPLED analysis of the SPI Manifesto represents a process model that provides a clear definition within the principles of what an organization should do to promote behaviours underpinned by the Principles and Values of the SPI Manifesto. These are further decomposed in the STEEPLED dimensions that would eventually help to reveal areas, practices and processes requiring improvement, help the organisation to understand where to focus their improvement efforts and will support the formulation, implementation, and follow-up of future activities.

The direct effects of using the self-assessment tool are that it will identify perceptions of relevant staff regarding required practices and processes, it will trigger discussions and collaborations, it will improve understanding of their own improvement capabilities, and finally it will increase understanding and insight into applied practices and processes. Above all, the main usefulness of the self-assessment tool could be that it can and will help the organisation to identify possible current and perhaps potential risks and gaps in its knowledge and application of the SPI Manifesto. Addressing them is the next natural essential action for improvement.

In this paper the authors use the STEEPLED analysis of the SPI manifesto’s principles and values [5, 6]. as the appraisal process model, as exposed in [7,8,9,10,11,12]. At its core, it is a three-phase approach, namely the assessment of compliance to the SPI three values and ten principles; analysis for the purpose of identifying gaps in awareness and knowledge regarding compliance with the values and principles; and addressing of the identified gaps. Self-assessment can help a company to reflect on their problems and extent of its abilities so that they can prepare to improve with guidance by official performance appraisers. A self-assessment should be the preparatory pre-cursor of a professional assessment by external and fully qualified assessors.

1.2 Software and Software Process are Multifaceted, Multidimensional, and Complex

Both software artefacts and (the) software processes are multifaceted, multidimensional, and complex. Because of these characteristics, many factors are involved which at any point in the process can create problems and failures. It is, thus, imperative for software producing organisations to continuously understand, control and improve their process. Requirements Engineering (RE) is a widely studied field of software development to denote the systematic handling of software and systems requirements and needs. Requirements elicitation is the first activity in the RE process and one of the most important and critical phases in software and systems development, which has direct influence on quality and cost. The requirements elicitation process is a communication intensive process that involves critical activities required for accurately capturing the requirements/needs of diverse stakeholders, who have a business interest in the system under development. To understand the requirements of the Multidimensional Self-assessment tool several multidimensional analyses of the SPI Manifesto were carried out in different stages and from different viewpoints, identifying the nature, importance and strength of relationships between the SPI Manifesto’s Values and Principles in terms of eight dimensions encapsulated in the STEEPLED analysis, which was also validated by ten (10) experts through capturing their viewpoints, which subsequently enabled the researchers to make suggestions for improvement.

Using the integrated STEEPLED Analysis shown in Table 1 the authors focus on self-assessment of a software organisation’s process, as a preamble to a professional assessment.

Table 1. STEEPLED analysis of the SPI manifesto (source [7])
Full size table

At this stage of the research, the authors aim to provide an opportunity to both the management and the developers in a software organisation to focus (using the above template) on identifying and understanding the SPI issues and current problems.

2 Research Purpose and Research Design

This research is both qualitative and quantitative. It aims to analyse the relevant requirements and organisational needs for the design and implementation of a self-assessment tool, which can potentially be used by a software developing organisation. By using the earlier analysis template created by Georgiadou et al. [7] and [8], the researchers and authors posit that an organisation’s software process can internally be understood through self-assessment. The outcomes of this stage of the pilot research study will facilitate a subsequent professional assessment by qualified assessors.

The research phases, tasks and expected outputs (shown in Table 2) encompass a longer-term plan. The self-assessment tool, implementation and the full, formal, external assessment as well as the training of assessors (through an ECQA job role training and certification) will follow.

Table 2. Phases of the research, associated tasks, and expected outputs.
Full size table

Using the eight STEEPLED dimensions within the relevant relationships of the SPI Manifesto’s Principles and Values, it can be seen for example that “Value 1: Must involve people actively, and affect their daily lives”, is related to “Principle 2: Motivate all people involved” involves five dimensions namely Sociocultural [S], Economic [E], Environmental [En], Ethical [Et], and Demographic [D]. Thus, the questions posed to the employees of the software development company should cover all these aspects.

Similarly with all thirty (30) possible connections/relationships between the Principles and Values as well as the eight dimensions involved, a total of 240 possible connections is likely to both confuse and deter employees of an organisation (at whatever level) from meaningful engagement in the self-assessment effort.

Thus, in the proposed pilot tool questions will concentrate on relationships that have already been identified by experts, who were involved in validating the STEEPLED Analysis of the SPI Manifesto, as having High or Very High importance.

The self-assessment will identify the perceptions of the relevant staff regarding the organisation’s required processes and indicate to which degree the practices are applied. The results of the self-assessment will also help the researchers/developers of the assessment tool to improve the design of the tool itself. The testing of the tool is planned to be carried out in a small number of organizations selected according to size and application domain. From the experience and learning from the field study of self-assessment, a tool for a novel and complete professional assessment will be developed for a full professional process assessment, which can potentially involve all 240 possible relationships between the STEEPLED dimensions and the SPI Manifesto’s Principles and Values.

The current structure of the SPI Manifesto shows that: 1) there is a need for ethical issues to be explicitly infused into the Values and Principles [10, 11] and [12], and 2) each principle only involves one of the three Values. The results of the STEEPLED analysis, however, showed that there is far more complexity, and thus more interconnections between the three Values and the ten Principles of the SPI Manifesto [7] and [8]. These inter-connections not only directly involve the principles, but also influence the principles that are either mentioned explicitly or are implied in the SPI Manifesto.

Ten experts from industry and academia were asked to grade the importance on a Likert scale: Very High (VH), High (H), Medium (M) and Low (L), for each original relationship proposed in [1] and [2]. The experts awarded a grade for every possible relationship of the form PiVj (where i = 1, 2..,10) and j = 1, 2 or 3. The weighted mean values (replacing VH with 4 and H with 3 and multiplying the values by the number of experts of every interconnection graded VH or H were calculated to identify which relationships were of the utmost importance. Choosing a weighted mean of 20 as a minimum, resulted in thirty (30) interconnections being judged as the most important.

The largest number of interconnections were found in the Sociocultural Dimension (14), followed by the Technical Dimension (8). In the Legal Dimension (5) dimensions reached the minimum and the Ethical Dimension (3). In the Environmental and Demographic dimensions no interconnection reached the minimum of 20, although several scored 18 or 19.

Thus, the most suitable questions for the 30 relationships that scored 20 (using weighted averages) must be formulated in preparation for the self-assessment phase.

3 Development Studies and Practice in Process Modelling Technology and Self-assessment Tools

3.1 Related Work

Process modelling and automated tool support are largely associated with process metamodeling tools [13] and method engineering (ME) technology [14] in MetaCASE and Computer Assisted Method Engineering (CAME) environments [15]. Traditionally, modelling a business process provides a form of hybrid understandability [16] for analysis and design and general understanding of the business subprocesses. In addition, process metamodeling technology advances process improvement efforts and handles people’s and organisation’s needs for communication, collaboration, co-ordination and change [17, 18]. The latter are realized through groupware technology that also improves organizational learning and project management while offering continuous software process improvement [19]. Multiple meta-tools (e.g., MetaEdit+, MetaView, and MENTOR) inspired from various process-oriented industries have traditionally been developed and used during the last 35 years [16, [20] and [21]. Some had to be adapted for benchmarking use and for particular business needs and requirements modelling aspects.

3.2 Self-assessment Considerations and Questions for the STEEPLED Dimensions and for the Automated Tool Version

There is a plethora of different categories of questions to be considered and asked in a bottom up or top-down approach while utilising the STEEPLED framework’s eight (8) dimensions. Herein a number of general and generic (polymorphic) questions are presented to be considered as a small representative sample of each dimension. Many of them can be under many categories and, as can already be seen, the most recent experts’ evaluations considered, perhaps, that many questions can be categorised as political and/or demographic, environmental and/or sociocultural, and the list can go on. The final decision of their classification as factors of one or other dimension of STEEPLED could be of personal perception, experience, knowledge or, simply, personal viewpoint or framework of thinking. Question asking itself is a craft and an art for learning and understanding [20]. Subsequently, automated question asking as part of self-assessment and benchmarking can be tricky but finally rewarding because it will, no matter what, improve performance (efficiency). Hence, the efficiency parameters will, eventually and sooner or later, be identified and concretized in the minds of people involved; and that itself constitutes personal process improvement and related business process improvement.

In so thinking and doing, the list that follows next is not exhaustive; it comprises only some instantiations (or nested meta-modelling constructs) of other more general question clusters. Theoretical tools such as the notion of weights [19,20,21,22,23] and meta-modelling requirements/needs prioritisation could be used in the self-assessment tool’s prototype and for the initial tool’s more developed and later on more advanced versions.

3.3 Facilitating Participation in Self-assessment

Although the STEEPLED Analysis of the SPI Manifesto shown in Table 1 is a succinct representation of the relationships between the SPI Manifesto’s Values and Principles, a more understandable and practical method is the direct questions (shown below) prompting the practitioners to contemplate and answer without direct recourse to the eight STEEPLED dimensions.

3.3.1 Sociocultural Dimension

  • What social attitudes and social taboos could affect your business? Have there been recent socio-cultural changes that might affect this?

  • How do religious beliefs and lifestyle choices affect the population?

  • Are any other socio-cultural factors likely to drive change for your business?

  • How do national and organizational cultures affect business and change orientation?

  • How do socio-educational and socio-technical cultures affect your organization?

    E.g. On which areas do local educational institutions focus their research?

  • Can you take advantage of the local know-how in your future workforce?

3.3.2 Technological Dimension

  • Are there any new (effective/efficient) technologies that the organization could be using?

  • Are there any new technologies on the horizon that could radically or periodically affect your work or your industry?

  • Do any of your competitors have access to new technologies that could redefine and diversify their products?

  • How have technical infrastructure changes affected work patterns (e.g., levels of remote/online working)?

  • Are there existing local or worldwide technological hubs and (virtual) communities-of-practice that you could work with or learn from?

3.3.3 Economic Dimension

  • How stable is the current economy? Is it growing, stagnating, or declining?

  • Are key exchange rates stable, or do they tend to vary significantly?

  • Are customers' levels of disposable income rising or falling? How is this likely to change in the next few years?

  • What is the unemployment rate? Will it be easy to build a skilled workforce? Or will it be expensive to hire skilled labour?

  • Do consumers and businesses have easy access to credit? If not, how will this affect your organization?

  • How is globalization affecting the economic environment?

  • Are there any other economic factors that you should consider?

3.3.4 Environmental Dimension

  • Are there any environmental problems regarding e.g., local natural environment?

  • Is the organization interfering consciously and/or unconsciously in the local socio-economic balance?

  • How important are the issues of the organisation’s local business and natural environment for operations management?

  • How does government approach corporate policy, corporate social responsibility, environmental issues, and customer protection legislation? What impact does this have on the company, and is it likely to change?

  • Are there any technological factors that affect (for better or worse) the environment you should consider?

3.3.5 Political Dimension

  • Who are the geographical regions formal and informal political powers?

  • Do trade unions have a strong role in employment/contract policies?

  • When is the country’s/city’s/region’s next state/local/regional election?

  • How could the current/next government councils affect regional/demographic/new/old job opportunities’ policies?

  • Who were/are the most likely contenders for power? What are their views on business policies, and on other policies that could affect the organization?

3.3.6 Legal Dimension

  • How well developed are property rights and the rule of law?

  • How widespread are corruption and organized crime?

  • How are these situations likely to change, and how is this likely to affect the organisation?

  • Could any pending legislation or taxation changes affect your business, either positively or negatively?

  • What is the likely timescale of proposed legislative changes?

  • Are there any other political factors that are likely to change?

3.3.7 Ethical Dimension

  • Are there ethical considerations organised by bodies and/or committees of Ethics?

  • Are trade unions involved in the understanding and dissemination of human and work rights?

  • Are workers’ associations involved in keeping e.g. knowledge workers privacy and other rights?

  • What are the personal beliefs of the staff members about Ethics? Do they follow a particular religious, philosophical, professional etc. body of ethical standards/regulations?

3.3.8 Demographic(al) Dimension

  • What is the population's growth rate and age profile? How is this likely to change?

  • Are generational shifts in attitude likely to affect what you are doing?

  • What are your society's levels of health, education, and social mobility? How are these changing, and what impact does this have?

  • What employment patterns, job market trends, and attitudes toward work can you observe? Are these different for different age groups?

4 Pilot Tool Design

An automated tool can be developed to automate the implementation of the concepts already mentioned and briefly explained in Sect. 3 in terms of self-assessment, official assessment, maturity assessment(s) and compliance to SPI Manifesto principles and values. Using a Likert scale, qualitative judgments can be made as to the degree of compliance of one of the eight STEEPLED Dimensions with a certain combination (relationship) of an SPI Manifesto Principle and an SPI Manifesto Value.

The pilot tool will provide three modules, namely A: SPI Self-Assessment by all relevant employees at all levels of the organization. B: SPI Official Assessment by certified internal assessors, and C: SPI Compliance score: a fully automated calculation based on the results of A and B.

SPI Self-assessment: This feature allows the internal SPI assessors to perform an internal self-assessment in order to determine their compliance for the SPI manifesto principles and values (considering the relationships mentioned in Table 1). The internal assessors will be prompted with the questions introduced in Sect. 3.3 (above) as a first step of self-assessment, without necessarily the STEEPLED dimensions. The responses will be in free text, which will be discussed, and manually quantified using a Likert scale (Very High, High, Medium, Low). These grades will be entered in a database using an interface as shown in the screen shot below. Each group of questions (Sects. 3.3.1–3.3.8) will, thus, provide a profile of scores and will focus the attention on the most significant areas that will require improvement. An example of a potential dialogue is shown in Fig. 1.

Fig.1.
figure 1

Interactive screen

Full size image

For each of the answers (a, b, c) provide a quantitative score on the Likert scale Very High, High, Medium, Low as shown in Fig. 2.

Fig. 2.
figure 2

Quantitative evaluation/assessment

Full size image
  1. A.

    SPI Official Assessments: This is a similar feature to A, but it allows only the certified SPI assessors (i.e., SPI managers) to perform these official assessments to determine the company’s compliance to the SPI manifesto values and principles (considering the relationships mentioned in Table 1: STEEPLED Analysis …).

  2. B.

    SPI Compliance Score: Upon completion of the Self-assessment or Official assessment, the tool shall generate a final score representing the % of compliance to the SPI principles & Values.

    The compliance score uses the normal SPICE rating scheme: N P L F;

    1. i.

      N: Not achieved - 0 to ≤15% achievement

    2. ii.

      P: Partially achieved - >15% to ≤50% achievement

    3. iii.

      L: Largely achieved - >50% to ≤85% achievement

    4. iv.

      F: Fully achieved - >85% to ≤100% achievement

  3. C.

    Accuracy Indicators: Upon the completion of the points A, B and C, the tool will provide a comparison feature by comparing the compliance score of Self-assessment vs. the compliance score of official assessment.

    1. a.

      If the deviation is more than 10%, then the Self-assessment in Inaccurate

    2. b.

      If the deviation is within range ≤10%, then the Self-assessment is accurate

If the Self-Assessment scores are different to the official assessment by a professional assessor, the accuracy KPI is indicated as in the example shown below in Fig. 3.

Fig. 3.
figure 3

An example showing the accuracy KPI

Full size image

During the design of the automated tool, the following three, among other, quality constraints need to be addressed and realised carefully:

  1. 1.

    Usability of the Tool: The tool should use easy and uncomplicated English language so that both native and non-native English speakers, from different companies and backgrounds would be able to use and understand the questions. Representing all the relations mentioned in table 1 into the simpler questions suggested in Sects. 3.3.1–3.3.8.

  2. 2.

    Length of the Questionnaire: Given the relations mentioned in Table 1, there will be too many questions. This carries the risk of hardly anybody devoting a lot of time to complete the questionnaire. This potential problem could be avoided by merging some of the questions together in a smart way so as to ensure a complete coverage of the relationships.

  3. 3.

    Priorities of the Relationships: from the Business (or People or Change) point of view, some of the relationships are more important than others. This issue should also be considered as a prioritisation aspect with suitable technological means while designing the tool. Calculation formulae should have a weight for each question and be supported online for continuous indication and feedback while utilising the tool.

5 Conclusion and Future Work

5.1 Why a Self-assessment and Benchmarking Tool?

The improvement of a (software) process through assessment and evaluation procedures has been the focus of research and development of many past and more recent research and development efforts [21]. An easy to use, widely acceptable and lightweight self-assessment tool is in the wish list of many organizations because it can reveal organizational dynamics and can create a potential for reflecting on future desirable opportunities for the organisation’s people, business and changes and for enhancing trust in knowledge experts in software development lifecycle [20]. In terms of SPI and the three values it supports (People, Business, Change) through SPI’s ten principles, a self-assessment technique based on STEEPLED dimensions could reveal a multidimensional palette of personal and business opportunities while providing informed advice on significant threats for people and business while planning for embracing changes.

Combined with process benchmarking, as a supportive technology, a suitable self-assessment tool combining values and principles of SPI through the STEEPLED framework could further assist in revealing the direction of change and innovation within the business environment. This, in turn, could shape what people will/could be doing, so that they are convinced to embrace change -and will not passively or actively resist it-, because they will understand the methods and processes. Additionally, a multidimensional tool could equip the compare-and-contrast approach of benchmarking as an integral part of a learning organisation for avoiding starting projects that are likely to fail, for reasons beyond organizational control (e.g., political, sociocultural, etc.). Finally, utilizing self-assessment (and formal assessment), a benchmarking (thinking or automated) tool can help in eliminating unconscious assumptions or management anti-patterns [24] and [25] in the traditional, agile and lean decision-making process [26] while entering a new, unknown country, region, market or starting a new project. The latter can be achieved smoothly because benchmarking, apart from reducing costs and unnecessary capital ventures to the unknown, can assist in creating a compare-and-contrast informed knowledge decision and, thus, developing a more objective and critical viewpoint about this new and future business environment for the organisation’s operations. This leads to people’s involvement for overall quality improvement efforts [27] and [28] focusing on performance [29, 30] in particular, through measurement and assessment instruments for capturing controllable and uncontrollable factors in software development [31, 32]. People’s involvement and successful benchmarking can be successful by-and-large when adopting an open-source development process model and open platform for continuous feedback for improvement and tool integration [33] and [34] according to formal standardization procedures [35].

5.2 Future Work

Following the SPI Pilot Self-Assessment Tool (SPIPSAT) as a working prototype, practitioners from industry will be invited to use the tool in order to focus their efforts for a subsequent professional assessment by external, certified assessors. In parallel to this, an ECQA [36] job role for SPI Assessors will be developed. The authors of this paper as well as independent participants and consultants from industry will attend the course. SPIPSAT will be extended for a full independent assessment tool (SPIPAT) used by the certified SPI Assessors in order to carry out assessments. A database with the results and lessons learned from both the self-assessments and professional assessments will be created. Longitudinal studies will ensure continuous improvement to the tools, to the SPI Movement as well as the SPI Manifesto.

With the help of experts in question asking a coherent set of questions on the STEEPLED dimensions will be developed and optimised for the needs of question asking as a vital process [37] of the self-assessment tool’s architecture. There is also a need to involve experts on both assessment and tool architectures for a more relevant and succinct set of usable questions without overloading the tool users and discouraging them to answer.