Keywords

1 Introduction

In the global economy, the most rapidly growing sector is the software industry, which has emerged as one of the key economic drivers for many nations [1]. Another significant entity that contributes to driving the world economy is the small software company (SSC). SSCs play an important role in their nations’ economies because of their ability to capture the kinds of markets that larger companies are incapable of reaching or have rejected. SSCs typically consist of less than 50 employees, and their aim is to create one or a few software products for their customers [2]. They also develop components for larger systems produced by other companies or offer maintenance services for the software products created by other firms [3].

To maintain their competitiveness in their target markets and to sustain a healthy relationship with their customers, SSCs need to supply increasingly faster and cheaper software products [4]. To develop such software products, these companies need to have efficient software development processes [5]. SSCs definitely need all the assistance in seeking the relevant information to make their software development processes efficient, but they often lack systematic process knowledge for determining which type of processes are more relevant to their context [3, 6]. One way to analyze software process knowledge is by systematically examining the existing literature on SSCs. Some systematic literature reviews (SLRs) have focused on analyzing the existing approaches toward software process improvement (SPI) in small and medium enterprises (SMEs). For example, Pino et al. [7] concluded that models such as Capability Maturity Model Integration (CMMI) and Software Process Improvement and Capability Determination (SPICE) are not suitable for SPI programs in SMEs and instead proposed the use of lighter-weight models. Sulayman and Mendes [8] concluded that very few studies have explicitly focused on SPI for Web companies. Paternoster et al. [9] and Klotins et al. [10] described the software development of start-ups. However, systematic review studies in the context of SSCs and their software development processes are limited [11]. Therefore, there is a need for systematic review studies in the SSC context with the goal to explore the state-of-practice of software processes and the types of challenges associated with it.

In this paper, we present a systematic review of the literature, with the aim of identifying the software processes utilized in SSCs. We have used the ISO/IEC 12207 [12] processes as an analyzing framework to describe both the current state-of-practice and challenges present in the software development processes. The paper proceeds as follows. In Sect. 2, the background and motivation are described. In Sect. 3, the systematic review is presented. Section 4 presents the results from the review, followed by a discussion of the research questions presented and our conclusions.

2 Background and Motivation

SSCs are considered as an important entity in the software industry and they represent up to 85 % of all software companies [13, 2]. They often face vicious competition from their competitors in developing quality software products on a strict deadline to fulfill customer requirements [2]. Their software development activity is generally in casual state [14]. Therefore, in order to improve their productivity, they need to improve their software development processes [15]. However, SSCs typically avoid process improvements [15]. This avoidance mainly appears due to the lack of financial and human resources [14]. To help SSCs, it is important for the software engineering research community to explore software processes in detail. Doing this will provide added value to the practitioners operating in the small-scale context.

There are some SLRs that have focused on the improvement of software development processes in SMEs. For example, Pino et al. [7] analyzed the existing approaches toward SPI in SMEs. They concluded that proper SPI programs based on models such as CMMI and SPICE are not suitable for SMEs and proposed the use of lighter-weight models. They also observed in their study that frequently improved processes in SPI programs are project management, documentation, requirement change management, process establishment, configuration management, and requirement elicitation. Sulayman and Mendes [8] conducted a systematic review study to determine the current state of research in SPI models and techniques used by small and medium Web development companies. They found that very few studies have explicitly focused on SPI for Web companies despite the large number of Web companies across the globe. Paternoster et al. [9] conducted a mapping study to explore the state of art of software development in start-ups. Their conclusion was that software engineering work practices are chosen based on the start-up context. Klotins et al. [10] also conducted a mapping study, in which they focused on the software engineering aspect of start-ups. They found that very few research papers have provided concrete evidence of software engineering knowledge areas in start-up companies.

However, systematic review studies in the particular field of SSCs focusing on the software development processes are limited [11]. To the best of our knowledge, an in-depth analysis of the studies reported in this context that describes the state-of-practice and the associated challenges with it does not appear to exist. To address this gap in the literature, we therefore decided to conduct a systematic review of the literature that focuses on software development processes in SSCs to explore this topic area. To analyze the software development processes in SSCs within a standardized framework, we decided to follow the classification of processes as defined in ISO/IEC 12207 [12]. ISO/IEC 12207 includes a process reference model that categorizes the processes related to software system into seven groups: agreement processes, organizational project-enabling processes, project processes, technical processes, software implementation processes, software support processes, and software reuse processes [12]. We did not consider ISO/IEC 29110 [16] as our analyzing framework since the standard is applicable to very small entities (up to 25 people), whereas our study focus is on SSCs (up to 50 people).

3 Research Methodology

Our literature review is based on the guidelines provided by Kitchenham and Charters [17]. We also used a software tool (StArt) to support our systematic review [18].

3.1 Research Question

The main goal of our review is to find all the research studies that relate to the software development processes of SSCs and that describe the key processes used and any related challenges. To achieve this goal, the following research questions (RQs) are defined:

  • RQ1: What is the state-of-practice in terms of the use of processes in small software companies? The outcome of this question is the current state of the processes used in SSCs. The state-of-practice is analyzed using the ISO 12207:2008 life-cycle processes.

  • RQ2: What are the challenges that small software companies face in the processes? The outcome of this question is the discovery of the challenges associated with the software development processes in SSCs.

3.2 Search Strategy and Data Retrieval

The initial sets of keywords for SSCs were taken from the work of Pino et al. [7]. Pilot searches in the key software engineering research domain databases were conducted, and new keywords were added to better target the searches to the desired data set. In the end, the search strings were formulated by combining the terms representing the population AND intervention. It can be summarized as follows: (X1 OR X2…OR Xn) AND (Y1 OR Y2…OR Yn), where X represents the population and Y represent intervention. The population (X) is represented as {software engineering process, software process, software engineering}, whereas intervention (Y) is represented as {small company, small enterprises, small organizations, small team, and small settings}. During the data retrieval process, the publication period was set to 2004 to 2014, with the purpose of summarizing the most recent related work. We used a total of four databases in our study:

3.3 Study Selection and Data Extraction

The database searches resulted in identifying 7967 papers, out of which 4022 were duplicates. The remaining 3945 papers were screened and assessed by two researchers. The inclusion/exclusion criteria were determined based on title, abstract, and keywords, implicitly or explicitly (see Table 1). The papers were categorized as “accepted”, “rejected” or “can’t decide” (in StArt terms, “unclassified” is used to the same extent). The “can’t decide” papers were discussed, and issues were resolved between the first and second author. After that, 104 papers were selected for full text reading. Thus, 41 studies were finally selected as the primary study papers. For quality assessment, we used the systematic and validated model [19] to assess the scientific rigor and industrial relevance of each primary study. For scientific rigor, we considered the following aspects: context, study design, and validity. For industrial relevance, we considered the following aspects: subjects, context, scale, and research method. More details about the papers, based on scientific rigor and industrial relevance, are shown in Fig. 1. Data extraction and analysis was done through qualitative data analysis using the NVivo software. We used the deductive approach for coding the primary studies [20].

Table 1. Inclusion/exclusion criteria
Full size table
Fig. 1.
figure 1

Overview of research method, rigor, and relevance distribution of papers

Full size image

3.4 Validity Discussion

In this section, we discuss the validity of our study in terms of construct validity, internal validity, external validity, and reliability [21]. Construct validity deals with whether the primary study papers focused on the software development processes and SSCs. During the search process, search strings were designed in such a way that they would collect a wide variety of papers related to this topic. To collect the papers, databases that are frequently used for accessing software engineering literature were used. During the full-text selection phase, manual screening was conducted twice to ensure that only papers relevant to the research questions were included. Reliability concerns whether the study can be repeated by the other researchers. Our study was based on a systematic review process protocol with well-defined search strings in the most common databases through the use of the SLR tool. Therefore, our review process can easily be reproduced by other researchers. The inclusion and exclusion criteria were piloted and followed during the review process. Internal validity refers to problems in the analysis of the data. During the execution phase, some primary study candidate papers did not show clear objectives and results, which made inclusion/exclusion criteria difficult. To overcome this situation, several meetings were arranged between the authors to evaluate the discrepant papers and eliminate the ones that did not match the inclusion/exclusion criteria. External validity concerns whether the results of an SLR can be generalized. In this situation, the results from this study were limited to the SSC context; therefore, both researchers and industrial communities interested in the software development processes of SSCs can benefit from our findings.

4 Results

As a result of the systematic review, a total of 41 primary studies were found to be relevant to the RQs. The descriptions of the 41 papers are shown in Table 2. In the following section, we give an overview of the primary studies. We then explore the state-of-practice and challenges associated with SSCs’ processes. To investigate the software development processes of SSCs in a systematic way, we classified the processes as defined in the ISO/IEC 12207 [12].

Table 2. List of primary studies
Full size table

4.1 Overview of the Studies

Rigor and Relevance.

The quality of the primary studies was evaluated according to rigor and relevance, as discussed in Sect. 3.3. As can be seen in Fig. 1, thirteen studies are in the upper-right quadrant (Rigor: High, Medium and Relevance: High, Medium) in the most appropriate region. Sixteen studies exhibited high industry relevance (Relevance: High), out of which ten showed low scientific rigor. Twenty-five studies exhibited moderate industry relevance, out of which seven demonstrated medium rigor and eighteen demonstrated low rigor. Based on this evaluation, we established the fact that most of the identified studies were conducted in cooperation with an actual SSC, thus scoring medium or high on the relevance scale. However, many of these studies had low rigor.

Research Methodology.

Most of the studies focused on empirical evaluations of theoretical concepts related to the software development processes and were applied in an SSC context, with the overall goal of assessing and/or improving software development processes. In Fig. 1, it can be seen that the majority of the primary studies are listed in the case study category. The other common methods were surveys and experiments. Also, looking at the left-lower quadrant in Fig. 1, it can be seen that the majority of the papers (28; 18 case studies, 5 surveys, 4 experiments, and 1 ethnographic study) fell in the low rigor category. Therefore, generalizability of the results of the given studies is low.

4.2 Processes

A detailed overview of processes as defined in the ISO/IEC 12207 and relevant primary study papers are shown in Table 3.

Table 3. Processes and relevant papers
Full size table

SW Implementation Processes.

Processes related to requirements are often considered critical processes for improvement since SSCs aim to quickly deliver what their customers want using their own practices rather than including customer commands in their development process [15]. Generally, functional requirements are the main focus during the requirement gathering of the software product [52]. One challenge that SSCs experience is understanding what the collected requirements really mean. The gathered information needs to be clearly understood to change it into clear product requirements. Clear understanding is crucial because it is important to determine which requirements match the business goal. Without a clear understanding, it is difficult to determine which product requirements need to put into which release plan to meet the business goals. [27] In respect to software qualification testing, some studies have reported that some companies have had troubles with testing their products [24, 25]. One problem was related to a deficiency of testing knowledge in the company. This lack of knowledge results in a lack of test plans, which causes uncertainty in companies during the testing process [25]. This lack of knowledge can be due to the absence of defined process descriptions or the quality of instructional manuals. However, Clarke and O’Connor [46] found that a majority of companies report general improvements in testing, including a periodic increase in test suites, the creation of a standardized testing process, and more emphasis on performance testing. Another improvement reported was measuring more code coverage of the automated tests [46].

SW Support Processes.

In regard to the software quality assurance (QA) process, Wilkie et al. [44] surveyed six companies and found that they each defined QA differently. For some, QA means testing, while for others, it is an instrument to assess a software product [44]. This confusion may be due to a lack of proper process knowledge. Regarding software configuration management (CM), one study [52] found that SSCs are able to perform basic CM tasks (i.e., version control, change management, and release management). One strength of the CM process in most of the companies was the code version control [52]. Some challenges with CM were also reported, such as uncertainty surrounding which work products should be with what version control and the lack of appropriate guidelines for CM [52].

Agreement Processes.

In the case of SSCs, no study has explicitly reported the use of processes within the acquisition process. The reason could be that SSCs are usually the ones who initially approach the customer. Since the amount of customers of SSCs is limited, this aspect is not well defined or of much interest. Regarding the supply process, Wangeheim et al. [3] found in their case study that this was one of the priority processes for improvement. In their other study [31] with eight companies, they found out that some of the companies had established systematic supply processes and had developed contract templates to improve relationships with their customers. Clarke and O’Connor [46], in their survey of 15 SMEs, found that those companies had made extensive improvements, specifically in the area of tendering, as well as in negotiating with customers.

Project Processes.

On the subject of project processes, two studies [3, 32] that assessed SSCs reported that project management is considered a high-priority process. Saastamoinen and Tukiainen [52] pointed out that project planning is an important aspect in SSCs. In addition, within the project plan, the development approach should be clearly described. For SSCs, a project is managed in an iterative or incremental way, tending to follow the waterfall model. A project’s work capacity and time approximation are usually calculated based on the project manager’s experience.

Organizational Project-Enabling Processes.

In relation to process establishment within life cycle model management, O’Connor et al. [29] explained that process establishment in SSCs is based on two categories: software development manager work background and market requirements [29]. From the perspective of an SSC, the benefits provided by a more standardized process could include increased competitiveness, superior customer satisfaction, and greater product quality [36]. Regarding process assessments, one study [14] reported that SSCs usually avoid the adoption of any process standards in their software development process. This may be due to the opinion that process standards are overly complex. However, in the context of process improvement, the objective for SSCs is to improve efficiency within the organization, improve productivity, and reduce development time [43]. SSCs tend to have problems with effectively improving their software processes [13], and the major difficulty could be a struggle to change [54]. In the context of knowledge management (KMP) within the human resource management process, Basri and O’Connor [55] stated that SSCs have clear KMPs within their organizations. They pointed out that knowledge waste is not a serious problem in a small-scale context. Also, informal communication and macro management can assist in creating team dynamics, which can further heighten KMP [55, 49]. Concerning quality management, requirements for ensuring that SSCs meet a certain level of quality are lacking. To improve the quality of their products, a clear definition of quality is needed, and goals for quality management need to be set. These quality goals can be altered based on the project [52]. In the context of software maintenance within technical processes, one study observed that software maintenance is poorly implemented in some companies and needs to be a priority for further improvement [24]. A number of companies have reported that they have amplified the refactoring level to reduce future support and maintenance costs [46].

Challenges in Process.

Some challenges were also observed in the primary study papers. A list of the challenges observed and experienced by SSCs within the processes is presented in Table 4. The challenges are mostly related to the following processes: software requirements analysis, software qualification testing, software quality assurance, software documentation management, software configuration management, and life cycle model management.

Table 4. Challenges in processes
Full size table

5 Discussion and Conclusion

In this paper, we have applied a systematic review method to analyze the literature related to processes within the SSC context, with the goal of exploring the state of the practice of the software development processes and the challenges associated with them.

In regard to RQ1: What is the state-of-practice in terms of the use of processes in small software companies? The current state of the practice related to the software development processes in SSCs can be seen in Table 3. It shows that software requirements analysis (49 %) within software implementation processes, project planning (39 %) within project processes, configuration management (22 %) in software support processes, and life cycle model management (56 %) in organizational project-enabling processes are the frequently considered processes for improvement when software process improvement (SPI) programs are conducted in SSCs. This suggests that SSC are mostly attracted to the above processes. The reason for the significant frequency of software requirement may be that requirements are considered an obligatory process and are often fundamental to any software company’s success [51]. Therefore, it is considered to be a critical process for further improvements since SSCs wish to quickly deliver what their customers want to remain competitive in a dynamic market and to maintain a healthy relationship with their customers [15]. Thus, focusing more on requirement engineering seems to be legitimate, based on the nature of SSCs.

In regard to RQ2: What are the challenges that small software companies face in the processes? Several challenges were also found in the context of processes, such as a lack of understanding in terms of what collected requirements really mean, a lack of testing knowledge, and a lack of CM guidelines (see Table 4). The reported challenges mostly fall under the frequently reported processes in Table 4. Typically, SSCs do not implement the necessary SPI programs for improving the process [15, 7]. In fact, most SSCs lack the required knowledge [14]. This could be a possible root cause for the many challenges revealed in the primary study papers.

Our observation from this review is that most of the studies we analyzed were conducted in collaboration with SSCs and that some also included medium-sized companies. Some papers referred to the SPI program based on models such as CMMI and SPICE within SSCs. We also observed that the SPI efforts carried out in SSCs are significantly reported in the literature, while descriptions of software processes in SSCs have received surprisingly little attention. An exploration of the empirical methods used in the primary studies showed that the majority of the studies were case studies. However, based on a systematic and validated model [19], our analysis shows that the majority of these papers fall under the low-rigor category (i.e., most of the studies lacked an adequate description of the study context, study design, and/or study validity). Therefore, further rigorous empirical studies within the context of software processes and SSCs are required. Various challenges were also found during the review that should be validated empirically in future research.