Keywords

1 Introduction

The potential of software startups has been widely acknowledged as one of the effective ways for disruptive innovation. Even though software startups are inexperienced, young, and immature, their innovative products and services are putting well-established market leaders under pressure [18]. Software startups are “organizations looking for a repeatable and scalable business model for an innovative product or service they develop where software represents a core element” [11]. Software startups offer new products, new business models, and new business value at high speed with cutting edge technology, e.g., Internet of Things (IoT), Artificial Intelligence (AI), Block-Chain, etc. However, despite the success of well-known startups such as Uber, WhatsApp, Airbnb, the failure rate of software startups remain alarmingly high. A recent report in 2019 revealed that only one in twelve startups succeed [7]. This phenomenon has caught the attention of researchers from multiple disciplines [22] to discover obstacles to be minimised, and opportunities to leverage the success of software startups.

While the earliest scientific publication on software startups can be traced back to the year 2000 [18], it was not until 2016 when software engineering academics attempted to establish software startup as a research area [19]. Some literature reviews on software startup research have been reported [3, 9, 12, 22]. The reviews indicate a rise of software startup research that reflects its significance in today’s modern economy. There has been observed an increased acceptance of software startup research as one of the research interests in leading software engineering and information systems conferences, e.g., XP, PROFES, ICSOB, ICIS, etc. Given the rapid growth in software startup research, we are interested in updating how the software startup research area evolves over time.

The most recent literature review on software startups was carried out by Wang in 2019 [22], reviewing a total of 133 studies indexed by Scopus from 1994 to 2019 examining the contributing disciplines. The paper by Wang [22] served as an inspiration for this paper, but we chose to (i) incorporate different terms for software startups into the search string, (ii) use two new digital libraries i.e., Clarivate Web of Science (WoS) and Association for Information Systems e-Library (AIS), (iii) and address new research questions not covered in earlier research. A revisit on these areas can potentially reveal how software startup research has evolved over the past years. To achieve our research objective, we formulated the following research questions:

  • RQ1 - What are the contributing disciplines in software startup research?

  • RQ2 - What research methods and theories are used in software startup research?

  • RQ3 - What are the past, present, and emerging topics in software startup research?

To answer our research questions, we employed a Systematic Mapping Study (SMS). The purpose of a SMS is to structure a research area by examining existing literature for the nature, scope, and number of primary studies [13]. SMS provides a broader view of wide and often poorly defined research areas, and has therefore been considered more appropriate and beneficial for this study.

The contributions of this paper are two-fold. The first contribution is to provide a mapping of current research in software startups area in terms of contributing disciplines and research methods and theories used. The second contribution is the identification of new and emerging research topics, which may allow us to update the research agenda to advance the software startup research area [19].

The remainder of this paper is organised as follows. Section 2 describes the related work in terms of the relevant studies for this paper. The research methodology is described in Sect. 3. Section 4 outlines the findings, followed by a discussion of the obtained results in Sect. 5. Finally, Sect. 6 concludes the overall study.

2 Related Work

To find relevant literature reviews, we run a series of searches in the Web of Science (WoS) digital library. We used different combination of terms for the search strings such as “software startup” AND “systematic mapping”. Following the identification of a relevant publication, the related work section was examined to find more related publications. We identified four literature reviews [3, 9, 12, 22] that are relevant. A summary of the related work is presented in Table 1.

Table 1. Overview of relevant literature reviews.
Full size table

The first SMS on software startup research appeared in 2014 [12] by Paternoster et al. to review state-of-art research in software startups. A year later Klotins et al. [9] conducted a SMS to identify the gaps for future research into software startups by categorising the software engineering knowledge areas utilised in the software startups. The SMS by Paternoster et al. [12] received a lot of citations, and inspired Berg et al. [3] to conduct a SMS in 2018 to identify how engineering activities in software startups have changed over time, and identify potential research gaps. In contrast to the two previous SMS [9, 12], Berg et al. attempted to synthesize startup descriptions in research and its associated knowledge areas of software engineering [3]. The review by Wang [22] focused on the evolution of software startup research area through an examination of the scientific publications and contributing disciplines.

Even though these studies are focusing on software startup literature, Table 1 highlights the variety of paper counts used in the existing literature reviews (i.e. 43/14/74/133). A reason could be because of the combination of the digital libraries used. For example, the coverage of the digital libraries used in the reviews vary. Furthermore, each review has different selection criteria. For example, the study by Paternoster et al. [12] included papers that presented a contribution (i.e in the form of an experience report, applied engineering practices, development models, or lessons learned). The level of maturity in the software startup research area may also play a role, as indicated by the later literature reviews greater quantity of papers.

The review protocol used by Paternoster et al. [12], Klotins et al. [9], and Berg et al. [3] adhered to the mapping study guidelines in [13], while the study by Wang [22] used a research method designed by Coccia [5]. In terms of data sources, both reviews by Paternoster et al. [12] and Berg et al. [3] used multiple data sources, e.g. Compendex, Web of Science, Scopus, etc., while the reviews by Klotins et al. [9] and Wang [22] used only one data source.

3 Research Methodology

To address the research questions, a systematic mapping study [13] was conducted to provide an overview of the growing research area of software startups.

3.1 Search Process

To find relevant literature for the systematic mapping a search string was constructed to generate search results within the field to select and analyse. The generic search string is:

“software startup” OR “internet startup” OR “digital startup” OR “web startup” OR “computer startup” OR “software entrepreneurship” OR “internet entrepreneurship” OR “digital entrepreneurship” OR “web entrepreneurship” OR “computer entrepreneurship”

The search string was used on two digital libraries: WoS (as it also covers IEEE Xplore and ACM Digital Library) and AIS for information systems literature coverage. A total of 407 papers were identified using the search string, of which 370 were discovered in WoS, and 37 in AIS.

3.2 Selection Process

Duplicates discovered between the two digital libraries were removed. Then, a screening of the papers was carried out on the metadata level, in particular on the title, abstract and keywords. In this study we have established inclusion and exclusion criteria, as listed in Table 2. We are only interested in the primary studies, and therefore exclude the secondary and tertiary studies. Texts not written in English, missing abstracts, or we cannot access the resource through the university library portal are excluded from this study.

Table 2. Inclusion and exclusion criteria.
Full size table

Each paper was evaluated by all reviewers. Any disagreement or confusion was resolved in the present of the reviewers. Applying the selection criteria in Table 2, we excluded 291 papers from WoS and 21 from AIS. In total 95 primary studies were included for further analysis.

3.3 Categorisation

The objective of this study was to investigate the contributing disciplines, the research methods and theories employed, and the research topics covered in the software startup research area. As a result, three categories have been constructed, each attempting to answer the corresponding research question, which are detailed in the next subsections.

3.3.1 Contributing Discipline.

Every research paper in the WoS digital library may be categorised according to multiple research areas, e.g. Computer Science and Business & Economics. However, in this study we only used one categorisation per primary study. It was decided based on the primary categorisation of the paper that appear in the list of research areas. For example, papers published in the proceedings of International Conference on Software Business are categorised into Computer Science and Business & Economics, but in this study they are categorised only into the Computer Science discipline. AIS is a special digital library for information systems, thus all papers retrieved from this digital library are categorised into the Information Systems discipline. In addition, the contributing discipline was decided by also taking into consideration the affiliation (e.g., department, school, and university) of the contributing authors of the publications. In most cases, an author’s discipline corresponded with their school or department affiliation.

3.3.2 Research Method.

The research methods listed in [6] were used as the basis for this category. The listed research methods are as follows: Controlled Experiments, Case Studies, Survey Research, Action Research, and Ethnographies. If a research method did not fit into any of these categories, we extracted the research method as it was reported. The categorisation of the research methods was performed by examining the abstract and research methodology section for each paper. Similarly, we also extracted the theory that was explicitly mentioned in the paper.

3.3.3 Research Topic.

The paper by Unterkalmsteiner et al. (2016) [19] was used as the basis for this category. The six research topics are as follows:

  1. 1.

    Supporting Startup Engineering Activities deals with papers that supports software engineering activities, for instance, Test Driven Development.

  2. 2.

    Startup Evolution Models and Patterns focuses on the progression of software startups over time.

  3. 3.

    Human Aspects in Software Startups covers research that investigates factors related to the actors involved in software startups.

  4. 4.

    Applying Startup Concepts in Non-Startup Environments investigates the effect of applying successful software startup practices in traditional environments.

  5. 5.

    Startup Ecosystems and Innovation hubs examines how supportive and thriving environments for software startups can be designed.

  6. 6.

    Theory and Methodologies for Software Startup Research covers research that develops methodologies and theories for software startup research.

The categorisation of the research topics was determined by primarily inspecting the keywords, abstract, and conclusion section of the paper. If the topic of the paper remained unclear, a deeper examination of the paper was carried out until it could be categorised.

3.4 Data Extraction and Mapping Process

The categorisation process was carried out by splitting up the research papers, such that each reviewer was given a part to categorise. For each paper a confidence level was included along with the category. A high confidence level indicated how certain the reviewer was of the categorisation. For instance if the paper contained the phrase “...in this study we performed a case study” it would subsequently be added to the Case Study-category, and given a high confidence level. Papers marked with low confidence level required an additional reviewer to read the paper and categorise it. Any ambiguities were discussed and addressed with the present of the reviewers. Excel was used to further structure the extracted data, allowing the data to fit the classification schemes, and finally the data was visualised using a histogram and a three faceted bubble plot.

4 Findings

This section details the evolution of software startup research uncovered from the SMS. This section reveals the distribution of publication per year, what the contributing disciplines are, and their preferred research methods, theories, and research topics. All of this information is captured in a histogram and a three faceted bubble plot.

4.1 Publication Sources and Years

Footnote 1

The distribution of the years and venues of the primary studies is shown in Fig. 1. The first research paper in software startups in this study was published in the year 2000. There were few research papers published on software startups in this period until 2015. The year 2015 saw the first significant peak of publications in software startup with 6 papers. Since 2015, the number of research papers published has approximately doubled every other year, peaking in 2019 at 25 papers. However, a sudden decline in the number of publications is observed from 2020 to 2022.

Fig. 1.
figure 1

Temporal distribution of primary studies.

Full size image

Most primary studies (68%) are published in computer science related conferences, e.g. International Conference on Software Business (ICSOB), XP Conference, Euromicro Software Engineering and Advanced Application (SEAA) and Product-Focused Software Process Improvement (PROFES). In the Information Systems-discipline, the majority primary studies are published in the International Conference on Information Systems.

4.2 RQ1 - The Contributing Disciplines in Software Startup Research

Figure 2 shows a three faceted bubble plot with the mapping of the contributing disciplines, research methods, and topics. The larger the point radius, the greater number of research papers published in that category.

Our results identified 4 disciplines that were responsible of contributing the 95 primary studies. The contributing disciplines to the software startup research area are, in the descending order, Computer Science (58 papers), Information Systems (16 papers), Business and Economics (13 papers) and Engineering (8 papers).

The earliest publication in the software startup research area uncovered in this study came from the Engineering discipline (Engineering Management Society Conference) back in 2000, followed by the Computer Science discipline in 2002 (IEEE Software) and the Business and Economics discipline in 2003 (Systems Dynamic Review). This indicates the closely tight aspects of engineering and business in the software startup research phenomenon. The first paper published in the Information Systems discipline was in 2013 (Asian Conference on Information Systems). As software startups grow and mature, the evolution models and patterns are becoming evident and may attract Information Systems researchers.

Examining the affiliation of the contributing authors, majority of the papers are written by authors with similar affiliations and/or disciplines. For example, publications in the Information Systems discipline are typically written by researchers affiliated with School of Business, Economics and Management. Only four studies are inter-disciplinary: Computer Science and Business [2], Computer Science and Engineering [10], Engineering and Economics [17], Engineering and Administration [16].

4.3 RQ2 - Research Methods and Theories Used in Software Startup Research

Figure 2 illustrates the distribution of research methods used in the software startup research area. This study identified a total of 13 research methods for studying the software startup phenomenon. Qualitative approach is the most common research approach, as the majority of the primary studies focus on qualitative properties of the software startup phenomenon, e.g., engineering activities, evolution process, interaction process with ecosystems, etc. However, we have found two studies use statistical analysis to establish causal relation between different factors e.g., AI or ecosystem infrastructure to the growth of software startups. A total of 52 research papers employed Case Studies as their primary research method, followed by Action Research method (9 studies). We have also seen research papers use methods such as Design Science Research and Delphy Study to investigate the software startup phenomenon.

Fig. 2.
figure 2

The mapping of contributing disciplines, research methods, and topics.

Full size image

Out of 95 primary studies, only 7 studies used theories to explain how and why some phenomenon occurred, e.g., causation and effectuation theory (3 papers), behavioural theory (2 papers), socio-technical theory (1), structuration theory (1 paper). These studies are published in all disciplines, except Engineering.

4.4 RQ3 - The Past, Present, and Emerging Topics in Software Startup Research

The distribution of past, present, and emerging topics in the software startup research area is shown in Fig. 2. Among the six major clusters of software startup research agenda as discussed in [19], our study found Supporting Startup Engineering Activities and Software Evolution and Patterns to be the most researched areas with 31 and 30 papers each. Research on Supporting Startup Engineering Activities addresses specific software engineering challenges encountered by software startups. Computer Science is the largest contributors with 24 papers in Supporting Startup Engineering Activities as illustrated in Fig. 2. Surprisingly, the Business and Economics discipline also contributes 4 papers in this area, compared to Engineering with 3 papers. Startup Evolution Models and Patterns focuses on the progression of software startups over time to understand the underlying factors that contribute to success or failure of software startups. Computer Science and Information Systems discipline are the largest contributors with 13 papers each in Startup Evolution Models and Patterns as seen in Fig. 2.

We have found two new research clusters emerged from the primary studies, which are Software Startup Education (9 papers) and Ethics in Software Startups (1 paper) since 2015. Software Startup Education focuses on pedagogical activities on teaching software startups approaches, for instance the Lean Startup Approach using a development of challenges-based framework [4] or a Massive Open Online Course (MOOC) [21]. Ethics in Software Startups examines ethics in a software startup environment, for instance best practices when developing AI software [20].

Within the Supporting Startup Engineering Activities research cluster, we identified a new emerging topic Experimentation in Software Startups (2 papers). This research topic focuses on identifying and evaluating processes, methods, and tools to conduct experiment to validate products and customer related assumptions or hypotheses. One study within the Cooperative and Human Aspects in Software Startup research cluster investigates learning in software startups at both individual and team level. Finally, one study within the Startup Evolution Models and Patterns research cluster investigate the process, methods and tools to support decision making in software startups.

5 Discussion

Our findings show that the software startup research area started to take off in 2015, doubling the number of publications every two years until the COVID-19 pandemic hit. The effects of the COVID-19 pandemic, which has led to the shutdown of universities and research institutions worldwide, can potentially be the cause for the sudden drop in publications. The shutdown of research institutes may have hindered research, as well as travel restrictions, infection risks, and other precautions taken by the authorities, may have led to the cancellation of several conferences [1]. However, we can expect to see the number bounce up, as we are at the beginning of 2022 and the COVID-19 restrictions worldwide are gradually lifted.

This study used an expanded version of Wang’s search string, which included additional terms, and it was therefore anticipated that a greater number of publications would be discovered. However, this study mapped 95 primary studies using the WoS and AIS digital libraries compared to Wang [22], where 133 publications were identified published and indexed in the Scopus digital library, in which 35 overlapped with our study. A reason for the smaller number of publications included in this study compared to Wang’s can be attributed to the stricter selection criteria as this study only included publications that report empirical work as primary studies, additionally to using different digital libraries.

It is important to note that the two disciplines that contributed the most Computer Science and Information Systems focus on different research clusters. For example, the Computer Science discipline emphasises on Supporting Startup Engineering Activities, while the Information Systems discipline emphasises on Startup Evolution Models and Patterns. There are often perceptions that Computer Science research focuses on technical issues while Information Systems emphasises on the behavioral and social implications of technology. Our analysis on the research topic support that this may be the case.

With the complexity of the scientific system, many social and engineering problems cannot be solved by one discipline, and thus interdisciplinary research has become an indispensable model of modern science [14]. However, this is not the case in the software startup research area. Contrary with the findings reported by Wang [22], our results did not show significant evidence in increasing inter-disciplinary studies. Most studies are conducted by researchers within the same disciplines and only 4% inter-disciplinary studies were identified in this study.

A further analysis of the primary studies allowed us to update the research agenda in [19]. As discussed in Sect. 4, two new research clusters emerged: Software Startup Education and Ethics in Software Startups, as illustrated in Fig. 3. We discuss these new research clusters and list some examples of relevant research questions in the next section. These research questions draw on the findings, analysis, and limitations of the primary studies.

Fig. 3.
figure 3

(adapted from [19]).

Updated Overview of the Software Startup Research Agenda

Full size image

5.1 Software Startup Education

The importance to provide entrepreneurial skills to engineering students has been widely recognised [15]. Technical proficiency is required but may not be enough for students, when they land their first job or seek to launch a software startup. In addition, the high failure rate of software startups is mainly due to software engineering practices [12].

The phenomenon has attracted both industry and academic community, and several attempts have been made to address the issues startups are facing. For instance, by offering entrepreneurship-focused programs and courses to better prepare students to create a startup or to be employed in a startup environment. However, there are not many studies that focused on how software startup processes based on, for instance, Lean startup approach is taught to computer science and engineering students [4]. This new research cluster investigates pedagogical approaches, frameworks, and tools to teach software startup in both a formal setting, for example at a university, as well as an informal setting, such as a MOOC.

5.1.1 Formal Software Startup Education.

Many universities now offer entrepreneurship courses to computer science students or students with similar backgrounds. This sub-cluster includes anything that involves improving existing courses or testing new courses in an effort to improve their software startup education in a formal setting. The idea is to equip students with the necessary skills to become value creators early in their careers. Examples of relevant research questions regarding this research track can be formulated as follows:

  • RQ1 – How can we bridge the gap between the classroom and the real world?

  • RQ2 – What are the most effective approaches for teaching software startup?

  • RQ3 – How can we encourage more students to launch their own software startup company?

5.1.2 Informal Software Startup Education.

Many new software startup practitioners have emerged as a result of the ever-increasing availability of free to low-cost educational resources, such as MOOCs and other online self-learning resources. How do these practitioners compare to others who have a more formal education, such as from a university? A common problem among self-learners is the low completion rate in online courses, for instance, It is 15% or less in MOOC Certificate programs [8]. Therefore, it is necessary to investigate how these self-learners traverse the wealth of online resources, select what is relevant to learn, ensure the quality of what they are learning, and maintain discipline to follow through on their plans. In broad terms this sub-cluster tries to answer the research questions such as:

  • RQ1 – How can self-learners tell the difference between bad and good learning sources?

  • RQ2 – How do successful self-learners stay motivated and disciplined?

  • RQ3 – Are there differences between those who have received informal software startup education and those who have received formal software startup education?

5.2 Ethics in Software Startups

The use of cutting-edge technologies such as AI, IoT, blockchain etc. have been common practice in software startups as part of their business models development [23]. For instance, the database CrunchbaseFootnote 2 lists over 79,000 startups related to AI as of February 2022. While these technologies can be seen as enablers to promote entrepreneurship and work well at the technical level, they also come with socio-cultural issues, that are rooted in ethics [20]. This new research cluster examines the role of ethics in software startups in terms of software development, organisational policies, and business practices.

5.2.1 Software Development Ethics in Startups.

This sub-cluster encompasses questions regarding the ethics in relation to how software development is conducted in startups. For instance, how privacy is handled, licensing, and the creation of blackbox systems using AI. In order to better understand and explore the role of ethics in the software development in a startup context, examples of research questions such as the following can be formulated:

  • RQ1 – What are the potential ethical challenges in software development in startups?

  • RQ2 – How do software startups follow best practices when it comes to protecting their clients’ privacy?

  • RQ3 – What are the ethical implications of the development of AI in software startups?

5.2.2 Organisational and Business Ethics in Software Startups.

This sub-cluster is responsible for all aspects of organizational and business ethics. All organizational practices and policies for ensuring a healthy and ethical work environment is covered. These practices and policies could for example be the code of conduct or more specific challenges, such as outsourcing, and the negotiation of salary. The following are examples of research questions for organizational and business ethics in software startups:

  • RQ1 – What organisational regulations do software startups need, to establish an ethical culture?

  • RQ2 – What business regulations do software startups need, to establish an ethical business culture?

  • RQ3 – How can software startups ensure that salary negotiations are ethical and fair?

  • RQ4 – What makes software startups outsource part of their work?

5.3 Threats to Validity

In this subsection, we identify and discuss the threats to validity of this study.

5.3.1 Research Methods that are Not Clearly Stated.

Most primary studies specifically stated the research method used, but a small number did not. Therefore, we sometimes had to derive the research method by reading the paper. Papers that were hard to categorise had more than one reviewer to reduce the threat to validity. Ambiguities in the papers were discussed before the final categorisation.

5.3.2 The Chosen Digital Libraries.

WoS assigns every research paper in their collection to multiple categories. In such case, we took into consideration the first keyword appeared in the category defined by the digital libraries and the affiliation of the authors. Similar to Wang [22], we also used the category given by the digital library. However, WoS and Scopus may not use the same categorisation scheme, which makes it difficult to compare papers. For instance, Scopus has a Business, Management and Accounting category while WoS has a Business & Economics. AIS is known as the digital library for the Information Systems discipline. Thus, the classification was done straightforward.

By including only two digital libraries, there are still many papers that are yet to be mapped, and this may be the reason why we found only one paper on Ethics in Software Startups.

5.3.3 Selection Process.

It is probable that some primary studies have been excluded from this study, because they were wrongly deemed irrelevant in relation to software startup research. This study used synonyms for “software startup”, such as “internet startup” and “digital entrepreneurship” as part of the search string. However, not everyone uses these terms the same way, and therefore made it harder to distinguish relevant papers in relation to software startup research. To minimise this threat, the reviewers discussed ambiguous papers until a consensus was made to include the paper or not.

6 Conclusion

Software startup is a growing research area. Due to its product-business nature, research in software startups has attracted academics from multiple disciplines. In this study, we identified four disciplines that give significant contributions to establish software startup as a research area. Computer Science and Information Systems are the two top contributors. Our study also revealed that the Case Study is the most common research method to investigate the software startup phenomenon. In terms of research topic, our study found that Supporting Startup Engineering Activities has received the largest interest since 2014. Finally, this paper also identified two emerging research topics: Software Startup Education and Ethics in Software Startups.

In terms of implications, this paper makes two contributions. The first contribution is to provide the map of contributing disciplines, research methods, and topics in the software startup research area. Moreover, our study identified the emergence of two research topics since 2015, which makes our second contribution. The findings of our study allow us to update the research agenda paper on software startup [19] 6 years after its publication in 2016. For researchers, our findings may be used to inform and navigate their current and future research. Moreover, our study calls for more empirical research in particular topics including Software Startup Education and Ethics in Software Startups. To help achieve this goal, we have provided examples of research questions within these topics.

In terms of limitations, the findings of our study are limited due to the fact that we only used two digital libraries, Clarivate Web of Science and AIS e-Library. This may also be the reason why we only got one paper on Ethics in Software Startups. There is a high probability that we did not cover all important and relevant studies, as they are not indexed by the two libraries. Future research could add different digital libraries for example Inspec and Compendex or EBSCOHost, which might give better coverage to relevant literature. In addition, future research could also examine grey literature to confirm or refute if the identified research topics are relevant for actual software startups. Future research could replicate the study by [15] to measure the trend in inter-disciplinary research in the software startup research area.