1 Introduction

The information technology (IT) world is shifting into a new era whereby cloud computing is becoming widespread. Cloud computing is a service delivery model within which computing resources are hosted, run and managed through virtual web data centres that can be accessed on a self-service basis or with little help from the provider (Mell and Grance 2011). A growing number of IT professionals are considering the feasibility of public, private and hybrid clouds for running business applications and services. The basic motivation is to attain cost-effective solutions. There is a rise in pilot projects aimed at evaluating the applicability of the cloud within organizations (Nordic council of ministers 2012). As many IT professionals are looking to enjoy the benefits of the cloud, such as reduced upfront costs and virtualization capabilities that enhance the delivery of centralized services, there is a demand for more easily accessible, secure and reliable applications.

Cloud computing research has mainly focused on understanding and defining cloud computing, the benefits and challenges of cloud computing and the implementation of cloud computing applications and systems (Armbrust et al. 2010; Buyya et al. 2009; Marston et al. 2011). Other researchers have taken their interests further to study cloud computing in different contexts such as small and medium sized enterprises (SMEs) (Sultan 2011) and the gaming industry (Ojala and Tyrväinen 2011). Security is one of the highly valued requirements for effective cloud computing, has often been cited as a risk and has been addressed by previous studies. For example, Zissis and Lekkas (2010) have evaluated cloud security issues and proposed a cryptography-based solution that helps to ensure that the security requirements are met. Technical cloud security issues have been evaluated by Jensen et al. (2009), and positive views on cloud security are discussed by Wilson (2011).

In the context of testing, cloud computing has been said to “offer resources such as virtualized hardware, effectively unlimited storage and software services that can aid in reducing the execution time of large test suites in a cost-effective manner” (Parveen and Tilley 2010). Yu et al. (2010) demonstrate by using an elastic cloud-based testing as a service platform to automatically cluster, schedule, monitor and manage simultaneous testing jobs. Some researchers have taken advantage of virtualization technology to show that the cloud can be used to support large-scale testing of different interdependent systems (Hanawa et al. 2010) and automatically run and manage testing tasks (Gaisbauer et al. 2008). Additionally, the cloud can support the building of on-demand test laboratories as is the case of Liu and Orban (2009).

The above studies focus on utilizing cloud-based resources for testing different applications and systems and consequently exhibit the potential possibilities of using the cloud for testing. Although there is a growing body of knowledge related to cloud-based testing, there is a shortage of empirical studies involving industrial practitioners. There is a need to observe real-life industrial experiences in order to develop a better understanding of the applicability of cloud-based testing. Given that cloud computing is adopted for different purposes, investigating practitioners’ real-life experiences can generate valuable knowledge which will be useful to both academia and industry. Therefore, in this study, we focus on various aspects related to the adoption and use of cloud-based testing in different organizational contexts by addressing these research questions: (1) What are the motivating factors that cause organizations to adopt cloud-based testing? (2) How is the cloud used to test applications, services and systems? In the context of this study, cloud-based testing entails the use of computing resources, environments and infrastructure hosted in the cloud to perform testing.

The rest of the paper is organized as follows: Sect. 2 presents the related work, including our earlier studies. Section 3 describes the research methodology used in this study. Section 4 presents the findings of the study. Section 5 discusses the results and their implications as well as the limitations of the study. The paper concludes in Sect. 6.

2 Related work

Various studies have explored different aspects related to cloud-based testing. The range of cloud-based testing services provides solutions for different needs such as on-demand testing, test environment management and test simulation (Gao et al. 2013). Robinson and Ragusa (2011) emphasize the business goals related to testing when developing the requirements for an effective cloud-based testing infrastructure. With cost reduction as a primary business goal, they identify seven vital criteria for infrastructure and test management—cost-effectiveness, simplicity, target representation, controllability, observability, predictability and reproducibility. They also name five patterns for performing cloud-based testing (Robinson and Ragusa 2011):

  1. 1.

    Testing in the cloud: A simple model of using a single-cloud infrastructure provider for hosting the software under test and test suite.

  2. 2.

    System under test in the cloud: The tester places the system under test on a cloud provider’s infrastructure but hosts the test suite locally. The test payloads are sent over the Internet to the remote system under test. The software under test could be either owned by the tester or a software-as-a-service (SaaS) solution under test.

  3. 3.

    Test suite in the cloud: Providers of testing software exist in the cloud such that the test suite is hosted and controlled outside the domain of the system under test.

  4. 4.

    Multi-site testing in the cloud: More than one provider is used for distributing both the system under test and the test suite in different domains.

  5. 5.

    Brokered multi-site testing in the cloud: The same as pattern IV with the inclusion of an intermediary for brokering test and infrastructure management requests and operations. The broker also provides support for resources selection, reservation and deployment.

Mohammed and Mcheick (2011) attempted to understand how cloud services themselves can be tested in order to protect the cloud provider and developer. They presented important questions, such as how to thoroughly test a service against its security policies and how to implement appropriate measures for ensuring test coverage. In addition, Jun and Meng (2011) outlined two ways of provisioning for cloud-based testing services—accessing cloud testing services via the web and by use of virtual machines, noting a definite distinction between the two. In the former, an organization accesses the service through the cloud testing service provider’s webpage. In the latter, the “cloud testing service providers provide the internet protocol (IP) of the virtual machine and clients connect to the virtual machine remotely” (Jun and Meng 2011).

Various studies have demonstrated methods of performing cloud-based testing (Candea et al. 2010; Ciortea et al. 2010; Ganon and Zilbershtein 2009; Jenkins et al. 2011; Koong et al. 2013; Oriol and Ullah 2010; Wu et al. 2011). Ganon and Zilberstein (2009) took advantage of the cloud’s scalable and affordable features to test the network management system (NMS) for a voice over IP (VoIP) telephony system. They were able to improve the quality of the software at a relatively low cost. In another study, Ciortea et al. (2010) developed a cloud-based testing framework called Cloud9. It supports two functionalities. Parallel symbolic execution of cloud-based computer clusters was observed to reduce the amount of testing time. Automated testing allows developers to perform tests, and allows end users to choose and evaluate the software they need (Candea et al. 2010). The York Extensible Testing Infrastructure (YETI) is an automated testing tool that provides an environment for testing a variety of different programmes. YETI can share the jobs across multiple machines, thus resulting in improved performance (Oriol and Ullah 2010).

Wu et al. (2011) propose an interesting cloud-based testing framework. They first outline eight key features of cloud testing: resource on-demand, infrastructure reliability, load scalability, behaviour representativeness, environment diversity, platform diversity, service transparency and data confidentiality. They then take advantage of the software-as-a-service (SaaS) and humans-as-a-service (HuaaS) models of cloud computing to introduce a hybrid platform for performance and compatibility testing of web applications. Cloud infrastructures and platforms can also be tested by small, independent programmes operating as plug-ins. The plug-ins work by invoking an application programming interface (API) and testing its capabilities (Jenkins et al. 2011).

On the use of crowdsourced testing, a practical example is that of uTest, which makes use of its pool of testers to provide testing services to its customers. uTest’s range of testing services includes functional, security, load, localization and usability testing (uTest 2012). Microsoft’s Security Essentials team is one case example demonstrating the application of crowdsourcing for functional testing (uTest 2012). The tests were performed in four test cycles. The first one was exploratory in nature, where globally dispersed testers were able to try out Microsoft Security Essentials and give feedback on any arising issues. During the second and third cycles, the testers ran Microsoft’s test scripts to perform installations and upgrades and provided feedback on generated bugs and other problems. The bugs were then approved and rejected according to different criteria as determined by the Microsoft’s testing manager and uTest’s project manager. The last cycle dealt with regression testing, where a specified group of uTest testers were used to check the backend changes made by the Microsoft Security Essentials team, so as to make sure that no new errors had been introduced in the process. This exercise is reported to have helped Microsoft to increase their test coverage (uTest, 2012).

2.1 Exploratory studies

Most of the related works discussed above are experimental in nature, and they relate to cloud-based testing from different perspectives. However, the suitability of cloud-based testing in enterprise environments is not well understood and is a challenge—organizations are not convinced about the impacts of cloud-based testing on their businesses and ways of working. To address the knowledge gap and empirical study shortage, we carried out two exploratory studies (Riungu et al. 2010; Riungu-Kalliosaari et al. 2012a) that focused on gathering and analysing views from industrial practitioners. In one study, we sought to understand the conditions that influence software testing as an online service (Riungu-Kalliosaari et al. 2012a). A pioneering conclusion was that software testing would be an online service delivered through the use of cloud-based resources. In the other study (Riungu et al. 2010), we investigated the issues which practitioners faced when carrying out software testing in the cloud. These included issues such as how to handle the test data, how to assign a sufficient number of testers and how testing in the cloud affects business outcomes. This paper also presented an initial conceptual overview of the facets of testing in the cloud encompassing testing of Software as a Service (SaaS) and non-SaaS software, testing resources hosted in the cloud and testing of the cloud itself (Riungu et al. 2010).

In the study (Riungu-Kalliosaari et al. 2012b), we identified the dynamics of testing in the cloud. By analysing the results from industry practitioners who used the cloud for testing purposes, we observed that cloud-based testing helps in improving performance testing, enables quicker testing and supports organizations to achieve more realistic testing results. Furthermore, cloud-based testing improves the availability of tools used for testing, enhances communication between developers and testers and supports better delivery of services. A road map towards cloud-based testing was also presented.

Following these preceding studies, we now observe why and how cloud-based resources are utilized to perform testing tasks within various organizations. We also consider the additional value that organizations might obtain as a result of cloud-based testing. The additional value is described in terms of value-added features of the cloud-based testing resources and the system under test.

3 Research process

Qualitative research methods can be used to explore areas that have previously not been studied or areas of which a lot is known with the aim of obtaining new findings (Strauss and Corbin 1990). Qualitative research methods aid in obtaining an in-depth understanding of the context being studied. Qualitative research methods generate a lot of data, which the researcher must carefully crystallize to address the research question with answers that are well grounded in the collected data. We adopted an exploratory qualitative approach, using the grounded theory method. We used grounded theory because it suits for research fields which have not been widely explored and it provides the opportunity to generate new theories and concepts (Seaman 1999). This study was conducted in the real-world context and investigated the adoption and use of cloud-based testing as described by the interview participants.

3.1 Data collection

We collected the data using semi-structured, theme-based interviews. Semi-structured interviews are suitable for qualitative studies because they provide for the interviewer the freedom to delve further into new trails of questions whenever unexpected information emerges in the course of the interview (Seaman 1999). We carried out three interview rounds to collect the data for this study. In preparation of the first interview round, one researcher developed the initial set of questions. The questions were generated after a preliminary literature review. The questions were later reviewed by the rest of the research team. In addition, the questions were also reviewed by a practitioner from a testing organization who did not participate in the interviews. Corrections were made accordingly, and the final set of questions was achieved. One of the researchers then ran a test interview with a practitioner from another software organization (other than the aforementioned testing organization). This helped to ensure the validity of the questions and to estimate the length of the interview.

Three researchers prepared the questions for the second interview round, receiving feedback from the remaining members of the research team. Emerging ideas for questions and themes generated during the analysis of a preceding interview round were incorporated into the next interview round. For the third interview round, two researchers prepared the questions, and the rest of the team gave feedback. Having multiple researchers in preparing the questions ensured sufficient coverage of the topics of interest. To ensure that the interviews would keep to the recommended limit (1 h), we relied on our previous interviewing experiences and we also compared the number of questions with that of interviews conducted earlier. The questions and themes of the interviews are available at http://www2.it.lut.fi/project/MASTO/material.html.

The majority of the interviews were carried out by one interviewer with one respondent, but there were six occasions with two respondents. We selected the organizations based on our existing contacts, theoretical sampling and snowballing. In theoretical sampling, the emerging theoretical understanding is allowed to influence the themes and sources for subsequent interviews (Strauss and Corbin 1990). This is vital especially when studying new areas mainly because it directs the researcher to select sampling instances that contribute richly to the theory formation (Strauss and Corbin 1990). Theoretical sampling enabled us to solicit views from real-life experiences of participants in organizations that were using cloud computing. We also used snowball sampling during the first and third interview rounds, whereby an interviewee recommended a representative from another organization as a suitable respondent.

When we contacted the organizations, our basic request was to interview individuals that were willing and able to discuss the themes of the interviews. The decision to participate or not and the choice of the particular interviewee were entirely up to the organization’s discretion. As detailed in Table 1, four of the organizations participated in all three interview rounds, seven participated in two of the interview rounds, and the remaining nine participated in one interview round. When an organization participated in several rounds, normally the same respondent represented the company. However, there were situations where different individuals took part in different rounds and this is indicated in brackets next to the interviewee role on the fourth column, e.g. one person from organization 4 took part in all 3 interview rounds, and each of the other two took part in one interview round. Overall, the respondents’ roles were mainly at managerial level, because they understood both the business and technical aspects related cloud-based testing within their specific organizations.

Table 1 Details of organizations and interviewee roles
Full size table

The themes of the interviews helped to provide a basis for the all the responses, regardless of how many rounds in which an organization participated, or whether the same person was representing the organization in more than one round. Due to the nature of participation, i.e. on a voluntary basis, we believe that the responses provided by the interviewed individuals were representative of the practice of cloud-based testing.

Organizations dealing with mission critical systems (organizations 8, 10, and 11) declined to participate in the first interview round, but became interested during the second and third interview rounds. With one exception, all interviews were held face-to-face at the respondents’ work locations. The exception was through email, because that particular respondent was widely on the move, and it was difficult to set a suitable time for a face-to-face interview. The interviews were recorded after which they were transcribed for analysis. The transcribed text generated a total of 445 standard A4 pages.

3.2 Data analysis

Qualitative data analysis is an iterative process which can be supported by the use of software tools, especially when there is a large volume of data. We used the qualitative data analysis software Atlas.ti, which is well suited to grounded theory studies, providing a compact way of handling the interview transcripts, allowing for interpretation of the data through linking, searching and sorting.

For our analysis, we used the coding procedures within the grounded theory method (Strauss and Corbin 1990). These include open coding, where concepts are classified according to their properties and dimensions to form categories; axial coding, where the identified properties and features are used to establish relationships among categories; and selective coding, where the categories are combined to build the theory (Strauss and Corbin 1990). We wrote memos whenever it was necessary to do so during the analysis process. Memos can appear in various forms such as statements, hypotheses or questions and become particularly useful in the process of crystallizing the theory from the emerging concepts (Coleman and O’Connor 2007).

3.2.1 Open coding

Open coding aids in revealing the underlying meanings, ideas and thoughts within the data. The identified codes are then consolidated to form categories which contain different properties and their dimensions. We began by reading all the interview transcripts and assigning codes to words, sentences or paragraphs. We assigned the codes according to meanings reflected in the text by asking: “What is the interviewee talking about?” Then we refined the codes by comparing them with each other, merging similar codes, making modifications, e.g. changing a code’s name, adding new codes, and deleting some of the codes, e.g. repetitive ones.

We then grouped relevant codes together to form categories. The seed categories can be derived from the goals of the study, the research questions and predefined variables of interest (Strauss and Corbin 1990). We deduced the initial set of categories from the areas of interest reflected by the interview questions. We were interested particularly in cloud-based testing, but the interviewees also discussed cloud computing in general. Therefore, when forming the categories, we had to be careful to identify separate codes relating to cloud-based testing and to cloud computing. This ensured that cloud computing-based responses did not overshadow those specific to cloud-based testing.

For example, we attached the code “cloud computing adoption” to the following interview quotation: “…small to medium sized companies have basically just realized that instead of buying a hardware rack, they will just get the stuff from Amazon and that’s it. So they use it opportunistically, without any strategic thought that, OK we’ll go cloud. They just wanted services, adopted some from Amazon, found that stuff works, and adopted [them for] some work” (CTO, Org. 16). We attached a memo to the code stating “Tactical (opportunistic) vs strategic approach”. While the above quotation was appropriate in the context of cloud computing, it did not reflect the adoption of cloud computing for testing. However, it provided a lead into examining the data for ways demonstrating how the organizations had adopted cloud-based testing. This gave rise to “organizational dynamics” (under subsection 4.2) in relation to testing resources.

3.2.2 Axial coding

There were several iterations between open and axial coding during which we developed the categories further by eliminating redundancies and merging them to form new ones whenever necessary. The main focus of axial coding was to identify the relationships between the categories. The identified relationships are based on similarities, associations and differences between the categories. For example, we classified “organizational dynamics” together with “need for testing resources”, “benefits of cloud-based testing resources” and “awareness of cloud-based testing resources” in one category named “testing resources”. We performed this grouping based on the fact that the concepts described aspects which related to how organizations addressed their need for more testing resources. During axial coding, we grouped the categories identified during open coding into four main categories. The grouping or classification performed in axial coding aims at developing the categories until when new information does not add any new meaning. This implies that “theoretical saturation” has been reached, and there is no need to continue with further category development (Strauss and Corbin 1990).

3.2.3 Selective coding

Selective coding is the last step of the analysis process in which the theory is integrated and refined (Strauss and Corbin 1990). The theory is characterized by a central category that is related to all other categories. The relationships between the categories take the shape of a factor model, which focuses on cause and effect such that the core categories reflect “antecedent conditions, influences on and consequences of the construct being explored; and the relationships may indicate causality, association, process-sequence, or any pattern that the researcher finds useful” (Gasson 2004).

The three coding steps resulted in the extraction of four main categories: testing resources, utilization of cloud-based testing resources, quality of cloud-based testing environments and quality of the system under test. Due to the prevalent need for testing resources, the category “testing resources” initially appeared to be a strong candidate as a core category. However, during the process of crystallizing the theory, it emerged that the need for testing resources represented “antecedent conditions” occurring before “utilization of cloud-based testing resources”—which eventually became the core category. In addition, the identified core category was connected to the other categories. We present the formulated theory at the end of the next section after describing the categories.

3.3 Development of the categories

During the analysis, we followed Strauss and Corbin’s (1990) recommendation which speaks of identifying the variety of conditions, actions/interactions, and consequences associated with a phenomenon. The phenomenon of the study was cloud-based testing. We asked questions such as what, why, how, where, when, how come, and with what results which led to four categories covering the phenomenon. Considering these questions helps to systematically identify categories and establish relationships between them. We identified a set of properties for each of the categories. The developed categories and their properties are inclined to the questions what (is going on here), why (is it happening), how (is it happening) and what results (does the phenomenon produce)? (Table 2).

Table 2 Categories and properties
Full size table

4 Findings

We identified four categories, with which we were able to explain the adoption and use of cloud-based testing in practice: testing resources, utilization of cloud-based testing resources, value-added features of cloud-based testing resources and value-added features of the system under test. This section explains the four categories and their properties in detail. The selective coding of grounded theory targets at a coherent theoretical explanation of the core category, answering to the question “What is going on here?” We selected “utilization of cloud-based testing resources” as the core category and in the following we focus our description on it. We characterize the core category with five properties, each of which describes how cloud-based testing resources are put to use after they have been adopted within the organizations (Table 2).

The category “testing resources” contains four properties describing the motivations and drivers for the adoption of cloud-based testing in organizations. Thus, it provides answers to the question why is “utilization of cloud-based testing resources” happening here? Therefore, it acts as a condition to and forms an antecedent relationship with the main category.

The third and fourth categories, containing three and two properties, respectively, describe the value propositions created when utilizing cloud-based testing resources. Hence, they provide answers to the question what results does “utilization of cloud-based testing resources” produce? and are therefore the consequences of the core category. In the following, we explain each of the categories and its properties.

4.1 Utilization of cloud-based testing resources

In order to understand the applicability of cloud computing for testing purposes, we solicited real-life experiences from organizations that were actually using cloud-based testing. This category is characterized by several properties describing how cloud-based testing resources were applied in practice. There are several ways through which cloud-based resources were utilized for testing. These are, for example, infrastructure for performance testing, multi-platform testing, applying iterative development with users and crowdsourced testing.

4.1.1 Infrastructure for performance testing

The infrastructure required for performance testing is usually expensive and incurs further maintenance costs. Cloud-based servers can be used, commissioned and decommissioned according to the need. The system architect from organization 18 gave an example of one of their customers that “built a separate testing platform in the Azure environment so that they were testing their cloud solution from [the] cloud. Because that way they could get enough load generated without paying for hundreds of test servers”. Another interviewee demonstrated cloud-based performance testing as follows:

You just cannot get 40 high-end, expensive servers for use in the traditional manner. There’s no way to do that. So that’s clearly an area where using cloud computing enables you to clearly do some kind of testing that is not possible at all otherwise. For example, I did some performance testing where I had 40 Amazon instances running for generating something like 20,000 realistic user experiences. They were actually running [a] full embedded browser suite, so it was not a simple HTML-HTTP request testing, but [used] some fully flexed browsers with DOM and JavaScript support and test[ed] a rich Internet application. (Chief technology officer, Org. 16).

Organizations consider it better to be able to test in environments that mirror the system’s or application’s ability to cope with changing resources. Similar to their views about performance testing, the interviewees also discussed about using the cloud-based resources to test for scalability.

…it’s easy to test, for example, how [new software] reacts if the user base increases fast. (Owner, Org. 15)

4.1.2 Multiplatform testing

The cloud hosts a variety of different platforms and operating systems that can be provisioned on demand for testing purposes. Organizations can acquire different platforms or operating systems from the cloud and test their applications as deemed appropriate.

…the cloud computing platforms make it really easy to test run all kinds of new services and software, because you can easily launch different kind of open source stacks and even license, like Windows Server 2003, 2008. It takes practically no money and no time to launch and install testing software on these servers and run the tests. I mean, it’s easy to test new software on different kind of platforms. (Owner, Org. 15)

4.1.3 Applying iterative development and testing with users

The interviewees gave hints of fast iterative development cycles: development using the cloud or even developing cloud applications follows an iterative development approach. The developed features can be released, and users can begin to use them, after which they can report the bugs. The bugs can be fixed and the application is updated. As mentioned by one interviewee, this iterative approach to developing and testing may help to produce better quality in the software:

With testing done in real production environments with real data, and real users …if people are willing to test new versions in production, then we get really good comments and feedback. And of course, if we achieve community goals, such as getting tax officials and accounting offices to talk to each other, then we get good partnership with the tax agency, so that we improve old processes to make better ones. If we can group all these parties together, we can maybe build something totally different and better than before. (Software manager, Org. 1)

4.1.4 Infrastructure for testing CPU-intensive tasks

The interviewees discussed the possibility of using cloud environments to process CPU-intensive tasks with large data sets. Examples of such tasks are those within scientific research and naval systems. Cloud-based testing infrastructure was deemed appropriate especially in situations where the tasks are performed on occasion, therefore, no need to invest in the infrastructure internally.

Probably, the first would be to use it [cloud computing] in our internal work. We have some calculations that need a lot of CPU time and stuff like some [work] very close to testing, and [requires] very fine calculations. It’s something that we don’t do on regular basis, so cloud computing or the Amazon cloud has been identified as a possibility to use as an infrastructure to run this very intense computing we do. (Manager, Org. 11)

4.1.5 Crowdsourced testing

Crowdsourced testing has been growing and becoming readily understood and accepted by software organizations. As an example, uTest (2012) has been successful in providing crowdsourced testing services. On the humans-as-a-service (HuaaS) level, we found out that some of the interviewed organizations were aware of the possibility of crowdsourced testing and had it only as an option that had not been actively utilized. Other organizations were not willing to use crowdsourced testing for reasons such as need for domain knowledge and internal organizational changes. The organization developing accounting software was planning to perform end user testing using a community of users from academic institutions, government institutions and independent companies. With good comments and feedback from the community of end users, the organization hoped to build better systems.

4.2 Testing resources

Cloud computing provides an infrastructure that supports various testing needs within an organization. The following quotations show that cloud infrastructure was considered to have the potential to be used for testing purposes.

We have had a lot of contacts from cloud computing platform providers [saying] that you have a growing company and you have this software as a service serving thousands of customers. We [i.e. the cloud computing platform providers] have this platform that you might use for testing purposes. (Software manager, Org. 1)

Well, I would say our recommendation is just use whatever works. So, on testing side, testing is important and you should not be hindered by the lack of resources. If you don’t have resources, go to the cloud and get them. (Chief technology officer, Org. 16)

Organizations can have access to testing resources from the cloud, as need arises, and at reasonable costs. This is influenced by different factors, which we identified as properties characterizing the “testing resources” category, as explained below.

4.2.1 Need for testing resources

The interview participants mentioned that there is a need for testing resources in terms of tools and skills. Sometimes, organizations have more demands for essential testing resources than they are able to provide or sustain internally. The need varies between organizations. Some organizations might lack enough testers and therefore hire more testers. Others might have the need to improve the testing processes and this leads them to being on the look-out for better tools.

In terms of security testing, there are not enough tools, if you look at the tool market saturation, there are usually tens of tools in any given market, but the choices for security testing tools are rather few. So I think there are more tools coming up and we need to keep our eyes open. (Testing and methodologies director, Org. 2)

Yes. At the moment we have new testing resources and we have to try to just keep training to get [more testers]. After that we can see the real situation. How much [we are] still lacking. (Mainline testing unit leader, Org. 5)

We have a lack of resources in testing … and then we are also planning to start recruiting testers. (Manager, Org. 11)

Every now and then we feel that we would need a server [or] a couple of servers for our testing purpose. We have looked at Amazon web services to kind of see how we can utilize that platform for testing. (Founder and partner, Org. 17)

The need for testing resources is an important motivation for adopting cloud-based testing. This is also fuelled by the convenience that comes with using cloud-based testing environments. Organizations can simply acquire and release the testing infrastructure as needed. The following two quotations are deliberations between two managers from the same organization regarding their need to adopt cloud infrastructure for testing. The name of the software, (marked with **), is withheld due to confidentiality.

… it came to [our] understanding that good test bed resources for different developers are missing or [have been] requested. [So] we have [recently] needed the so-called cloud test resources for software developers to test their code within a cloud environment. The environment is done within the ** test environment. (Manager - Computing environments group, Org. 20)

Actually, the test environment is quite [a] good point. Well, we have the ** test environment, but could we have general test environments for cloud computing? For example, we could take [an] Amazon cloud instance, and use that as a part of testing, and then we could take one part from our computing environment, and use that in comparison with the external testing environment, and see what the difference is. (Manager—Software engineering group, Org. 20)

4.2.2 Benefits of cloud-based testing resources

During the past several years, there has been a growing popularity about cloud-based testing within the software testing community. Practitioners are more willing to adopt cloud-based testing after they have heard more about it and are convinced about its benefits. The benefits of cloud-based testing include reduced costs, flexibility and access to global markets among others (Riungu-Kalliosaari et al. 2012a). These benefits were reiterated throughout the data and were continually a source of motivation for the organizations to utilize cloud-based testing resources.

…for instance, if we need to set up new environments, large, small or flexible environments, it would be easier in the future and it would even be more cost-effective. (Head of software development, Org. 9)

It’s easier for the [employees] to use our credit card to buy some resources for Amazon than ask for a new computer. (Manager, Org. 11)

Testing on demand is definitely good. You don’t have to buy, let’s say, a monthly license for some tools, if you just need [them] for 2 weeks, you just pay for the usage. (Functional testing unit leader 2, Org 4)

4.2.3 Awareness of cloud-based testing resources

Some practitioners have been optimistic and sought to understand how their organizations can benefit from using cloud computing. On the other hand, others have been pessimistic, disregarding cloud computing as merely marketing hype. Nevertheless, over the course of the data collection phase, we observed a growing interest about cloud computing and consequently, cloud-based testing. For example, during the first interview round, organization 11 was opposed to the idea of cloud-based testing. However, during the third interview round, the manager from that organization mentioned that they had selected Amazon cloud to perform “testing and very fine calculations”.

Organizations follow the market trends and recommendations primarily with the aim to improve their working techniques and increase the return on investments (ROI). In creating the awareness about the available cloud-based resources that could be used for testing, the information used was generated by respected parties, e.g. technology analyst firms such as Gartner, and also by word of mouth from one organization to another. One interviewee reported this as follows:

There are many organizations which do whatever Gartner tells them to do. So if Gartner says you must make [a] cloud strategy they will make cloud strategies. So, that’s one aspect, and Gartner has now been promoting that cloud computing is very important now …, Salesforce has [gained] pretty good speed now, so quite many people are finding them. And then when they speak to those using Salesforce, we get recommended. (Founder and partner, Org. 17)

The increasing awareness of cloud-based testing pushes organizations to make decisions about the use of cloud-based testing resources. This causes organizations to trade between in-house testing resources and cloud-based testing resources. The considerations to be made call for balancing the potential gains and trade-offs between using already existing internal testing infrastructure against that hosted in the cloud. Our interviewee projected that the trend would be more towards organizations “taking the step” to utilize cloud-based testing resources (Testing and methodologies director, Org. 2).

4.2.4 Organizational dynamics

We observed that small and medium sized enterprises (SMEs) as well as testing consulting organizations were more willing to take cloud-based testing into use. These organizations were also optimistic about the possibilities of utilizing cloud-based testing. Generally, SMEs do not have big budgets and therefore aim at minimizing operational costs by using cloud-based testing resources only when needed. SMEs use cloud-based testing resources out of need and opportunity as opposed to making strategic plans.

It takes practically no money and no time to launch and install testing software on these servers and run the tests. (Owner, Org. 15)

We didn’t decide that everything is cloud, but we looked at overall cost of managing applications, and each time it seemed like it made more sense to use cloud over anything else. And we think for us also, we want to focus on working with our customers rather than having the issues with the servers and so on, so we can let others manage that. (Founder and owner, Org. 17)

On the other hand, large organizations require carefully thought out strategic plans and decisions to adopt cloud computing, and consequently, cloud-based testing.

…let’s say system testing phase that only lasts for 1 month. [A] customer has two choices, they can either use whatever they have, that is, a smaller server, smaller infrastructure, than the real one would be. So they make [a] compromise of how valid the testing results would be. Or, they can buy time from a cloud and buy time for a bigger environment that is already quite close to what they will have in the production environment. If they make an investment in a cloud, they will actually have more realistic testing results, and they can have more confidence that their system will be working correctly when it’s live. So it’s a budgetary choice, but most companies say—we already have this environment; let’s use it and be happy with the results it can give us. So, that is the case, but as you see the advantages in using clouds as a testing environment, it will certainly be at some point, probably near future, that somebody will take that step. (Testing and methodologies director, Org. 2)

As demonstrated by the quotation above, large organizations have to consider various aspects. These may include details such as what platform to use, which service provider to select, and how to disseminate the decision across the organization.

4.3 Value-added features of cloud-based testing resources

When analysing the data, we noted that interviewees pointed out some aspects of cloud-based testing that bring additional value to the cloud-based testing experience. The value was viewed in terms of the organizations being able to improve their development and testing practices, as well as being able to adequately meet the customers’ needs.

I think about value from the customer’s point of view. If it solves a customer’s problem and if you can just deliver it faster to the customer, it adds more value. (Software manager, Org. 1)

[Cloud-based testing] will just create more versatile opportunities [and] options for serving the customers better, finding the best way to serve the customer in different situations and finding the most economical and most feasible way of doing the testing. (Chief executive officer, Org. 6)

We classified the value brought about by cloud-based testing into two categories. The first one relates to the value added as a result of using the cloud-based testing resources (described in this subsection), and the second one deals with the value added to the tested applications and services (presented in the next subsection, 4.4).

4.3.1 Reduced maintenance efforts

As mentioned in the benefits of cloud-based testing resources, cloud-based testing resources can be acquired on an on-demand basis and at low costs. The flexible approach to setting up cloud-based testing environments contributes to reducing efforts, costs and resources associated with maintaining testing resources. The minimized maintenance complexities are a value proposition that is appreciated by customers. As exemplified by the chief technology officer from Organization 16, cloud-based testing resources help to reduce the difficulties associated with traditional test laboratories which contained “lots of computers for every variation of the operating system environment such that [it] was a huge investment with huge difficulties in maintaining that environment”. Organizations were eager to relieve themselves of the burden of having to sustain the testing infrastructure internally.

So our customer doesn’t have that kind of organization that they could actually have the maintenance for these systems that we are providing for them, so it would be easier for them also, if we would do the maintenance for the system and then they could only use [the system], as a service. (Head of software development and hardware design, Org. 8)

4.3.2 Security of the cloud-based testing environments

Security has popularly been mentioned as a concern associated with cloud computing and, consequently, cloud-based testing (Timilsina et al. 2012). The interviewees stated security as an important requirement and a risk associated with cloud-based testing and cloud computing in general. Cloud service providers have taken proactive steps to guarantee security of the cloud services (Timilsina et al. 2012). If need be, the possibility for customers to test the security of the platforms in use gives an opportunity to ensure security while using the cloud for testing.

Many of the platforms nowadays already contain a lot of intrusion detection and prevention mechanisms. How would you go about performing, for example, security testing? A wise thing is often to notify your platform provider that you are going to test [the platform]. Otherwise your testing might trigger the intrusion detection systems, and the platform provider will think that someone is making a denial of service attack. (System architect, Org. 18)

By allowing customers to confirm the security of the cloud-based testing environment, cloud service providers demonstrate their willingness to assess any prevailing security risks. On the other hand, customers are able to address their security concerns. As a result, this could foster customers’ trust in the viability and applicability of using the cloud for testing.

4.3.3 Defined testing parameters

Cloud-based development and testing environments provide a unified platform for hosting, running, developing and testing applications. Some implementation details required for running an application are already taken care of by the cloud service provider, with guidelines on how to efficiently operate within a given platform. Therefore, customers know what they are expected to do so that their applications can run or be tested within a particular platform. The guidelines provided by the cloud service provider can be translated into parameters that can be used during cloud-based testing. For example, within the SalesForce cloud, the implementation details required for running an application are handled by SalesForce. So a customer needs to focus on testing “the data model, business logic and user interface” (Founder and partner, Org. 17)

4.4 Value-added features of the system under test

Cloud-based testing increases the opportunities for producing applications and services with various value propositions.

4.4.1 Better applications and services

The interviewees emphasized on the need to constantly aim at producing applications and services with good quality, be it using conventional or cloud-based testing. As discussed in 4.2, the cloud favours iterative development cycles that encourage continuous improvement.

Using the cloud for development and testing might produce systems that are released with more bugs, but since they are easier to fix, you get into a better quality solution after a few [development] iterations. (System architect, Org. 18)

The cloud also encourages development activities that foster closer collaboration between developers and testers (Riungu-Kalliosaari et al. 2012b). Tightly integrated development and testing teams can have better environments for interaction and communication. For example, the system architect from Organization 18 described a situation whereby a tester is able to pass an actual virtual machine snapshot to the developer, so that the developer can see the non-reproducible bugs that he/she would otherwise not be able find again. Consequently, this may promote better applications and services.

4.4.2 Quality assured applications and services

Cloud-based testing environments can mirror real production environments. When used for testing, cloud-based testing environments may help to produce realistic testing results (Riungu-Kalliosaari et al. 2012b). By obtaining realistic testing results, customers are able to give clear descriptions of the systems behaviour under certain (tested and proven) conditions. For example, a customer who uses cloud-based testing resources to perform system testing “can have more confidence that their system will be working correctly when it’s live” (Testing and methodologies director, Org. 2). Furthermore, a customer can generate measurable and traceable quality expectations of the cloud-based tested applications, services and systems.

4.5 Summary of the categories

Figure 1 shows the condition-action-consequences relationships between the categories. The core category of this study is Utilization of cloud-based testing resources. It demonstrates how cloud-based resources can be used to support various testing needs in software organizations. The factors found in the category testing resources are prerequisites occurring before cloud-based testing resources are utilized. The category value-added features of cloud-based testing resources describe the effects of utilizing cloud-based testing resources. Similarly, the category value-added features of the system under test describe the possible positive effects of cloud-based testing on the tested applications, services and systems.

Fig. 1
figure 1

Conditions–actions–consequences relationships between categories

Full size image

The main motives for organizations to take cloud-based testing into use are to seek for better testing tools and resources (Need for testing resources) and to harness the benefits of cloud-based testing. The information generated by respected parties, e.g. Gartner, influences the practitioners’ interests in using cloud-based testing (Awareness of cloud-based testing resources). The smaller an organization is, the more willing it is to utilize cloud-based testing resources (Organizational dynamics). On the other hand, larger organizations approach cloud-based testing through strategic practices (Organizational dynamics).

In practice, cloud-based testing resources are utilized in various ways (Utilization of cloud-based testing resources). These include, for example, performance testing, multi-platform testing and testing software that requires heavy computing capabilities. Cloud-based testing resources also support organizations in involving users during iterative development and testing activities. Furthermore, crowdsourced testing provides the option of using more human resources to test software. Utilizing cloud-based testing resources leads to improved applications and services; whose quality can be guaranteed with specific measurements (Value-added features of the system under test).

Cloud-based testing environments come with value-added features. Cloud-based testing resources are easier and cheaper to maintain, a factor that is much appreciated by practitioners (Reduced maintenance efforts). Before utilizing cloud-based testing resources, customers can be able to address their security concerns in collaboration with the cloud service provider (Security of the cloud-based testing environment). This helps customers to build more confidence in the cloud-based resources theychoose to use for testing. In addition, the guidelines for using cloud-based testing resources, usually given by the cloud service providers, enhance the testing experience by making it easier for the customers to know how to avoid or solve potential bugs (Defined testing parameters).

4.6 A strategy for adopting cloud-based testing

Even though there is a continuous need of testing resources, practitioners are wary of cloud-based testing for reasons such as security and uncertainties with the management and legal aspects surrounding cloud-based testing (Riungu et al. 2010; Riungu-Kalliosaari et al. 2012a). We mapped the observations of this study to steps forming a strategy that can provide a structured approach for adopting cloud-based testing (Fig. 2). The arrows in the figure represent the transition from one step to another. This transition can be applied according to the needs of an organization.

Fig. 2
figure 2

Cloud-based testing strategy

Full size image

4.6.1 Evaluate the line of business

Our earlier findings (Riungu-Kalliosaari et al. 2012a) show that organizations dealing with mission critical software are careful to consider the suitability of cloud-based testing for the organization and all possible restrictions that must be observed. In addition, some legal regulations might prohibit customer data from being stored in data centres situated in other countries. For example, one interviewee noted that they “have legal restrictions on [the] data that [they] would not be able to think about, Amazon for example, because the data have to be stored in Finland [Regulation of banks]” (Software manager, Org. 1). These observations suggest that the line of business plays a vital role in determining the appropriate approach for sourcing, accessing, using and managing cloud-based testing resources. By first taking the line of business into account, it helps in determining whether the use of cloud-based testing fits within the restrictions that might be as a result of the specific domain in which the organization operates. For instance, in the example mentioned above, the data storage requirement would guide the organization to select a cloud provider with data centres situated within the country in question.

In Sect. 4.1, we observed that SMEs seem to be more willing to adopt the use of cloud-based resources for testing needs because of the cost advantages that they can get. On the other hand, larger firms might have to consider potential changes that might occur due to cloud-based testing, and whether they are willing and able to address these changes. These findings suggest that the size of the organization has an influence on the decision to adopt cloud-based testing, hence our including this factor within the strategy.

4.6.2 Assess the need

As described in Sect. 4.5, testing needs and prospects of potential benefits are some of the primary motivations for organizations to adopt cloud-based testing. Hence, after making considerations related to the line of business, the next step would be to assess the (testing) need so that an organization is fully aware of the problem for which it seeks a solution. Some of the things to consider when assessing the testing needs are the motivation for and benefits of cloud-based testing for the organization, the customer’s needs, as well as the knowledge, and tools required.

The trade-offs can be considered across various aspects. For instance, there is an example in Sect. 4.1 showing that cloud-based testing resources can scale to replicate a production environment. This would produce realistic test results and would be compared to those obtained when using limited in-house resources (Riungu-Kalliosaari et al. 2012b). An organization may decide to settle for the test results gained from using the in-house resources, or it could choose to harness the cloud and achieve more realistic results. On the other hand, the in-house resources provide a more secure environment than what is mostly expected of the cloud-based resources.

4.6.3 Identify and select the cloud provider

We observed that the organizations were taking time to indentify a suitable cloud provider. The interviewee from Organization nine stated that they would be inclined to utilizing the Microsoft Azure cloud environment because they were “a Microsoft house, [with] Microsoft development platforms and environments”. Additionally, the interviewee from organization 17 explained that they had looked at utilizing Amazon platform services for testing. Instead of purchasing the testing servers directly from Amazon, they bought the servers through the cloud brokerage services provided by organization 15 because they were readily “available and [at] low costs”. Similarly, organization 1 had identified Amazon as having “interesting opportunities for testing”, and sect. 4.1 mentions that organization 11 had identified Amazon for some of its testing tasks.

With the rising number of cloud service providers, it is worth taking some time to scrutinize various providers and select one that best fits the needs and goals of the organization. The cloud service providers can be considered based on security policies, data centre locations, service-level agreements and terms of service. An example of dealing with security would be to opt for a private cloud, to enact security testing mechanisms and to always have backups in case of failure.

4.6.4 Utilize the cloud service

We observed several ways through which cloud-based resources can be utilized for testing (Sect. 4.2). For example, Organization 16 was using Amazon for scalability and performance testing. Once the cloud service provider has been selected and all the security restrictions, service-level agreements and terms of service (TOS) have been agreed upon, and it is natural for an organization to perform cloud-based testing. It is important that all the contractual agreements are adhered to by both the provider and customer throughout the cloud-based testing process.

4.6.5 Re-evaluate

This step is not mapped with any of the findings, but we consider it as a part of any service experience whereby it is good to continuously assess the extent to which the customer’s needs and goals are being fulfilled. Here, the customer organization works together with the cloud provider to evaluate the overall effectiveness of the cloud-based testing activities. Any possible changes occurring within an organization are assessed, and possible solutions are implemented, e.g. an organization might realize that there is need for acquiring new skills in order to improve the success of the cloud-based testing process. An organization might have to re-consider each step of the strategy, hence the cyclical flow of the strategy.

5 Discussion

We applied a qualitative approach to study the adoption and use of cloud-based testing in different organizational contexts. This study described (1) the motivating factors that drive organizations to adopt cloud-based testing, (2) the various ways in which cloud-based resources are utilized for testing, (3) the favourable consequences of utilizing cloud-based testing resources, and (4) the positive effects of cloud-based testing on the system under test. An additional outcome of the study is the cloud-based testing strategy for adopting and assessing cloud-based testing in organizations.

Our findings show that practitioners are in continuous need of testing resources. We identified cost-effectiveness as one of the advantages of cloud-based testing and an important motivation for adopting cloud-based testing. This observation resonates with Robinson’s and Ragusa’s (2011) emphasis on cost reduction being the primary business goal that should be considered when developing requirements for cloud-based testing. Cloud-based testing should ultimately contribute towards reducing the total cost of ownership, which is a vital business goal for cloud computing in general.

The findings indicate that cloud computing is used for performance and scalability testing. In addition, provisioning cloud-based testing infrastructures is fast and cost-effective, thus enabling organizations to conduct feasibility tests as well as testing across various platforms. Other ways for performing cloud-based testing are usability testing, testing CPU-intensive tasks and crowdsourced testing. These findings contribute to growing inquiries about ways to perform cloud-based testing (e.g. Candea et al. 2010; Ganon and Zilbershtein 2009; Jenkins et al. 2011; Koong et al. 2013; Oriol and Ullah 2010) across different testing levels, e.g. system testing, and testing types, such as functional, compatibility and privacy aware testing (Incki et al., 2012; Priyanka et al. 2012; Wu et al., 2011).

Parveen and Tilley (2010) evaluated the characteristics of an application and types of testing as criteria for considering the migration of software testing to the cloud. They identified unit and performance testing as possible through cloud computing environments. In our study, we observed similar tendencies whereby some of the interviewees were actually using the cloud for performance testing. In general, security is a major requirement for the majority (if not all) of cloud users and our interviewees were not an exception. Testing of security issues may be challenging. This also includes testing applications that make use of large test data sets, some of which may contain confidential information, e.g. real customer information. However, uTest (2012) reports security testing as one of the services provided through its crowdsourced pool of testers.

While the experiences highlighted in our study suggested that cloud-based testing may improve testing, many testing-related problems remain unsolved. For example, cloud-based testing widens both manual and automated testing offerings, but it does not offer a generic test automation environment that covers testing needs. Hence, we can expect that hybrid solutions which include a combination of in-house and cloud-based resources will be used widely. Some examples of hybrid solutions exist (Robinson and Ragusa 2011; Wu et al. 2011).

The ISO-IEC 29119 testing standard (ISO/IEC 2012) maps quality characteristics to types of testing to show which testing types can be used to test certain quality characteristics. For example, in the standard, performance-related testing is mapped onto performance efficiency and portability testing is mapped onto portability. We found that cloud-based testing resources can be used for various testing needs, such us performance and multiplatform testing. The cloud-based testing techniques that we have identified in this study may be applied to test for corresponding quality characteristics of the system under test. For example, multiplatform testing can be used to test for an application’s portability, which is an essential quality characteristic for web and mobile applications (Murthy and Suma 2014).

For cloud-based testing service providers, security breaches and system downtime are critical failures that should be avoided as much as possible. These failures can lead to costs associated with disaster recovery, and even a negative image in the market. Hence, cloud-based testing service providers must ensure that testing is safe, works efficiently and with clear disaster recovery abilities. Lee et al. (2009) developed a quality model for evaluating software as a service (SaaS). They considered six important features of SaaS (reusability, customizability, pay-per use, data managed by providers, scalability and availability) and used them to extract five quality attributes with different measurement criteria. This approach may be adopted for evaluating the quality of cloud-based testing resources. As a starting point, security as a value-added feature of cloud-based testing resources may be used as an evaluation or measurement criteria for determining the quality of cloud-based testing resources.

Our earlier findings also reveal that practitioners are weary of cloud-based testing for reasons such as security and uncertainties with the management and legal aspects surrounding cloud-based testing (Riungu et al. 2010; Riungu-Kalliosaari et al. 2012a). Therefore, practitioners need supportive approaches to guide and support decisions for adopting, conducting and managing cloud-based testing. We address this need by proposing a strategy that provides organizations with a structured approach for adopting cloud-based testing.

Other researchers have proposed cloud adoption methods covering the adoption of cloud computing for a wide spectrum of purposes (Khajeh-Hosseini et al. 2012; Zardari and Bahsoon 2011). Khajeh-Hosseini et al. (2012) presented cloud adoption toolkit aimed at supporting cloud migration decisions. The toolkit includes five techniques for analysing the suitability of cloud computing for an organization namely, technology suitability analysis, energy consumption analysis, stakeholder impact analysis, responsibility modelling, and cost modelling. Zardari and Bahsoon (2011) proposed a cloud adoption approach which advocates for mapping an organization’s requirements with the cloud provider’s features. The aforementioned cloud adoption methods cover the adoption of cloud computing for a wide spectrum of purposes. However, the strategy we present in this paper differs from these cloud adoption methods because it is aligned to support the adoption of cloud-based testing. Nevertheless, it can be used to complement other cloud adoption methods as considered appropriate.

Our previous study revealed that the application domain seemed to hinder the adoption of cloud-based testing (Riungu-Kalliosaari et al. 2012a). Interestingly, organizations which deal with mission critical systems had declined to participate in the first interview round, but later became interested in the study. This would support our thesis that the strong influence of the application domain on adoption of cloud-based testing is going to diminish over time. This may be attributed to the fact that once people become more aware of cloud computing, they understand it better and are therefore more willing to take it into use within their organizations. Therefore, with an increasing awareness of the possibilities for utilizing cloud-based testing resources, we can expect the adoption of cloud-based testing to increase.

Cloud computing is changing the way software is accessed, delivered and used—away from licensing software to on-demand usage based on economies of scale (Gartner 2009). Similarly, cloud-based testing affects: (1) the acquisition model, which emphasizes software services instead of software products; (2) the business model, which causes a shift from licence based fees to pay-per use pricing; (3) the access model, whereby dedicated testing environments are replaced by cloud-based testing environments; and (4) the technical models of testing, for example, new opportunities for performance testing. Nevertheless, testing still remains a task requiring human effort, depending on the nature of the application domain, quality requirements, testing skills and resources. Cloud-based testing changes the control (ownership) of the testing environments from own in-house environment to multi-user environments. This might result in reduced testing time because of reusable environments.

5.1 Threats to validity

Threats to external validity relate to the generalizability of the results, and the researcher tries to evaluate whether the results can apply in other situations outside the studied context (Runeson and Höst 2009). The generalizability of our results is threatened by the fact that we gathered data from 35 interviews. Nevertheless, owing to the qualitative nature of the study, our objective was not to make statistical interpretations, but to establish relevance grounded in the data. Hence, the results may be relevant for organizations with similar characteristics as those featured in this study (Runeson and Höst 2009). In addition, the main limiting context variable is that all the organizations involved in this study were located in Finland, although some of them were international companies. Different views might have come from organizations in countries with different cultures. However, the data were gathered through several interview rounds, which allowed us to have a variety of views from participants in different domains. Therefore, the results may be applicable in other countries as well.

In qualitative studies, the researcher is the main instrument of the research and is required to employ a great amount of effort in exploring and interpreting the underlying meanings within the data. The first author was involved throughout the interviewing and coding processes. It can be argued that there might have been chances of bias in interpretation which might influence the outcome of the coding process and hence reduce the reliability of the results. We employed triangulation and audit trail to address this limitation.

Triangulation is a validity procedure which aims at obtaining concurrence from a variety of information sources (Creswell and Miller 2000). Four types of triangulation exist across data sources, theories, methods and investigators (Creswell and Miller 2000). In this study, we used triangulation among different investigators (researchers). When preparing the interview questions, the first author sought comments from the second and third authors so as to produce questions that were “neutral” in a sense and hence generate a balanced view from the interviewees. Towards the end of the coding process, the first author documented and discussed the results with the second and third authors in several occasions. We also used triangulation across data sources by interviewing respondents from different organizations.

Audit trail entails the use of documentation as evidence of the steps taken to generate the research conclusions. All except one interview were recorded and transcribed. The exception was answered through email as explained in section 4. All memos, notes and codes are documented on the analysis tool Atlas.ti.

Maxwell (2005) mentions reactivity as another type of threat in qualitative studies, which refers to the researcher’s potential to influence the interview atmosphere and the interviewees. By virtue of using open-ended interviews, there might be a risk of the researcher imposing his/her opinions on the interviewees. This issue was addressed by the first author being cautious during the interviews in order to steer clear of imposing any of her opinions on the interviewees.

In order to gather views that were more representative of the true reality, we looked for organizations that were in some way applying cloud computing, or at least had plans to take it into use. Eight out of fifteen organizations that participated in the third interview round had adopted the use of cloud computing—five were utilizing the cloud for testing, two were cloud service providers and one was a testing consulting company whose application of cloud-based testing was determined by the customers. The views obtained from these organizations helped us to obtain balanced interpretations that reflect the reality as presented in this study. In another paper (Riungu-Kalliosaari et al. 2012b), we have discussed some effects of cloud-based testing on testing work and on delivery and support of testing services.

6 Conclusions

This study investigated the adoption and use of cloud-based testing in participating organizations, employing a qualitative grounded theory method. We conducted thirty-five semi-structured interviews with respondents from 20 organizations. The results reveal cloud computing has the potential to supply organizations with sufficient resources to support various testing needs. In this paper, we have described (1) the motivating factors that drive organizations to adopt cloud-based testing, (2) the various ways in which cloud-based resources are utilized for testing, such as performance testing, multiplatform testing, and testing data intensive applications, (3) the favourable consequences of utilizing cloud-based testing resources, (4) the positive effects of cloud-based testing on the system under test, and (5) a cloud-based testing strategy that can be used to assist organizations when adopting cloud-based testing.

The study contributes to the growing interest in the adoption and use of cloud-based services, and specifically in cloud-based testing. It helps both researchers and practitioners to understand how cloud-based resources can be used to perform testing, and the possible positive outcomes that can be obtained. The study provides an opportunity for other researchers to perform further empirical studies within the growing field of cloud-based testing. The proposed adoption strategy for cloud-based testing may be used by practitioners that might wish to utilize cloud-based testing resources. The future work entails the application of the results from this study to conduct further empirical studies.