Keywords

1 Introduction

Software engineering is a branch of computer science that studies the software life cycle, from the requirements gathering to the maintenance phase; it aims to build quality software that meets the user requirements, budgets, and times established for its development [1]. The life cycle comprises several phases containing several activities [2], which usually require a series of support tools to complete these tasks in the best way and quality [3]. There are several software life cycle models, each with its characteristics, advantages, and disadvantages. These models agree on the main phases of software development: analysis, design, coding, testing, and maintenance; all models include the software test management phase as a fundamental phase of the software development process, for example, the Test-Driven Development methodology the software development is based purely on the execution of tests [4].

The software testing process is essential because it enables developers to deliver high-quality standards and minimize risks [5]. There are metrics such as Capability Maturity Model Integration (CMMI) or Quality Improvement Paradigm (QIP) that manage software testing in traditional methodologies. Still, they do not frame agile development testing because they do not measure the agility of the process [6]. Conversely, software testing is an integral part of the development; however, developers usually do not use an adequate framework, causing that test management is informal; they do not perform sufficient test cases for code validation. Furthermore, documentation is non-existent or incomplete [7]. Failure to manage software testing correctly generates problems such as the deterioration of software quality that negatively affects the software production process, causing the maintenance phase to become complex. It can also lead to incomplete software functionality and delays in developing new software projects due to lack of time. Other associated problems are increased resources regarding costs and time allocated for software development, but the most severe is the inconvenience generated to end-users [8].

In this context, we base this study on the Design Science Research (DSR) approach [9] to evaluate the value and usefulness criteria of the software product, so we pose the following research question: Does the application of a test management framework in the development process improve the quality of the software product? Therefore, the paper aims to answer this question by developing and implementing a software testing framework based on the processes described in ISO/IEC/IEEE 29119–2 and the documentation templates of ISO/IEC/IEEE 29119–3 [7]. We evaluate this proposal through a case study applied in the “Cooperativa de Ahorro y Crédito Atuntaqui” in Ecuador [10]. The rest of the paper is structured as follows: Sect.  2) Research Design: we establish the research activities based on DSR, theoretical foundation, design, and framework implementation (artifact). Section 3) Results: Evaluation of quality-in-use results of the framework artifact. Section 4) Discussion: discussion of the research. Section 5) Conclusions and future work.

2 Research Design

We followed the Design Science Research (DSR) guidelines for the research methodology, see Table 1.

Table 1. Research design methodology.
Full size table

2.1 Population and Sample

We evaluate this proposal through a case study applied in the “Cooperativa de Ahorro y Crédito Atuntaqui” in Ecuador [10]. The population of the present study was 8 people from the Technology Department. They are made up of a Director, a Development Administrator, three Programmer Analysts and three Technical Support staff.

2.2 Theoretical Foundation

V-model.

Different software development models include the waterfall model, the general V-model [11], or agile programming models. Figure 1 shows the V-model containing two phases; the first one corresponds to the project development phases and the second one to the project testing phases. Phases of the same level of development can run in parallel. For each development level, there is a test level. The tester must ensure that the results comply with the software verification and validation [12].

Fig. 1.
figure 1

The V-Model as a systems engineering process [13].

Full size image

Software Testing.

Testing is a software engineering discipline that analyzes software or components to detect differences between the requirements and the existing functionality or to detect software failures [14]. It is difficult due to the exponential increase of test sequences; advantageously, testing techniques help perform this task [15]. The International Software Testing Qualifications Board defines software testing as the process in the static and dynamic life cycle activities; related to the planning, preparation, and evaluation of software products to determine that they meet and are suitable for the specified requirements detect defects [16].

A test is a set of activities that need to be planned and performed systematically [17]. For this reason, during the software development process, a template for elaborating and executing tests is defined; it consists of a set of steps that includes test methods and test case design techniques [18]. Software testing is part of a broader topic, usually referred to as software verification and validation (V&V) [19]. Validation is a set of tasks that ensure that the built software follows the requirements requested at the beginning of the process. Verification is the set of tasks that ensure that the software correctly implements a function [20].

Testing Techniques.

Testing aims to detect as many faults as possible; therefore, there are many techniques to accomplish this purpose. The techniques verify a program as systematically as possible, identifying the inputs that will produce expected behaviors of the program [21].

Unit Testing.

is the basis for verifying the smallest unit of software design: the software component or module. These tests can be performed simultaneously on multiple components and are considered adjuncts to the coding process. Typically, unit tests have access and are executed to the source code and with the support of debugging tools [22].

Integration Tests.

Their purpose is to detect defects not found during unit tests. They focus on integrating and testing two or more components. Once the tests no longer detect new defects, they are added in additional components [23].

System Testing.

Unit and integration tests find defects in individual components and the interfaces between components. System tests demonstrate that components are compatible, interact correctly, and transfer the correct data at the right time through their interfaces [13].

Acceptance Testing.

This type of testing is performed when the product is ready to be deployed in the customer’s environment. Then, they focus on testing the user requirements, i.e., to demonstrate compliance with the acceptance criteria of the requirements. Once these tests are passed, the customer must accept the product [12].

White Box Testing.

Focus on designing test cases to validate the internal behavior and structure of the program. The design of these tests aims to execute at least once all program statements and all conditions to check both true and false values. Thus, they examine the internal logic of the program without considering performance aspects [24].

Black Box Testing.

They are a way of selecting conditions, data, and test cases from the system requirements documentation. Black box testing tests only the inputs and outputs of the system, i.e., it ignores the internal mechanism of the software. Instead, they consider the behavior of the software from the point of view of an external observer [24].

Reviews.

The verification and validation process requires software inspections and reviews. The latter analyzes and checks system requirements, design models, program source code, and even proposed system tests. They are also named “static” techniques, where it verifies the software without executing it [25].

ISO/IEC/IEEE 29119.

Provide a set of internationally agreed standards for managing software testing in any software development life cycle or organization [26]. They are the only internationally recognized standards for software testing; they provide a high-quality approach communicable worldwide [27]. The standard aims to cover the software life cycle, including aspects related to testing organization, management, design, and execution [28]. Figure 2 shows the structure of the ISO/IEC/IEEE 29119 standard.

Fig. 2.
figure 2

ISO/IEC/IEEE 29119 standard structure [29].

Full size image

Part 1 - Concepts and Definitions.

This section introduces the set of standards, includes common definitions of all its parts, and describes elementary testing concepts. It explains the scope of the components and describes how to use standards for different lifecycle models [26].

Part 2 - Test Processes.

Defines testing processes using a three-layer model. The top layer corresponds to the organizational test process to generate and maintain organizational policies and strategies for testing. The middle layer includes test management processes for test planning, monitoring, control, and completion. Finally, the bottom layer corresponds to dynamic testing processes because the overall model does not include static testing processes such as static analysis and reviews [26] (see Fig. 3).

Fig. 3.
figure 3

Test processes defined in the standard [29].

Full size image

Part 3 - Test Documentation.

This section provides templates with content descriptions for the main types of test documents. There is a strong link between the standards in Part 2 (Processes) and Part 3 (Documentation), as the results of the processes defined in Part 2 correspond to the documentation specified in Part 3 [26].

Part 4 - Testing Techniques.

These techniques require users following Part 2 to develop test plans that specific test case design techniques and criteria for achieving test completion. Part 4 defines a wide range of test techniques and corresponding coverage measures [26].

Part 5 - Keyword Driven Testing.

This section defines requirements for keyword-driven testing and minimum requirements for the supporting tools needed to utilize the keyword-driven testing approach fully [26].

2.3 Methodological Proposal

The proposed testing framework covers only functional testing. We intend to iteratively increase other types of tests in the framework in future work as it is implemented in new software projects. The proposed methodology offers the following activities:

  • Elaborate a plan for software test management.

  • Establish the team structure for software testing.

  • Define the testing execution process.

  • Establish the documentation and deliverables of the testing process.

Roles.

The work team involved in the software testing process has the following roles and responsibilities, see Table 2.

Table 2. Roles and responsibilities of the testing team.
Full size table

Process Testing Flow.

Contains the activities and the order of execution in the framework, see. Fig. 4.

Fig. 4.
figure 4

Software testing process flow [30].

Full size image

Planning.

The structure of this phase contains five activities:

Identify Test Requirements.

This activity determines the software requirements or features included and excluded for testing and the scope of testing.

Prioritization.

This activity prioritizes the list of test requirements according to a test priority. Two factors must be evaluated for each requirement: failure priority and frequency of use (see Table 3).

Table 3. Priority factors for requirements testing.
Full size table

Identify Resources.

Estimate the number of resources needed to design, build and execute software testing. The types of resources are data, environment, tools, and human resources.

Create the Schedule.

This activity includes estimating time to design, build and execute the tests based on previous experiences and metrics.

Generate the Test Plan.

This activity identifies and defines the deliverables to be created, maintained, and available during test execution. Next, it will document the delivery schedule for those deliverables. Finally, it will combine the data from the previous steps and create a Software Test Plan.

Design.

This phase contains the following activities:

Identify Test Cases.

A test case combines conditions and inputs for a specific test requirement. Generally, each test requirement should have more than one test case.

Identify Test Data.

This activity identifies the data required for the test cases defined in the previous activity.

Construction.

This phase primarily develops the procedures and data needed for testing.

Test Procedure Creation.

A test procedure is a set of detailed instructions for preparing, executing, and evaluating the outcome of a test case or set of test cases.

Create Test Data.

This activity creates the data needed to execute the test procedures.

Execution.

This phase consists of executing the steps established in the test procedures; it verifies that the result obtained from the test cases matches the expected result; if it does not match, the test engineer must record and report the defect found.

Defect Registration.

The Test Engineer records and reports to the Development Area the defects found in the test execution. After the development team corrects the reported defect, the test engineer will validate the defect again and report the result. If, when validating the correction of a defect, another defect is found, the test engineer must register it as a new defect.

Monitoring and Control.

At the end of the testing period or even during the execution phase, the test engineer evaluates the test results using previously established metrics.

Test Coverage.

Provides an indicator of the number of requirements passed out of the total number of requirements specified, using the following formula:

$$TC= \frac{ETC}{TCD}$$

Where:

ETC is the number of executed Test Cases.

TCD is the total number of Test Cases designed.

Test Maturity.

This indicator measures the satisfactory results of test cases calculated with the following formula:

$$TS=\frac{TCS}{TCD}$$

Where:

TCS is the number of Test Cases with a satisfactory result.

TCD is the number of Test Cases designed for all requirements.

Defect Density.

Provides a measure of the ratio of defects to the number of specification items calculated with the following formula:

$$DD=\frac{DF}{SIR}$$

Where:

DF is the total number of defects found.

SIR is the number of specification items reviewed.

Defect Trend.

It is the number of defects as a function of time in an established classification.

Defect Percentage by Type.

This metric identifies, categorizes, and prioritizes defect types calculated with the following formula:

$$DT=\frac{NDT}{TDI}\,x\,100$$

Where:

NDT is the number of defects by type.

TDI is the total number of defects identified.

Basic Path.

Corresponds to the percentage of primary independent paths tested concerning the total ways, sum the cyclomatic complexity of the program modules, calculated by the following formula:

$$\mathrm{PCB}=\frac{NDT}{C(G)}\,x\,100$$

Where:

NDT is the number of designed tests.

C(G) is the calculated cyclomatic complexity.

Finalization.

The status of the tests can be evaluated at any point in the execution using the metrics established in the previous sections to decide if the software release is possible.

2.4 Test Documentation

Documentation corresponds to the software testing forms and templates available for organizations, specific projects, or individual testing activities.

Test Plan.

The formal document prepared by the Test Leader plans the detail of software testing activities, times, and responsible persons.

Test Design Specification.

This document contains the set of software features to be evaluated, with their respective execution priority. The Test Leader elaborates it.

Test Case Specification.

This document details the test cases to be executed for each specified software feature to be evaluated in the Test Design.

Test Procedure.

It specifies the order and sequence of execution of the test cases, restrictions, and previous actions for each test case.

Defect Report.

The record of executed test cases whose obtained result does not coincide with the expected result; therefore, the test is not satisfactory.

Results Report.

It is a management document that summarizes the results obtained in the execution of the test cases to determine if the software is suitable to be put into production or, on the contrary, some errors still need to be corrected.

3 Results

In this section, we validate the proposed software testing framework through three instruments applied in the “Cooperativa de Ahorro y Crédito Atuntaqui” in Ecuador in 2019: 1) Implementation of the testing framework project involved in real-world practice. 2) A comparison of the number of failures reported in the maintenance phase of 6 software projects implemented in a production environment. 3) A satisfaction survey of the use of the proposed testing framework.

3.1 Implementing the Testing Framework in a Software Project

In this section, we validate the study proposal by implementing the proposed testing framework in a software development project; after its implementation in production, we measure the impact through 3 cycles of test plan execution in the maintenance phase. Table 4 shows the implementation result of the design framework in a software project.

Table 4. Results of the design framework implementation.
Full size table

In the results obtained (Table 4), we note that the Test Coverage in the three cycles executed is one hundred percent, i.e., we executed all test cases designed in the test plan. The Test Maturity metric starts with a value of 77% in the first cycle, increases to 99% in the second cycle, and ends with 100% in the third cycle; this means that as we executed the test cycles, software errors were detected and corrected, and at the end all the test cases designed were satisfactory. The Defect Density in the first cycle was 95%, with the correction of the detected errors; in the second cycle, it decreases to 5%, and in the third cycle, it ends with a value of 0%, which means that all the software errors found were corrected.

The testing framework artifacts are available in the following Zenodo open data repository [31].

3.2 Comparison of the Number of Failures of Projects in the Maintenance Phase

In this validation, we analyzed the number of software failures reported in production environments of three projects implemented without using the framework and three projects using the proposed framework. The selection of the software projects was made with the advice of the Development Leader, considering the number of functional requirements and the development time of each project, see Table 5.

Table 5. Selected projects to compare the number of failures.
Full size table

Table 6 shows the number of failure support cases reported for the production environment projects in the study period.

Table 6. Reported software failures of production environment projects.
Full size table

We analyzed the number of software failures of the projects that used the proposed testing framework against the projects that did not use the framework. As a result, the proposal’s impact corresponds to the percentage of software failures achieved by the two types of projects, measured by the following formula:

$$\mathrm{\% }\,of\, failures\, achieved\,=\,\frac{No. \,of\, failures\, with\, the\, proposal}{No. \,of \,failures\, without\, the\, proposal}\,\times \,100$$
$$\mathrm{\% }\,of \,failures\, achieved\,=\,\frac{4}{32}\,\times \,100$$
$$\mathrm{\% }\,of \,failures\, achieved\,=\,\mathrm{12,5 \%}$$

The result of the calculation of the impact caused by using the software testing framework shows a reduction to 12.5% of the reported software failures instead of the projects that did not use the proposed framework in this study. In addition, none of the reported incidents corresponds to errors in the data, avoiding direct affectation to the database.

3.3 Satisfaction Survey

To measure the perception of the technical staff of the Development Area about the use of the proposed testing framework, we first conducted a diagnostic survey and another survey after the implementation of the proposed testing framework. Therefore, we conducted a diagnostic survey and then another satisfaction survey after implementing the proposed testing framework in the study company. We designed the surveys using and rating using the 3 Likert response scale (Very Good, Good, Fair). Table 7 shows the rating of the surveys and the impact of the implementation of the proposed framework.

Table 7. Impact of the implementation of the testing framework.
Full size table

4 Discussion

We consider it necessary to clarify that the development environment, the Framework artifacts, and templates are in Spanish since we believe that the replication of this study should consider carried out in the same language. However, we do not recommend doing it in a different language because we do not know if the translation of the artifacts would affect the validity of the research.

During the study, we also observed that the expertise of the development team members influences the quality of the software product; for this reason, we see the convenient use of a tool that manages software testing and describes the necessary templates for the documentation of the software evaluation process.

5 Conclusions and Future Work

In this study, based on the DSR research approach, we pose the research question: Does implementing a test management framework in the development process improve the quality of the software product? First, we answered this question by developing a software testing framework based on the processes described in ISO/IEC/IEEE 29119–2 and ISO/IEC/IEEE 29119–3 documentation templates. Then, we validated this instrument using three artifacts to measure the influence of the proposed framework on the quality of the developed software. The principal validation results show an increase in error detection in the range of 77% to 100%, reduction of defect density in the range of 95% to 0%, and a 12.5% reduction of software failures previously deployed in production environments. Therefore, we conclude that implementing a test management framework does improve the quality of the developed software product.

As future work, we propose to study the impact of the use of this proposal in software development teams with different levels of experience to check which segment has a more significant impact and is more necessary.