Keywords

1 Introduction

Software system consists of a number of separate programs, configuration files, which are used to set up these programs, system documentation, which describes the structure of the system, and user documentation, which explains how to use the system and web sites for users to download recent product information [59]. Nowadays, software systems have become an essential part of our daily life. We use these software systems daily and have generally tried to keep them updated as much as possible. While in many times these software systems do not work as expected. Therefore, creating high-quality software is an intellectual challenge. Generally this quality is ensured by a test activity. However, this activity is time consuming, and too demanding as far as resources are concerned, and it is essential to ensure a certain percent of software quality. Indeed, a defect in software can have serious consequences for users and companies. These consequences may cause trivial issues viz; loss of money, time, business credibility or even loss of life. This was the case, for example in medicine in 1985, the Therac 25, was a radiation therapy machine for treating cancer. A dysfunction of software has led to an overdose of radiation and was the cause of several deaths. Also the flight 501 failure of Ariane 5 in 1996 caused by a problem in specification, the estimated loss of this failure is 8.5 BILLION dollars. And more recently, on 6 January 2010, the German bank card holders designed by Gemalto had the unpleasant surprise of not being able to use their cards in some terminals. However, even if this bug lasted only one day, Gemalto has estimated the cost associated between 6 and 10 million. These accidents show that regardless of the scope of the software, it is necessary to validate and verify its operation and quality before using it. To ensure the quality and operation of software, testing activities have become primordial in the software development life cycle (SDLC). Indeed, software testing is a process of validating and verifying that a software product works as expected. Recently, the Software testing activity has changed dramatically, the complexity of IT systems has increased, the applications areas have been expanded, and the costs and consequences of bugs became higher, turning testing into an essential activity that should not be overlooked during the software development cycle. Testing is a vital part of software development, and it is important to start it as early as possible, and to make testing a part of the process of deciding requirements [60]. For that, in this present paper, we will present the tools, the processes, and the approaches of existing testing techniques. Also, this paper offers a detailed comparison between these testing techniques, especially, two major techniques viz; the model based testing technique (MBT), which is an application of model-based design to perform software testing or system testing. Models can be used to represent the desired behaviour of a System Under Test (SUT), or to represent testing strategies and a test environment; and the risk based testing technique (RBT), which is a type of software testing that functions as an organizational principle used to prioritize the tests of features and functions in software, based on the risk of failure, the function of their importance and likelihood or impact of failure [61].

The rest of this paper is organized as follows. Section 2 exposes general processes of MBT and RBT techniques, used tools and stockholders of each technique. Section 3 presents a classification of approaches related to each technique. Section 5 present some MBT and RBT Advantages and Limits, Sect. 6 discuss and presents the results of our analysis. Finally, Sect. 7 describes conclusion and future work.

2 MBT and RBT Processes

2.1 Model-Based Testing

The MBT (Model-Based Testing) is a form or a technique that aims to automatically generate test cases from formal specification or models describing the expected behaviour of the system under test. The behaviour model or formal specification are built from requirements and represents the software characteristics. It consists in managing and listing the requirements, creating a behavioural model of the SUT based on this list of requirements in order to generate abstracts tests cases and Requirements Traceability Matrix (RTX) by using the selection criteria that aim the model coverage and detect certain types of fault. The tests cases generated are then concretized, making them executable on the SUT, the actual result and the expected result are then compared in order to get a verdict. Based on MBT application steps, used tools and stockholders, we present the following process.

As shown in Fig. 1 above, the process of Model-based testing can be splitted into five main steps. These steps will be explained further below.

Fig. 1.
figure 1

Model-based testing general process

Full size image

  • Requirements management: The first step of model based testing is to collect customer needs, desires and constraints, manage and classify them as requirements. This first step is potentially the most important step in a process of testing software such as MBT. Requirement management step in MBT involves the collection, analysis, prioritization, validation, definition and control of all customer business requirements, it serves to create a requirement repository that is the basis of communication between analysts and testers and is define in a structured way the expected result for the software, in different terms (functional, technical, security, load and response time…).

  • Modelling: The main purpose of the modelling step in MBT is to model the system requirements, it consists in creating a behaviour model that describes the expected behaviour of the system under test and suitable for test purposes, this model is created by a tester analyst using requirements resulting from requirements management step and described in many ways, depending on the discipline. It can be described by use of diagrams, tables, text, or other kinds of notations. It might be expressed in a mathematical formalism or informally, where the meaning is derived by convention kinds of notations. They might be expressed in a mathematical formalism or informally, where the meaning is derived by convention.

  • Generation: The generation step is realized on the basis of a test generator which takes as input the model designed in the modelling step and the test selection criteria selected by the test analyst and produces the abstract test cases from the model and a requirements traceability matrix that illustrates the link between tests and model elements covered by the tests.

  • Concretization: The concretization step consists to translate the abstracts test cases to executables test cases in order to be executed on SUT. It consists in making the link between the model elements and the system’s concrete elements, and involving specific adapters and manual intervention that requires the expertise of the test engineer.

  • Execution: The execution step can be manually or automatically. In this phase, all the test cases are executed on the system under test, eventually the obtained results are then compared with the expected results to give a verdict for test cases and consequently give a status on the operation of the product [1, 2, 13, 15,16,17,18].

2.2 Risk-Based Testing

When we cannot test exhaustively, we must test selectively and achieve better with less in time and resources and without affecting the product quality. The RBT is a software testing technique or method that uses risk as a basis for test planning, It uses risk to select, prioritize and manage the appropriate tests during test execution and consequently to make sure that the limited time and resources are used to test the most important things [3, 5]. In simple terms, Risk is an undesirable event whose appearance is not certain and having as consequence negative results on the project objectives such as impact on completion date, costs, the quality, the company image, etc. Thus, the risk may be considered as the composition of two factors viz; the probability of occurrence of an undesirable event and the severity of the potential consequences of the undesirable event. Based on RBT application steps, used tools and stockholders [3, 7, 62], we present in Fig. 2 below, the process of risk-based testing that can be divided into five main steps.

Fig. 2.
figure 2

Risk-based testing general process

Full size image

  • Requirements management: As the MBT process, RBT process start with requirements management step to extract and identify requirements from system specifications.

  • Risks identification: Identifying risks is an absolutely essential activity in RBT process, it involves making a list of everything that might potentially come up and disrupt the normal flow of project, and provides the indicators that allows the organization to identify major risks before they impact operations and hence the business. It consists to identify and describe all requirements in terms of risk involved in the project. Thus at the end of this step all risk items are identified.

  • Analysis and Evaluation: Risk analysis and evaluation is the second step of risk management in RBT process. It consists in studying the risks identified in risk identification phase, categorizing them, determining the level of risk by specifying likelihood and impact of the risk and then assigning the level of risk to each item.

  • Mitigation/Reduction: The objective of Risk mitigation step is to reduce the Risk Impact or Risk Probability. It consists in looking for risk Mitigation where tests are built to mitigate the risk.

  • Execution: The execution step in RBT consists in executing test cases resulting from reduction step according to prioritization and acceptance criteria identified in the risks report.

3 MBT and RBT Approaches Classification

3.1 Model-Based Testing Approaches Classification

Arilo et al. [16, 19] have classified MBT approaches into four categories, some of these approaches use UML diagrams, whereas, the others use a non-UML notations to represent software requirements or software internal structure. This classification is described and detailed in the form of table as show in Table 1 below.

Table 1. Classification of MBT approaches
Full size table

Utting et al. [2] have defined a taxonomy of model based testing that allows the characterization of different approaches to model-based testing, they have defined three general classes viz; model specification, test generation and test execution. Each of these classes is divided into various categories viz; model specification: It is divided into scope, characteristics and paradigm categories; test generation: It is divided into test selection criteria and technology categories; and test execution: It is divided into online test execution and offline test execution. Based on this taxonomy, Utting et al. have classified a collection of existing approaches in order to show the characteristics of those approaches to target various application domains. They have classified the existing approaches into two main categories: approaches to model-based test case generation and approaches to model-based test input generation. Felderer et al. [6] have defined a novel classification of model-based security testing approaches. They have classified the existing approaches into two dimensions viz; automated test generation and risk. The first dimension describes how much of the system and the security requirements is captured by formal models. The second dimension “risk” can have the values integrated into the model or not integrated into the model. Anand et al. [58] have performed a survey of methodologies for automated software test case generation. They have classified the MBT approaches into three categories viz; axiomatic approaches that use scenario-oriented notations, finite state machine approaches that use state-oriented notations, and labelled transition system approaches that use process-oriented notations.

3.2 Risk-Based Testing Approaches Classification

In other respects, for RBT technique, Erdogan et al. [5] have classified the approaches that use both tests and risks into two global categories, some approaches when risk is used to focus testing, and others when test is used to focus risk. It defines the first category for test-based risk analysis (TR), and the second category for risk-based testing (RT). In Table 2 below we expose the approaches studied by Erdogan et al. [5] and Alam et al. [3] in order of this classification. Otherwise, depending on main focus, Erdogan et al. have classified RBT approaches into eight categories viz;

Table 2. TR & RT approaches classification
Full size table

  • Approaches that combine risk analysis and testing at a general level as Amland2000, Felderer2012, Felderer2013 and Redmill2004 and Redmill2005;

  • Approaches with main focus on model-based risk estimation as Gleirscher2011, Gleirscher2013 and Ray2013;

  • Approaches with main focus on test-case generation as Kloos2011, Nazier2012 and Xu2012;

  • Approaches with main focus on test-case analysis as Chen2003, Chen2002 and Entin2012;

  • Approaches based on automatic source code analysis as Wong 2005 and Hosseingholizadeh2010;

  • Approaches targeting specific programming paradigms as Kumar2009 and Rosenberg1999;

  • Approaches targeting specific applications as Bai2009, Bai2012, Casado2010 and Zech2011;

  • Approaches’ aiming at measurement in the sense that measurement is the main issue as Schneidewind2007 and Souza2009.

4 MBT and RBT Supporting Tools

To benefit fully from any technique or approach as MBT or RBT, the automation supports are required to automate as much as possible and to increase the reliability of the software testing process. In MBT, the challenge is that from a formal, semi-formal or informal models generate complete test cases without human interference. On the other hand, in RBT approach, the challenge is how to manage, select, and evaluate risk in testing process. In this context, when practitioners want to adopt an MBT or RBT approach, they therefore seek associated tools. For MBT, a number of model-based testing tools have been proposed [8, 10,11,12, 18]. We can classify these tools in different criteria viz; tool category: Commercial (CL), Open Source (OS) or Academic (AC); model type: UML, SysML, FSM, EFSM, Textual Models (TM), and more; and test type: Functional Testing (FT), Non Functional Testing (NFT), Structural Testing (ST). The Table 3 below describes these tools according to these criteria. For RBT, the most used tools are test management systems that support RBT approaches [14]. Table 4 exposes some of the test management tools that support RBT and some tools intended for RBT technique.

Table 3. MBT tools classification
Full size table
Table 4. RBT tools classification
Full size table

5 MBT and RBT Advantages and Limits

The MBT and RBT solutions are highly effective testing techniques that can be used to perform and manage software testing. Each solution has distinct benefits and limits. In this context, Legeard [4], has classified the major MBT benefits that solved some problems of classical approaches into six areas viz; SUT fault detection, reduced testing cost and time, improved test quality, requirements defect detection, tractability management, and requirements evolution. Also, he discussed some of fundamental limitations that limit the usage areas of MBT approaches. In the other hand, Alam [3] in his paper highlight some benefits and limits of RBT. Table 5 present some major advantages and disadvantages of RBT and MBT Approaches.

Table 5. MBT and RBT limitations and advantages
Full size table

6 Analyse and Discussion

Both studied techniques have their own processes, approaches, tools, merits and demerits. For risk-based testing, all approaches described in this paper use risk to prioritize what to test and focus on activities related to risk identification, analysis and prioritizing. Most RBT approaches are black-box testing that takes as input software requirements. In this category we find some approaches which are intended to functional testing as Amland, Chen, Bach, and others that are proposed for non functional testing as Zech, Xu and Bai. Otherwise, for white-box testing, we find a limited number of RBT approaches which are intended to Structural testing like Wong and Hosseingholizadeh. For model-based testing, the general idea is that from an explicit behaviour model that represents behaviour of system under test, generate test cases to validate the expected behaviour of the system under test. Based on studied classifications and after analyzing different approaches of model-based testing, we concluded that we can also classify existing MBT approaches according to different criteria viz; testing type, testing level, testing sources, and notation type used to represent testing sources. This classification is very detailed and it facilitates the selection of MBT approaches according to the test context (Table 6).

Table 6. MBT approaches classification
Full size table

Based on MBT Approaches Characterization proposed by Arilo and Guilherme [19], we expose some approaches in order of proposed classification (Tables 7, 8, 9 and 10).

Table 7. MBT approaches classification in term of testing type
Full size table
Table 8. MBT approaches classification in term of testing level
Full size table
Table 9. MBT approaches classification in term of testing source
Full size table
Table 10. MBT approaches classification in term of testing notation type
Full size table

For RBT technique, After Analyzing existing approaches, we concluded that we can also classify them according to the following criteria: Testing type, Testing level, Testing sources, and Notation type used to represent testing sources if exist (Tables 11, 12, 13 and 14).

Table 11. RBT approaches classification in term of testing type
Full size table
Table 12. RBT approaches classification in term of testing level
Full size table
Table 13. RBT approaches classification in term of testing source
Full size table
Table 14. RBT approaches classification in term of testing notation type
Full size table

7 Conclusion and Perspective

In this paper, we have studied the main two techniques of software testing. The idea of the first technique is to use the abstractions of a system under test and its environment to automatically generate test cases. MBT consists to create an abstract system model that specifies the behaviour of the SUT, and then generate test cases. The key points of MBT are the modelling behaviour of the SUT for test generation, the test generation strategies and techniques, and the concretization of abstract tests into concrete, executable tests. The second is a technique that aims to minimize the software risks and testing problems. RBT consists of a set of activities regarding risk factors identification associated to software requirements. Once identified, the risks are prioritized according to its likelihood and impact and the test cases are projected based on the strategies or approaches for treatment of the identified risk factors. The test efforts are continuously adjusted according to the risk monitoring. Based on our study between MBT and RBT techniques we are identifying the following research tasks in the area of model-based testing and risk based testing viz; Proposition of a meta-model that represent model-based testing technique; Proposition of a meta-model that represent risk-based testing technique; Proposition of a novel testing approach based on model based testing and risk based testing techniques to overcome some testing limitations; and Make some case studies by applying the novel testing approach to obtain empirical results and compare our approach over existing approaches.