A Literature Review of the Integration of Test Activities into the Product Development Process

Abstract

The purpose of this paper is to investigate product development test processes. A literature review examines research on test activities in product design, product development and systems engineering research fields. The publications reviewed have been categorized based on the stage in development and placed into a proposed test process framework. The proposed framework sets an agenda of functions and characteristics important for the integration of test processes into model-based systems engineering. The findings presented are of interest to researchers by structuring test activities from product development, systems engineering and prototyping research into a context for the design process. The findings also allow practitioners to identify research at the level of planning and development stage relevant to their test processes.

1 Introduction

Testing is an essential component of the development process; however, it’s integration into the design process has received limited attention (Engel 2010; Tahera et al. 2018). A test of a design provides the possibility either to confirm a rationale or to learn from the discovery of unknown (and unexpected) outcomes, offering vital information to the design process. From this viewpoint, the purpose of a test in product development can be considered as a method to reduce uncertainty (Bjorkman et al. 2013). The reduction of uncertainty supports decision-making at any stage in development, but the value of information and new knowledge is inversely proportional with time (Kennedy 2008). Reduced uncertainty offers two outcomes. If the performance is as expected, the result is evidence which confirms quality in a context. If the performance is not as expected (or expectation unknown), the result provides a pathway to new understanding.

Experience tells us that test activities incur a significant financial burden, typically accounting for a substantial proportion of total development costs (Tahera et al. 2017). In contrast, even greater costs from a late-stage failure are often attributed to the deficiency of untested decisions (Kukulies and Schmitt 2018). The planning of test activities during development requires carefully balancing the potential benefits of new knowledge relevant to risk mitigation against the applicable programmatic constraints.

Model-based systems engineering (MBSE) promotes the utilization of models throughout the development process, from needs analysis and requirement definition to the end of a product life cycle, to enhance development (INCOSE 2007). Such models offer the potential for the explicit connection of test processes to design, supporting traceability to both knowledge gains and risk mitigations. Model-based testing has arisen from the need to test formalized models generated from analysis and simulations. The research on how MBSE and model-based testing should be integrated, especially in a context outside of software development, is lacking. Raz et al. (2018) have used Design of Experiments methodology to link system architecting and the system design space through formal models showing the potential for the further integration of test processes. In this study perspectives from research on test activities of products and systems are presented to support the integration of testing perspectives into MBSE.

The motivation of this paper is to orient the reader to research into test activities and to provide a synthesis of research into test processes related to early development stages. A literature review has been conducted to classify research on test activities based on their aims, perspectives to planning and stage in development. The objective of the review is to develop a framework of test activity that supports integration into the development process. Two research questions were proposed as a foundation for establishing the framework.

• 1. In which stages of product development have the integration of test activities into the development process been researched?

• 2. What perspectives are taken in the planning of test activities?

The paper is structured as follows: Section 2 Methodology presents the literature search strategy; the findings are given in Sect. 3 Results, analysed in Sect. 4 Discussion and summarized in Sect. 5 Conclusion.

2 Methodology

2.1 Search Strategy

The review was performed with guidance from procedural literature by Machi and McEvoy (2016). The SCOPUS electronic database was searched for any combination of product development keyword AND testing keyword shown in Table1. The search was conducted within the title, abstract and keywords. The search was limited to the journal sources listed in Table 2 with no restriction on year of publication. The sources were selected due to their focus on technical engineering design and their high reputation within the field, but are not considered exhaustive. Sources focused on engineering management were excluded to concentrate on the technical implementation of testing rather than related business and management topics.

Table 1 Keyword search terms

Table 2 Journal sources

2.2 Inclusion and Exclusion Criteria

The discovered articles were processed in three subsequent steps to remove articles not relevant to the intention of the study. First articles were removed based on title, then abstract and finally after reading the paper. The sample population for each stage is shown in Fig. 1 with a final population of 34 articles.

Fig. 1

Search processing stages

The article was included if the research addressed:

• The engineering design process, new product development practice, system engineering/design, development activities or design approach.

• Test activities or processes were integral to the paper’s research question/thesis.

• Test activities were discussed in relation to the design process (e.g. use of test results contributing to design process/decisions).

• Tests were analysed as a method of verification/validation or source of discovery/learning/reusable knowledge.

The article was excluded if:

• “Test” used in reference to testing paper’s hypothesis and not product development test activity. This includes the “testing” of a new design method – if test activities are not relevant to that design method.

• Studies addressing solely software products or construction projects.

• Studies focusing solely on the design improvement of a specific product for the benefit of that specific product – as opposed to development process/design methodology in general.

• “Design methodology” referred to as the methodological design of the study itself (i.e. not product development methodology).

• Virtual prototype, virtual testing, simulation or analysis research not discussed in a context of impact on the design/development process (i.e. research into improving specific modelling technique for a design problem was not included).

3 Results

Results from the literature review are summarised in the following two subchapters. First, the number of articles addressing each stage in development is presented, followed by their categorization into a model of perspectives on product development testing. Appendix A provides a complete list of the reviewed literature showing the classification of each article according to the presented frameworks.

3.1 Stage of Product Development

The research on test activities identified in this paper has been categorized with respect to the stage of the development process being investigated. Four stages, unique in respect to test activities, have been defined by the authors covering the fundamental stages from textbook literature (Ulrich and Eppinger 2012). The stages are concept generation, detailed design, product qualification and in-service life. The number of articles addressing each stage in development is shown in Fig. 2.

Fig. 2

Stage of product development in which testing was being investigated

A test process is not necessarily unique to only one stage in development. It is possible for research to be conducted solely on testing during concept generation (one stage) or with a broader perspective of the development process covering multiple stages. Figure 2 therefore includes boxes spread across the applicable stages with the number of articles for each box specified.

3.2 Test Process Perspective

The perspective of testing that was studied was categorized into three areas of activity suggested by the literature. In test design, the research addressed the best way to perform a specific test. A second area was defined as test objectives and focused on determining what to test. Finally, test strategy is where a test campaign of defined test objectives was studied for optimization. The number of articles discovered for each perspective is shown in Fig. 3.

Fig. 3

Test process perspectives

4 Discussion

Splitting the development process into stages allows the objective of testing in each phase to be discretized. In essence, testing with respect to stage in development can be considered as progressive investigations to discover: Will the idea work? Will the solution work? Does the product/system work? And finally how well did the product/system work? The greatest number of research articles addressed the earliest phase, concept development, which reflects the importance of starting with the right idea by frontloading activities and generating knowledge when it is most valuable. Whilst the understanding and maturity of the product is often limited in the concept phase, expanding the potential to answer the subsequent questions as early as possible achieves compounding benefits. Furthermore, it is specifically these compounding benefits that are the focus of integrating test activities into the development process.

Three test process perspectives are proposed as relevant for the integration of test activities in the development process. The following discussion highlights the key insights from the identified research from each perspective. The purpose is to distil the important considerations for the integration of testing into the development process.

4.1 Test Strategy

The category “test strategy” considers a holistic perspective of test activities during development. It concerns overall aspects of development, such as the duration, cost, quality or risk management. This requires analysing the test activities collectively and establishing the appropriate approach.

The systems engineering processes of verification and validation align with this perspective. Testing is technically a method of verification and validation. However due to testing’s critical role in the process, it is considered a test strategy in the context of the proposed framework. Several articles were discovered modelling the set of activities in a verification and validation plan to compare and query development approaches.

Engel and Barad (2003) and Engel and Shachar (2006) developed a quantitative methodology of modelling the cost and duration of activities in a verification plan. Subsequently, Hoppe et al. (2007) performed a multi-case study analysis of the quantitative methodology, along with qualitative guidelines, and showed how frontloading of test verification activities was critical for the development life cycle. This advocates for the close integration of test strategy in the design phase. Similar models, which estimate the characteristics of a set of test activities for the mathematical assessment of optimal solutions, have been developed by Salado (2015) and Shabi et al. (2017). These approaches achieve optimized test strategies for given designs which, although useful for comparison of different designs, do not influence directly the design process. They view an optimized test strategy as output for a given design problem. This is similar to Tahan and Ben-Asher (2005) who investigated specifically the ideal number of incremental stages for verification for a design.

Two studies which did explicitly integrate test strategy into the design process were (Tahera et al. 2018) and (Shin et al. 2017). Tahera et al. (2018) addressed the importance of incremental stages of testing with direct dependences to design iterations, and Shin et al. (2017) modelled the sequence of design tasks and test activities to establish the process with the shortest duration. These studies emphasized the efficiency and effectiveness to overall development which testing can provide when considered early and throughout development.

A different area of research concerning test strategy was research on prototyping. The following three studies examined the role of prototypes on a strategic level: Barkan and Iansiti (1993), Camburn et al. (2015) and Lauff et al. (2018). They all addressed aspects of using prototypes throughout development, such as timing, scope and prototype characteristics, to understand how they influenced the process.

4.2 Test Objectives

Articles in this category investigated the objective of specific test activities. Three general approaches were discovered in the identified research. Test objectives could be defined from either (i) evaluation of design uncertainty, (ii) supporting virtual/simulation activities or (iii) leveraging benefits of physical tests.

Goh et al. (2007) proposed a method of modelling uncertainty to inform design decisions. The method addressed the trend of increased analysis during development by structuring the uncertainty of design decisions based on untested assumptions. In a similar manner, Bjorkman et al. (2013) and Kukulies and Schmitt (2018) have both established test objectives by evaluating the performance uncertainty of functional product characteristics. Whilst those studies prioritized attention to design parameters with greatest uncertainty, Sanders et al. (2013) examined unforeseen and low-probability aspects with high consequences. These potential high-consequence characteristics required discovering them in very early testing before necessary changes were unfeasible.

Another key area of focus within the test objectives category was physical versus virtual testing. Research in this category was addressing directly the role of simulation to reduce the need for physical testing or prescribing physical testing for model correlation (to achieve even greater use of virtual models). Sutcliffe and Gault (2004) exposed potential benefits that can be achieved by integrating virtual tests, from CAD models to augmented reality, highlighting possible objectives during development from such test methods. Mejía-Gutiérrez and Carvajal-Arango (2017) reported on the latest integration of development software in a case study directly linking virtual prototypes into systems engineering modelling software.

The final area of study defining objectives for test activities focused on leveraging the benefits of physical testing. Viswanathan has published a number of studies (Viswanathan et al. 2014; Viswanathan and Linsey 2012, 2013) investigating the effect of design fixation and sunk costs on physical models. Design fixation considers an unnoticed integration of test activities, which prevents improved solutions being discovered. Campbell et al. (2007) on the other hand showed that physical models provide the best understanding to the customer and therefore allow the best feedback to be gathered.

The research into test objectives considered a greater level of integration into the design process than the other framework perspectives due to the strong link between key design parameters and definition of tests.

4.3 Test Design

Research in the final category, “test design,” investigated methodology for a defined objective. This category would be broad and extensive if it was to consider methodology for specific applications, e.g. what test methodology is best for measuring the health of batteries or the performance of passive dampers. However, the nature of such application-specific methodology has been excluded, according to the criteria defined in Sect. 2.2, as it is independent to the development process (or at most only applicable to development of a specific product type).

The eight articles identified in this category discuss the methodology in relation to the development process which means the methodology is generalizable to different development contexts. Two general areas of test design were discovered. The first was a Design of Experiments or Robust Engineering approach where the test process is structured statistically around the identification of parameters that will have greatest impact on performance. The second topic was related to understanding customer needs or, in a wider context, stakeholder analysis.

The impact of Design of Experiments theory on the design process was presented in case studies by Herrmann (2007, 2009). The case studies show how implementation of Taguchi methodology identifies the parameters with greatest effect on performance. This allows for informed concept tradeoffs to be performed based on maximizing intended performance and minimizing undesired effects.

The second area of test design methodology researched addressed the less quantitative field of stakeholder analysis. Research by Deininger et al. (2019), Starkey et al. (2019) and Wall et al. (1992) all investigated methodology relating to the influence of product representations on conclusions gathered. In contrast, Tovares et al. (2014), Artacho et al. (2010) and Engelbrektsson and Söderman (2004) emphasized the importance of capturing stakeholders’ perceptions and preferences.

5 Conclusion

A broad search was conducted on a limited set of prominent sources within product development research. The resulting literature was not exhaustive for the discovered topics yet achieved a wide overview covering all stages in the product development process. The purpose of the review was to uncover the literature addressing the definition and utilization of testing throughout the development life cycle.

The literature review identified methods from the start of the development process to qualification testing and in-service life. This answers the first research question by showing that integration of test activities is important and has been considered in all stages of development. Research focusing on the integration of testing was most prominent in concept development where the greatest impact is achievable.

Answering the second research question, What perspectives are taken in the planning of tests?, prompted the first author to devise a framework of test planning perspectives that established three areas of importance. The research was categorized based on the contribution in defining: what key objectives can be realized by testing, how such tests should be designed and how to establish the overall strategy of test activities in development. The proposed framework with classification into three areas provides insight into different levels of detail needed during planning of tests in the development process. The dependencies and overlap between these groups highlighted both their sequential and iterative nature as represented in the framework in Fig. 3.

This study provides an overview of relevant research structured in a framework to assist the future analysis and development of test processes for integration into MBSE approaches.

Appendix A. Literature Review Study Sources, Evidence Categories and Aims

Study	Evidence categorya	Stage of developmenta	Aim
Salado and Kannan (2019)	TS	2	Formalize the application of Bayesian networks to verification problems to facilitate instruction and communication among verification engineers and with researchers from other domains
Tahera et al. (2018)	TS	1–4	Establish importance of testing in product development to inform the development of pragmatic support methods
Shabi et al. (2017)	TS	2,3	Propose a method for determining the optimal verification activities with respect to product quality/risk
Shin et al. (2017)	TS	3	Demonstrate that model-based integration of T&E process and system safety process reduces development time
Salado (2015)	TS	3	Demonstrate the benefit of trade space exploration in the optimization of test strategy
Hoppe et al. (2007)	TS	1–4	Develop a generic verification, validation and testing methodology guideline and an economic VVT process model in order to realize improved product quality
Engel and Shachar (2006)	TS	2,3	Measure systems quality cost/times in a typical development project as well as suggest ways to optimize it in order to meet business objectives
Tahan and Ben-Asher (2005)	TS	3	Demonstrate that incremental integration offers both time and cost benefits vs single stage integration
Engel and Barad (2003)	TS	3	Propose a novel approach for modelling VVT strategies as decision problems
Lauff et al. (2018)	TS	1–3	Define the roles of prototypes in industry
Camburn et al. (2015)	TS	1,2	Provide a method to repeatedly enhance the outcome of prototyping efforts
Barkan and Iansiti (1993)	TS	1–4	Examine roles which prototyping plays in product development
Isaksson et al. (2000)	TS	1–3	Evaluate alternative design strategies and methods with respect to their impact on the development process time
Kukulies and Schmitt (2018)	TO	3	Investigate the use of uncertainty modelling to support design verification
Bjorkman et al. (2013)	TO	3	Present a methodology that uses an MBSE framework and Monte Carlo simulation to define uncertainty reduction goals for test planners to use in developing test strategies and detailed test designs for evaluating technical performance parameters
Sanders et al. (2013)	TO	1	Propose model for discovery of low probability events in the formative stages of the requirements definition and risk management planning activities in order establish the safety requirements and responsive conceptual designs for mitigation
Goh et al. (2007)	TO	4	Create framework for organizing uncertainty in product development simulation results therefore improving understanding between simulations and tested results for the purpose of assisting design decisions
Takala (2005)	TO	1	Propose a concept that bridges the gap between physical and virtual domains prototyping
Sutcliffe and Gault (2004)	TO	1	Propose guidelines for configuring virtual engineering technology and design of requirements analysis sessions
Mejía-Gutiérrez and Carvajal-Arango (2017)	TO	1,2	Investigate the usefulness of integrated virtual design verification simulation
Kiefer et al. (2004)	TO	1,2	Present the design development of a new product that explored many of the different prototyping technologies
Wang and Chen (2011)	TO	1	Introduce users’ participation in the conceptual design stage of product development to avoid interpreting biased from marketers’ information
Viswanathan et al. (2014)	TO	1	Study how physical models can assist novices in mitigating design fixation on undesirable features
Viswanathan and Linsey (2013)	TO	1	Investigate physical modelling role in idea generation and design fixation
Viswanathan and Linsey (2012)	TO	1	Investigate if physical models supplement designer’s mental models and if physical models induce design fixation
Campbell et al. (2007)	TO	1–3	Demonstrate that physical models are the single presentation format that is readily understood by most customers
Wall et al. (1992)	TD	1–4	Develop a systematic method of evaluating prototyping processes in order to determine the best process for a given situation
Engelbrektsson and Söderman (2004)	TD	1	Investigate the use and perceptions of methods and product representations in Swedish companies and its possible impact on problems associated with late-discovered customer requirements
Starkey et al. (2019)	TD	1	Investigate the impact of prototype fidelity, concept creativity and risk aversion on perceived riskiness and concept selection
Deininger et al. (2019)	TD	1	Provide insights into how prototype type, group membership (stakeholder characteristics) and question type can influence stakeholders’ perceptions of a design concept and the resulting feedback they provide
Tovares et al. (2014)	TD	1	Develop a method to elicit, capture and model consumer preference through experiential preference judgements
Artacho et al. (2010)	TD	1	Analyse how slight changes might affect users’ perception as well as influence their intention to purchase a product
Herrmann (2009)	TD	2	Demonstrate that the successful use of Taguchi test methodology provides efficient and reliable design knowledge
Herrmann (2007)	TD	2	Demonstrate the successful use of Taguchi test methodology to support system design

1. ^a TS test strategy, TO test objectives, TD test design, 1 concept generation, 2 detailed design, 3 product qualification, 4 in-service life