1 Introduction

This paper provides guidelines for the selection of experimental strategies in parameter design. This paper focuses on two alternatives, one-at-a-time plans and orthogonal arrays. The strength of interactions between variables and the degree of pure experimental error are found to be the two most important characteristics in determining the choice of experimental design. If these characteristics can be estimated a priori, the methods in this paper allow one to determine which experimental strategy will most likely provide the greatest benefit.

The term "parameter design" here denotes the process of selecting nominal values for the set of design parameters. Parameter design can be undertaken to optimize robustness or to optimize the nominal response. Parameter design is necessarily preceded by concept design, which defines the set of design parameters, and is often followed by tolerance design, which defines allowable variations of the design parameters about the nominal values.

At present, parameter design is usually carried out using a range of experimental arrays. These arrays are developed on a statistical basis to achieve desirable properties such as balance, orthogonality, and rotatability, which are important for maximizing the information gained from a limited number of experimental trials. A surprising result of this paper is that a simple adaptive experimental plan will, under some conditions, provide greater improvements on average even though it is less statistically efficient and yields less information to explain the improvement. More discussion of experimental designs and related research is presented in Sect. 2.

The structure of this paper is as follows: Sect. 2 provides some background on one-at-a-time experiments and orthogonal arrays as well as research on these experimental designs; Sect. 3 describes the research methodology used; Sect. 4 presents the results of the investigation; and Sect. 5 presents a discussion of the results with an emphasis on recommendations for use of these data.

2 Background

2.1 Adaptive one-at-a-time experiments

In a one-at-a-time plan, the experimenter seeks to gain information about one factor in each experimental trial [33]. This procedure is repeated in turn for all factors to be studied. In an adaptive one-at-a-time plan, the experimenter also seeks to optimize the response along the way. The adaptive one-at-a-time plan studied in this paper is described by the rules below:

  • Begin with a baseline set of factor levels and measure the response.

  • For each experimental factor in turn:

    • change the factor to each of its levels that have not yet been tested, keeping all other experimental factors constant;

Table 1 presents an example of this adaptive one-at-a-time method in which the goal is to improve the response displayed in the rightmost column, assuming that larger is better.Footnote 1 Trial 2 resulted in a rise in the response compared to trial 1. Based on this result the factor A is held at level 2 throughout the rest of the example. In trial 3, only factor B is changed, and this results in a drop in the response. The adaptive one-at-a-time method requires returning factor B to level "1" in the trial in which factor C is modified. In trial 4, factor C is changed to level 2. It is important to note that although the response rises from trial 3 to trial 4, the conditional main effect of factor C is negative because it is based on comparison with trial 2 (the best response so far). Therefore factor C must be reset to level 1 before proceeding to toggle factor D in trial 5. The procedure continues until every factor has been varied. In this example, the preferred set of factor levels is A=2, B=1, C=1, D=2, E=1, F=1, G=1, which is the treatment combination in trial 5.

Table 1. An example of the adaptive one-at-a-time method
Full size table

The adaptive one-factor-at-a-time method requires n(k−1)+1 experimental trials given n factors each having k levels. The method provides estimates of the conditional main effects of each experimental factor but cannot resolve interactions among experimental factors nor provide a guarantee of identifying the optimal factor settings. Both random experimental error and interactions among factors may lead to a suboptimal choice of factor settings.

2.2 Orthogonal arrays

An alternative to one-at-a-time experimentation emerged with the development of factorial experimental design methods, starting in the 1920s through the work of R.A. Fisher. An orthogonal array is an important class of experimental designs that emerged from this work. An example is the OA8(27) Footnote 2 depicted in Table 2. Unlike the one-at-a-time designs, the orthogonal array is "balanced": all of the levels of each factor are represented in the array an equal number of times.

Table 2. An orthogonal array OA8(27)
Full size table

The OA8(27) array depicted in Table 2 allows one to estimate the main effects of seven two-level factors in eight experiments. In general, orthogonal arrays require n(k−1)+1 experimental trials given n factors each having k levels, the same scaling as for one-at-a-time designs. Although orthogonal arrays require the same number of experiments as one-at-a-time plans, they provide greater precision in effect estimation. The variance of the factor effect estimates is proportional to the inverse of the replication number. For example, the OA8(27) provides factor effect estimates with one-quarter of the variance as the one-at-a-time plan because each factor is held at each level four times.

2.3 Prior work on orthogonal arrays and one-at-a-time designs

Orthogonal arrays are recommended within the statistics and design methodology literature for several uses:

  • For screening experiments, in which the objective is to reduce the list of candidate variables to a small number so that subsequent experiments can be more efficient [18].

  • For analyzing systems in which one is confident that the experimental factors do not interact strongly. Some call orthogonal arrays "main effect plans" [4].

  • For use in robustness optimization in which the objective is to induce noise via an "outer" orthogonal array while searching for robust parameter settings via an "inner" orthogonal array. This approach is recommended by Taguchi and several other authors [30, 21, 7], but is not preferred by others [33, 14].

Use of one-at-a-time plans is generally discouraged by modern texts on experimental design and quality improvement [33, 14]. Reasons cited include:

  • More runs are required for the same precision in effect estimation.

  • Some interactions between variables cannot be captured.

  • The conclusions from the analysis are not general (i.e. only conditional main effects are revealed).

  • Optimal settings of factors can be missed.

  • One-at-a-time plans essentially rule out the possibility of randomization and can be susceptible to bias due to time trends.

While the cautions mentioned above should be taken into account in considering the use of a one-at-a-time plan, the data shown in this paper suggest that there is a role for one-at-a-time plans and that they are more effective than orthogonal arrays under certain conditions. A key question, for which the answer does not appear to exist in quantitative terms, is thus under what specific conditions should one make use of this approach. For example, Friedman and Savage [8] suggested that one-at-a-time plans are useful for optimization, but provided no criteria for deciding when to employ them. Daniel [6] suggested that one-at-a-time plans should be limited to those cases in which factor effects are of magnitude 4σ or more. However, the literature contains no empirical or theoretical validation of this "4σ rule" nor an analysis of the coupled effects of pure experimental error and interactions. This paper addresses these issues by providing a quantitative guide for selection between orthogonal arrays and one-at-a-time experiments.

3 Research method

To demonstrate the role of one-at-a-time experiments we simulate the optimization of 66 different responses of 27 different engineering systems that span a wide range of disciplines, including mechanical, electrical, materials, civil, and chemical engineering. A brief description of the responses and the associated engineering systems is included in the Appendix. Each of these systems was examined by analysis of a full-factorial study presented in the science and engineering literature. Comparison is made between one-at-a-time and orthogonal array results in terms of average improvement in a response.

Full factorial studies were used for two reasons. First, they allow the strength of interactions to be assessed and compared to the strength of main effects. The way that we quantify interaction strength is described in Sect. 3.1. Second, they include as a subset all possible orthogonal array plans and all possible one-at-a-time plans. This enabled simulation of both of these methods as described in Sects. 3.2 and 3.3.

3.1 Quantifying strength of interactions

Strength of interaction is quantified in this paper by analysis of the sum-squared variations in the data. In general, the sum-squared variations in an experiment can be computed by the formula

$$ SS = \sum\limits_{i = 1}^n {(y_i - \bar y)^2 } , $$
(1)

where n is the number of experimental observations, y i is the measured response on the ith observation, and \( {\bar y} \) is the average response. This sum-squared variation has two components: one due to experimental error and one due to the factor effects or treatment conditions. This sum-squared variation from all factor effects in an experiment can be computed by the formula

$$ SS_{{\rm FE}} = {{\rm n} \over {\rm p}}\sum\limits_{t = 1}^p {(\bar y_t - \bar y)^2 } , $$
(2)

where t represents a treatment condition or row number in the experimental design, \( {\bar y_t } \) represents the mean response at that treatment condition, and \( {\bar y} \) represents the overall average response across all treatment conditions.

The sum square from factor effects can be further decomposed into two contributing factors, the sum square from main effects and the sum square from interactions.

$$ SS_{{\rm FE}} = SS_{{\rm ME}} + SS_{{\rm INT}} . $$
(3)

The sum-squared variations from main effects include linear terms and, for the three-level factorial experiments, simple quadratic terms as well. The sum of squares from interactions is the sum of all interactions, including two-way, three-way, and higher-order interactions. In this paper, strength of interactions is characterized by the ratio

$$ {\rm Interaction}\;{\rm Strength} = {{SS_{{\rm INT}} } \over {SS_{{\rm FE}} }} = {{SS_{{\rm FE}} - SS_{{\rm ME}} } \over {SS_{{\rm FE}} }}. $$
(4)

This ratio can range from 0 to 1. A value of 1/2 implies that the summed effects of interactions are the same size as the summed main effects. Given the method of calculation above, all sum-squared variations from treatment conditions not from main effects were allocated to factor effects. By adopting this approach, we implicitly assumed that the mean response at any given treatment condition represents the best unbiased estimate of the system response at that treatment condition. (An alternative approach would be to invoke the sparsity of effects, hierarchy, and inheritance principles to remove some interactions from our model and pool their sum-squared variations as pure experimental error.) It is recognized that our choice of assumptions may lead to overestimating the strength of interactions.

3.2 Simulating orthogonal arrays

An orthogonal array approach to improving an engineering system was described in Sect. 2.2. This approach was simulated on the basis of published full factorial data by means of the following steps: (a) select the smallest fractional factorial design of at least resolution III that accommodates all the factors and levels. (b) Then for e=0 to 1 in steps of 0.1 (e is a variable used to modulate the strength of pure experimental error):

  • for 1000 trials:

    • Form a random permutation of the factorial design by random assignment of factors to columns and levels to coded levels.

    • For each row of the factorial design: (a) look up the mean response for the treatment condition from the table of data from the full factorial experiment. (b) To the mean response add a normally distributed random value with zero mean and standard deviation of \( e\sqrt {{{SS_{{\rm FE}} } \mathord{\left/ {\vphantom {{SS_{{\rm FE}} } n}} \right. \kern-\nulldelimiterspace} n}} \).

    • Compute the estimated main effects of the factors based on analysis of means of the simulated data.

    • Select the levels of the factors that correspond to improved response.

    • Look up the response corresponding to the selected factor levels (without adding noise).

    • Normalize the response from the previous step by subtracting the average of all the full factorial data and dividing it by the difference between the maximum mean response at all treatment conditions and the average of all the full factorial data.

    • Store the normalized response.

    • Perform the next trial.

  • Compute the mean and standard deviation of the stored normalized response for the current value of the degree of pure experimental error e.

  • Next e.

Finally, graph the mean and standard deviation of the normalized response versus e.

Some explanation may be required regarding the method of adding simulated experimental error. The formula for the standard deviation of the noise was \( e\sqrt {{{SS_{{\rm FE}} } \mathord{\left/ {\vphantom {{SS_{{\rm FE}} } n}} \right. \kern-\nulldelimiterspace} n}} \) where SS FE is the sum-squared variations from factor effects (computed using Eq. (2) and the published data for that engineering system), n is the total number of observations published for that engineering system (see Eq. (2)), and e is a parameter that allows the amount of experimental error to be adjusted. This formula was chosen so that when e is unity, the mean-squared variations from pure experimental error will be equal to the mean square variation from factor effects. This paper will use the parameter e on the abscissa of many plots. It can be interpreted as a ratio of the strength of pure experimental error to the strength of factor effects.

3.3 Simulating one-at-a-time plans

An adaptive one-at-a-time approach to improving an engineering system was described in Sect. 2.1. This approach was simulated on the basis of published full factorial data by means of the following steps for e=0 to 1 in steps of 0.1 (e is a variable used to modulate the strength of pure experimental error):

  • for 1000 trials:

    • Select a random order for varying the controllable factors.

    • Select at random a starting point for each factor level.

    • Look up the mean response for the treatment condition from the table of data from the full factorial experiment.

    • To the mean response add a normally distributed random value with zero mean and standard deviation of \( e\sqrt {{{SS_{{\rm FE}} } \mathord{\left/ {\vphantom {{SS_{{\rm FE}} } k}} \right. \kern-\nulldelimiterspace} k}} \).

    • For each experimental factor in turn: (a) change the factor to each of its levels that have not yet been tested while keeping all other experimental factors constant. (b) Look up the mean response for the treatment condition from the table of data from the full factorial experiment. (c)To the mean response add a normally distributed random value with zero mean and standard deviation of \( e\sqrt {{{SS_{{\rm FE}} } \mathord{\left/ {\vphantom {{SS_{{\rm FE}} } k}} \right. \kern-\nulldelimiterspace} k}} \). (d) Retain the factor level that provided the best "simulated" (i.e. noisy) response so far.

    • If the number of experimental conditions simulated in this trial is smaller than that required for the smallest resolution III factorial design, then return to the first factor in the randomly selected order and continue the process in the preceding step until the number of experimental conditions simulated in this trial is equal to that required for the smallest resolution III factorial design.

    • Look up the response corresponding to the selected factor levels (without adding noise).

    • Normalize the response from the previous step by subtracting the average of all the full factorial data and dividing it by the difference between the maximum mean response at all treatment conditions and the average of all the full factorial data.

    • Store the normalized response.

    • Next trial.

  • Compute the mean and standard deviation of the stored normalized response for the current value of the degree of pure experimental error, e.

  • Next e.

Finally, graph the mean and standard deviation of the normalized response versus e.

One aspect of the algorithm above requires further amplification and justification. Depending on the number of experimental factors and levels, a simple one-at-a-time plan can require fewer runs than the smallest available orthogonal array. For example, for six two-level factors, the smallest available orthogonal array requires eight runs, while the one-at-a-time plan only requires seven runs. One of the steps listed above adds extra runs to the simple one-at-a-time plan until it uses an equal number of runs as the smallest resolution III factorial design. This feature was added to the algorithm above in order to make the comparison between the two methods more consistent. The cost of carrying out the one-at-a-time plan should be equal to the cost of carrying out the orthogonal array plan.

4 Results

This section presents the results of applying the methods described in Sect. 3 to the data from the full factorial experiments. Section 4.1 presents detailed results from 1 of the 66 responses to show the kind of results the method provides. It is not practicable to present such a detailed analysis of every data set in the sample, but a summary of the results from all the 66 responses follows in Sect. 4.2.

4.1 Example results

The example system chosen for presentation is a detonation spray process for alumina coatings [22]. This example is particularly clear and generally representative of the results obtained across the sample of 66 responses. The detonation spray process is depicted in Fig. 1. Alumina powder, carrier gas, and hydrocarbon gas enter one end of a barrel. A spark plug detonates the mixture, sending the gases out of the end of the barrel and onto the work piece to be coated. The responses of interest were porosity, roughness, hardness, and wear rate, but in this section we analyze only the porosity of the coating, expressed as percent porosity. The smaller the porosity, the better. There were four controllable factors, each having two levels as listed in Table 3.

Fig. 1.
figure 1

A detonation spray process for alumina coatings

Full size image
Table 3. Controllable experimental factors of the detonation spray process
Full size table

Saravanan et al. [22] carried out a full factorial (24) experiment, and the results are listed in Table 4. The strength of interactions (Eq. (4)) can be computed via analysis of variance of a linear regression model. The sum of squares due to linear regression is 107.6, and the total sum of squares due to variation about the mean is 127.4. The published full factorial is unreplicated. We assumed that the residual error (19.8) is attributable to interactions rather than pure experimental error, making the interaction strength ratio (Eq. (4)) equal to 0.16.

Table 4. Experimental results from the detonation spray process
Full size table

Simulations were conducted of orthogonal array methods as described in Sect. 3.2, and the resulting mean performance is represented by squares in Fig. 2. The strength of simulated pure experimental error (the parameter e) is the abscissa, and the ordinate is the normalized improvement. The standard deviation divided by 5 is indicated by error bars. The orthogonal array provided roughly 75% of the possible improvement on average over the range of pure experimental error from 0 to 1, although there was a weak decrease in improvement for e greater than 0.5.

Fig. 2.
figure 2

Results of simulating one-at-a-time and orthogonal array plans applied to porosity in the detonation spray-coating process

Full size image

To better understand these results, it is helpful to consider some details in the data not presented in Fig. 2. When the pure experimental error was zero, the orthogonal array consistently predicted that the preferred level of Ratio was +1, the preferred level of Freq was –1, and the preferred level of Dist was –1 (Table 4). However, the preferred level of Ratio as predicted by the orthogonal array was not consistent; it depended on which orthogonal fraction of the full factorial was used. The orthogonal array plan led to a porosity of either 2.13 or 2.17 (Table 4). The mean porosity after optimization was 2.15. Since the average porosity in Table 4 is 6.03 and the best porosity is 1.02, the normalized improvement was consistently 75% and never 100%. The orthogonal array consistently failed to reach the optimal porosity of 1.02.

Simulations were conducted of the adaptive one-at-a-time method as described in Sect. 3.3, and the resulting mean performance is represented by diamonds in Fig. 2. When the pure experimental error was zero, the one-at-a-time plan yielded values of porosity of 1.02, 2.13, or 2.17, depending on the choice of random order of the factors. The mean porosity (averaged over all trials) was 1.5, which is 90% of the possible improvement from the average porosity of 6.03 to the best porosity of 1.02. As the pure experimental error was increased from 0 to 1, the normalized mean improvement in porosity dropped to 52%, and the standard deviation of the improvement rose by a factor of about 3.

The crossing point on Fig. 2 is of practical significance for the experimenter. For strengths of pure experimental error of about 0.6 or less, the one-at-a-time method provided greater improvements on average than the orthogonal array method. It is therefore important to understand whether this break-even point is consistent across many engineering systems and whether the location of such a point depends on other factors, like the strength and structure of interactions among the controllable experimental factors. This will be explored in the next section.

4.2 Results from the entire sample population

The interaction strength of each of the 66 responses for the entire sample was quantified according to Eq. (4). To present the results from the sample population, we found it useful to group the responses into four classes in terms of the ratio of the strength of interactions compared to the strength of the main effects. The categories of "mild," "moderate," "strong," and "dominant" interactions were defined as in Table 5. The particular ranges were chosen to be round figures and to subdivide the sample population into categories with roughly the same number of responses.

Table 5. Categories of interaction strength
Full size table

One way to summarize the data from this study is shown in Table 6. The number in each cell of the table is the ratio of times the one factor at-at-time method performed as well as or better than orthogonal arrays. For each cell, there is a category of interaction strength listed in the row heading and a degree of experimental error e listed in the column heading. The cells marked with an asterisk (*) indicate that the ratio is 0.5 or greater. The data show that for systems with mild or moderate interaction strength, it is advisable to use one at-a-time plans in preference to orthogonal arrays only if the experimental error is greater than 25% of the factorial effects. For systems whose interactions are strong or dominant, however, it is advisable to select one-at-a-time plans in preference to orthogonal arrays even if the pure experimental error is as great as the factorial effects.

Table 6. Probability of selecting a better design (or the same design) using one-at-a-time method (as compared to orthogonal arrays). Elements with an asterisk (*) indicate a probability of greater than 0.5
Full size table

Another way to present the data from this study is to graph the normalized improvement of the engineering system. Figure 3 shows the normalized improvement from the orthogonal array method versus the strength of experimental error with different interaction strengths plotted parametrically. Figure 4 presents the same type of graph for the one-at-time method. Table 7 summarizes the data in Figs. 3 and 4 in a way that simplifies selection between the two methods.

Fig. 3.
figure 3

Normalized improvement versus strength of experimental error for orthogonal arrays

Full size image
Fig. 4.
figure 4

Normalized improvement versus strength of experimental error for adaptive one-at-a-time

Full size image
Table 7. Map of average normalized improvement versus interaction strength and strength of experimental error. The first number is for one-at-a-time method. The second number is for orthogonal arrays
Full size table

When interactions are mild and experimental error is low, both one-at-a-time plans and orthogonal arrays lead to selection of the best set of control factor levels from among all the discrete alternatives. This is indicated on Figs. 3 and 4 in that the curve for systems with "mild" interactions intersects the ordinate at a normalized improvement of almost 1. As the degree of pure experimental error rises, the normalized improvement drops for both one-at-a-time plans and orthogonal arrays. The curves in Fig. 3 are less steep than those in Fig. 4, indicating that, on average, orthogonal arrays are less affected by pure experimental error than one-at-a-time plans. On the other hand, the spacing between curves is greater in Fig. 3 than in Fig. 4, indicating that, on average, orthogonal arrays tend to be more affected by interactions than one-at-a-time plans are.

5 Implications for engineering practice

The results in this paper suggest that to maximize improvements from small numbers of experiments one-at-time plans should be considered. The tables and figures of Sect. 4.2 are useful guides for deciding when to use one-at-a-time plans, especially if the engineering scenario can be characterized in terms of degree of pure experimental error and interaction strength. Judgments of pure experimental error may be made based on past experience, error analysis of the experimental apparatus, or by replicating some experiments. An a priori estimate of interaction strength may be based on prior experience with similar systems or a comparison of the system at hand with those listed in the Appendix. Once pure experimental error and interaction strength have been estimated, it is possible to locate the engineering system within a cell or range of cells of Table 6. The number in the table can be interpreted as the probability that the adaptive one-at-a-time method will provide better results than orthogonal arrays.

One barrier to using the results of this paper is that judging interaction strength a priori can be a difficult task. However, there are two mitigating factors. One is that, to use Tables 6 and 7, it is sufficient to classify an engineering system into rather coarsely grained categories (mild, moderate, strong, or dominant). The second mitigating factor is that, depending on the strength of pure experimental error, the choice of design may be determined independent of the strength of interactions. For example, when pure experimental error is very low, the one-factor-at-a-time design outperforms the orthogonal array for all categories of interaction strength.

To use the results of this paper, one should estimate the strength of pure experimental error. Three issues regarding pure experimental error should be noted:

  1. 1.

    Deterministic computer simulations of engineering systems do not have pure experimental error. Although the estimates from simulations include errors (rounding errors, discretization errors, etc.), the errors are not random—the simulation results are exactly the same for every replicated trial. Therefore, in applying the results of this investigation to optimization of computer simulations, it is probably best to consider the strength of pure experimental error to be zero. The data in Table 7 show that, when pure experimental error is zero, the adaptive one-at-a-time approach provides greater gains on average than the orthogonal array approach for all categories of interaction strength. A logical conclusion is that when optimizing a system by means of deterministic simulations, one-at-a-time plans are generally to be preferred when the number of simulation runs is limited to approximately the number of experimental factors.

  2. 2.

    Taguchi recommends using an outer array to explore different noise conditions for every setting of the controllable parameters in the inner array. The noises induced deliberately in this way are not pure experimental error. If an outer array of noise is replicated, then pure experimental error can be estimated by calculating the variance in the signal-to-noise ratio among the replicates. Increasing the distance between the factor levels in the outer array increases the amount of induced noise, but probably lowers the degree of pure experimental error in the signal-to-noise ratio because the induced noises can overwhelm the uncontrollable variations. We suggest that experimenters using outer arrays for robustness optimization should estimate strength of pure experimental error as 0.3 or less.

  3. 3.

    It is often possible to carry out robustness optimization more efficiently by including induced noises along with controllable factors in a single experimental array. Using an adaptive one-at-a-time approach for robustness optimization requires that robustness be assessed at each experimental step. Therefore, the outer array approach is required. Efficiency of an outer array approach can often be improved by compounding noise factors.

There are options other than orthogonal arrays or one-at-a-time plans that can also be considered in parameter designs. In this context Figs. 3 and 4 provide some additional information for deciding whether to consider higher resolution designs. After estimating interaction strength and degree of experimental error, the system at hand can be located on Figs. 3 and 4. The larger of the two values is an estimate of the expected fraction of possible improvement likely to be realized from a resolution III design, and it thus provides an estimate of the additional improvement one might make using a higher resolution design. The potential additional improvement can be weighed against the costs of the additional experiments and increased time to market.

A further attribute of one-at-a-time methods is that a project manager can carry out several experimental investigations simultaneously and dynamically allocate resources among them. Those that are not providing improvements can be canceled in favor of other investigations or new opportunities.

It is important to acknowledge the limitations of the results presented in this paper. One limitation is that the set of full factorial experiments used in the study included six or fewer experimental factors. The quantitative conclusions should not be applied to systems with more experimental factors. Another limitation is that, as mentioned just above, there are a variety of methods available other than the two alternatives explored here, although most of these require substantially more experiments. The results of this paper will be most useful when budget limitations or a need for quick time to market limit the number of experiments that can be carried out. Last, realized improvements in the response are not the only factors to consider in choosing an experimental design. In some cases, the knowledge gained through the experiment can be as useful as the improved performance itself. Nevertheless, when performance improvement is the primary purpose of the experimental effort, one-at-a-time plans will often be the best choice.

6 Future work

There are a number of extensions of the study that seem worthwhile to pursue. For example, most of the improvements provided by one-at-a-time plans occur within the first half of the experiment. As such one-at-a-time plans are potentially of even greater benefit in organizations where budget and schedule changes are likely to interrupt on-going experiments. A useful study might be to quantify the effects of interruptions on both one-at-a-time and balanced factorial designs. If the differences prove to be substantial, it would be an additional impetus to use adaptive one-at-a-time experiments in dynamic product development environments.

One-at-a-time plans may provide additional benefits if the experimental factors can be ordered a priori based on the expected size of main effects. It is thus of interest to quantify how well a typical product development team is able to order the experimental factors. An investigation based on simulations similar to the ones in this paper could then assess how much additional benefit this a priori knowledge provides through one-at-a-time experiments.

Another issue is how the interplay between the increased use of modeling and simulation and experiments factors into the attractiveness of experimental methods. For example, it is possible that one-at-a-time plans provide advantages when integrating a physics-based model with an experimental investigation because the model can provide predictions of each upcoming outcome. Each experimental trial can potentially lead to modifications of a physics-based model. This dynamic interplay between model and experiment is often an important part of engineering design.

Finally, the authors recommend investigations of the mechanisms by which one-at-a-time plans provide greater improvements on average than orthogonal arrays. Two possible mechanisms are that adaptive one-at-a-time plans exploit interactions on average even though they cannot resolve them, and that orthogonal arrays, on average, fail to fully exploit the main effects because of the confounding of main effects with interactions.