Design and Statistical Analysis of Multidrug Combinations in Preclinical Studies and Phase I Clinical Trials

Abstract

Multidrug combination is an important therapeutic approach for cancer, viral or microbial infections, hypertension and other diseases involving complex biological networks. Synergistic drug combinations, which are more effective than predicted from summing effects of individual drugs, often achieve increased therapeutic index. Because drug-effect is dose-dependent, multiple doses of an individual drug need to be examined, yielding rapidly increasing number of combinations and a challenging high dimensional statistical modeling problem. The lack of proper design and analysis methods for multi-drug combination studies have resulted in many missed therapeutic opportunities. Although systems biology holds the promise to unveil complex interactions within biological systems, the knowledge on network remains predominantly topological until very recently. This article summarizes recent work on efficient maximal power experimental designs on multi-drug combinations, and statistical modeling of the resulting data. The design and analysis of vorinostat and cytarabine combination study is presented to illustrate the approach. We then introduce a model based adaptive Bayesian phase I trial design for drug combinations utilizing the modeling concept. To tackle the challenging problem of combinations of more than three drugs, we present a novel two-stage procedure starting with an initial selection by utilizing an in silico model built upon experimental data of single drugs and current systems biology information to obtain maximum likelihood estimate.

1 Introduction

In the past decade the identification of a variety of novel signal transduction targets amenable to therapeutic intervention has revolutionized the approach to cancer therapy. These targets were identified based on improved understanding of the molecular mechanisms of action of second messengers, other components of signal transduction pathways and systems biology. These advances have also made available large number of potential agents and call for new quantitative approaches for combination therapy (Xavier and Sander 2010; Fitzgerald et al. 2006; Hopkins 2008), which then motivated the development of design and analysis methods for three drugs (Tan et al. 2009; Fang et al. 2015).

Multi-drug combination is an important therapeutic approach for diseases such as cancer, viral or microbial infections, hypertension and other diseases involving complex biological pathways. Synergistic drug combinations, which are more effective than expected from summing effects of individual drugs, offer the potential for improved therapeutic index. Because drug-effect is dose-dependent, multiple doses of an individual drug often need to be examined, yielding rapidly increasing number of combinations that prohibit experimentation, and yielding a challenging high dimensional statistical problem. The lack of proper design and analysis methods for multi-drug combination studies have resulted in suboptimal utilization research resource and missed therapeutic opportunities.

The past decade has seen significant progresses in developing proper design and analysis methods for multi-drug combination studies have increased the chances of identifying optimized combinations for further therapeutic opportunities for combinations of two, three, or more drugs utilizing optimized designs and systems biology (see, e.g., Tan et al. 2003; Fitzgerald et al. 2006; Fang et al. 2008, 2015, 2016; Calzolari et al. 2008) as well as adaptive phase I clinical trial designs that attempt to identify the best possible maximum tolerated doses through modeling of the joint dose-toxicity relationship (see, e.g., Yuan and Yin 2008; Yin and Yuan 2009a, b; Yang et al. 2016). The non-model based designs include the approach using a partial order of toxicity discussed in Wages et al. (2011) and a two-dimensional extension of the biased coin design (Sun and Braun 2015). These approaches are a welcome step-forward as they all have done away with the problematic assumption that the dose-limiting toxicity (DLT) increases monotonically with increasing doses. It is known this assumption is reasonable in single agent phase I trials, it may not hold in drug combinations since the ordering of the probabilities of DLT of these combinations is not known at the design stage of the trial.

This article is to review the development of the vorinostat (SAHA) combinations for leukemia from nonclinical studies to clinical trials. We then summarize recent statistical methods motivated by and used in the vorinostat development as well as lessons learned moving the therapy to clinic. Specifically, we present an efficient experimental design on selected multi-drug combinations, statistical modeling of the resulting data and the proof of its statistical properties. Drawing experience from the vorinostat studies, we present an adaptive Bayesian trial design for multidrug combinations with interaction modeling and an optimized design for multidrug combinations. We also discuss applications and areas that are likely to assume an important role in future drug discovery and development research, such as ways of dealing with the difficult high dimensional problem with multidrug combinations utilizing in silico models that integrate statistical modeling, experimental data of single drugs and current systems biology approach.

The rest of the article is organized as follows. Section 1 introduces the combination study of vorinostat and the efficient experimental design, the maximal power method for drug combinations using Loewe additivity. Section 2 describes the analysis of vorinostat and cytarabine (ara-C) combinations, how it affects the clinical trial design, and the clinical trial results, how it has impacted the development of methodology to design multidrug combination studies and how a potentially useful approach for phase I trial design that utilizes the modeling approach can be derived. Section 3 introduces the Bayesian adaptive phase I trial deign for drug combinations while modeling the interaction based on Bliss independence. Section 4 introduces current work on multidrug combinations of three drugs. Section 5 presents a novel two-stage procedure starting with an initial selection by utilizing an in silico model built upon experimental data of single drugs and current systems biology information to obtain maximum likelihood estimate by integrating modern statistical methods and systems biology approaches. We conclude with a discussion on the future of this field.

2 Vorinostat Combination Studies and Maximal Power Design

Vorinostat (suberoylanilide hydroxamic acid, SAHA) is a small molecule histone deacetylase (HDAC) inhibitor that is currently the most potent HDAC inhibitor available clinically. The vorinostat combination trial is a phase I trial to determine the maximum tolerated dose of vorinostat used in combinations of the mainstay of anti-leukemia chemotherapeutic agents (Gojo et al. 2013). To investigate the potential activity of the combination, extensive preclinical in vitro cytotoxicity studies on vorinostat combined with ara-C and etoposide as well as the sequence of administration have been performed to test the interaction (synergy or antagonism) of the combination for treating acute leukemia (Shiozawa et al. 2009). Ara-C is one of the most active agents available for treating acute leukemia. Etoposide has been shown to be an effective anti-leukemia agent, particularly when given in combination with other chemotherapeutic agents. It exerts its effects by interfering with topoisomerase II activity, binding to and stabilizing the covalent linkage between topoisomerase II and DNA, and inhibiting the re-ligation of the resultant DNA double strand breaks.

Experimental approaches to characterizing combination therapy typically involve determining dose–response curves for inhibitors individually and in combination. When experimental dose–response data match the predictions of Loewe additivity, the inhibitors are said to be additive (corresponding to the zero-interaction case); greater than predicted potency indicates synergism (positive interaction); and lower potency argues for antagonism (negative interaction). With different dose–response curves of individual inhibitors, various measurements for combinations have been developed according to Loewe additivity. Based on the median-drug effect analysis method which assumes that two drugs alone or in combination will result in sigmoid concentration-effect curves, Chou and Talalay (1984) defined a combination index. Assuming the dose–response curves of individual drugs can be characterized by Hill models, Greco and his colleagues proposed an equation to characterize interactions of two drugs (Greco et al. 1995). Peterson and Novick (2007) derived a nonlinear blending measurement for the assessment of combination drug synergy.

The very first set of experiments were conducted based on one fixed dose ratio of vorinostat of ara-C and etoposide, which missed the complexity of the interaction and precluded attainment of data on important interactions among these agents. Consequently, we have designed the study to include various combinations that are determined based on an efficient statistical design so that the statistical power to demonstrate the departure from additivity is maximized (Tan et al. 2003, 2009; Fang et al. 2008). Indeed it was shown that vorinostat interacted additively or synergistically with etoposide; but not with ara-C because vorinostat diminished cells in cell cycle S-phase, where cells are most vulnerable to ara-C toxicity (Shiozawa et al. 2009). However, the sequential administration of vorinostat followed by ara-C with a 72-h interval demonstrated synergy, where the time between administration of vorinostat and ara-C allows cells to re-enter into S-phase (Shiozawa et al. 2009; Gojo et al. 2013). This article focus on the vorinostat plus ara-C combination study to demonstrate the methodology.

To introduce the maximal power design (MPD) to detect departures of additivity, i.e., detecting synergy or antagonism, we first review how additivity, the expected dose effect when the two drugs are considered additive, is defined. There are two commonly used definitions, the Bliss independence and the Loewe additivity (Berenbaum 1989; Fitzgerald et al. 2006), although validity of this model as a universal reference model has been questioned (Greco et al. 1995). The Loewe additivity assumes that two inhibitors exert their effect through a similar mechanism (e.g., pathway), where the effects of each inhibitor and the combination are related through equipotent dose ratios. Bliss independence, however, assumes that the two inhibitors act through independent mechanisms (e.g., multiple pathways), in which combination therapy is represented as the union of two probabilistically independent events. The Loewe additivity correctly predicts the trivial case in which the two drugs are actually the same compound, i.e., drug A and a dilution of it are additive (Berenbaum 1989).

The Loewe additivity is embodied in the isobologram method for characterizing departures from additivity. To describe the joint action of two drugs A and B at a specific dose level, the additivity of Loewe (1955) is based on single drug dose-effect and is defined by the following isobole equation

$$ \frac{x_A}{X_A}+\frac{x_B}{X_B}=\tau $$

(1)

where x _A and x _B are doses of the constituent drugs A and B of the combination needed to yield a given level of effect, e.g., 50 % inhibition (ED ₅₀), or 50 % death in experiment animals (LD ₅₀), where X _A and X _B are the doses needed for each drug alone to yield the level of effect. The τ is called the interaction index of the drugs A and B at the combination (x _A , x _B ).When τ = 1, the drugs A and B is additive (zero-interaction) at the combination (x _A , x _B ); when τ < 1, they are synergistic, namely, the combination (x _A , x _B ) is more effective than expected from their single drug dose–response curves, otherwise (τ > 1), they are antagonistic.

Let the dose–response relationships for individual drugs A and B be $y={f}_A\left({X}_A\right)$ and $y={f}_B\left({X}_B\right)$ respectively. Denote the combination dose-effect (response) by f _com (x _A , x _B ), and with (1) we have

$$ \begin{array}{c}{f}_{com}\left({x}_A,{x}_B\right)={f}_A\left({X}_A\right)={f}_A\left(\tau {X}_A\right)+\left[{f}_A\left({X}_A\right)-{f}_A\left(\tau {X}_A\right)\right]\\ \hspace*{3.4pc}={f}_A\left({x}_A+\frac{X_A}{X_B}{x}_B\right)+\left[{f}_A\left({X}_A\right)-{f}_A\left(\tau {X}_A\right)\right].\end{array} $$

(2)

The term $\left[{f}_A\left({X}_A\right)-{f}_A\left(\tau {X}_A\right)\right]=0$ if the drugs are additive (τ = 1). Then, the regression line for the combination with additive action of two drugs is $y={f}_A\left({x}_A+{\rho}_B\left({X}_B\right){x}_B\right)$ where the relative potency ρ(X _B ) is a function of X _A  and X _B , $\rho \left({X}_B\right)={f}_A^{-1}{f}_B\left({X}_B\right)/{X}_B$. As show in Fang et al. (2008), the potency ρ(X _B ) is generally not a constant, the additive model (2) has no closed form.

Since we typically do not know much about the joint effect of the combinations before experiments, we have proposed a general semiparametric model for the joint effect of the constituent drugs (Tan et al. 2003),

$$ y={f}_A\left({x}_A+\rho \left({X}_B\right){x}_B\right)+f\left({x}_A,{x}_B\right)+\varepsilon $$

(3)

where f(x _A , x _B ) is an unspecified function since the term $\left[{f}_A\left({X}_A\right)-{f}_A\left(\tau {X}_A\right)\right]$ in (2) is a function of (x _A , x _B ), $\varepsilon$ is the error term due to variation in experiments and is assumed to be normally distributed with mean 0 and variance σ ². Then, testing the additive action of the two drugs is equivalent to testing the null hypothesis ${H}_0:f=0$. Suppose that there is a one-to-one invertible transformation from $\left({x}_A,{x}_B\right)\to \left({z}_1,{z}_2\right)$ such that ${f}_A\left({x}_A+\rho \left({X}_B\right){x}_B\right)=\left(\mathrm{or}\approx \right){g}_1\left({z}_1\right)+{g}_2\left({z}_2\right)$, where the functions g ₁ and g ₂ are linearly independent, an F-test is derived using a lack of fit type of test with the sum of squares with and without the term f. Specifically, let the m mixtures z ⁽¹⁾, …, z ^(m) be in the experimental domain. Assume that there are n _i experiments at the dose-level ${\boldsymbol{z}}^{(i)}={\left({z}_1^{(i)},{z}_2^{(i)}\right)}^T$ with corresponding responses ${y}_{ij}\left(j=1,\dots, {n}_i;i=1,\dots, m\right).$. Denote $n={n}_1+\cdots +{n}_m$, y the $n\times 1$ vector with elements y _ij ordered lexicographically, Z the $m\times 2$ matrix with i-th row (g ₁(z ⁽ⁱ⁾₁ ), g ₂(z ⁽ⁱ⁾₂ )). Let $V=UZ{\left({Z}^T{U}^TUZ\right)}^{-1}{Z}^T{U}^T$, $J=U{\left({U}^TU\right)}^{-1}{U}^T$, and the $n\times m$ matrix $U= diag\left({\mathbf{1}}_{n_1},\cdots,\ {\mathbf{1}}_{n_m}\right)$. Then, if the hypothesis H ₀ is true, the statistic

$$ F=\frac{{\boldsymbol{y}}^T\left(J-V\right)\boldsymbol{y}/\left(m-2\right)}{{\boldsymbol{y}}^T\left(I-J\right)\boldsymbol{y}/\left(n-m\right)}, $$

(4)

has a central F-distribution with degrees of freedom m−2 and n−m (Tan et al. 2003; Fang et al. 2008).

The question is which combinations should be chosen for experiment to demonstrate synergy, antagonism or additivity efficiently and with adequate statistical power. The MPD then utilizes individual dose response data and uniform measures (Fang and Wang 1994) to select a moderate number of combinations with a preset number of replications for experimentation (Tan et al. 2003, 2009; Fang et al. 2008). This method maximizes the minimum (among potential forms of departures from additivity) power of the F-test in (4) to detect departures from the additive action of drugs.

Although there exists conceptual statistical work, e.g., the maximal power F-test, for finding doses and sample sizes needed to detect departures from additivity. However, the method depends on dose–response shapes of individual drugs, namely, different classes of drugs of different dose–response shapes require different derivations for sample size and dose finding.

Upon completion of the experiments, the F-statistic (4) is to test the hypothesis of the additive action of two drugs and calculate the p-value of the F-test. If the p-value is greater than 0.05, we can accept the hypothesis of the additive action of two drugs. Otherwise, we calculate the interaction index (τ) as follows. Let y _ij be the j-th response at (x ⁽ⁱ⁾ _A , x ⁽ⁱ⁾ _B ). With the single dose–response curves, the interaction indexes at (x ⁽ⁱ⁾ _A , x ⁽ⁱ⁾ _B ) are

$$ {\tau}_{ij}=\frac{x_A^{(i)}}{f_A^{-1}\left({y}_{ij}\right)} + \frac{x_B^{(i)}}{f_B^{-1}\left({y}_{ij}\right)},\kern1em j=1,\dots, k;\kern0.5em i=1,\dots, m. $$

(5)

The method of two-dimensional B-splines (thin plate splines) is employed to estimate the interaction index surface $\tau =h\left({x}_A,{x}_B\right)$ (Fang et al. 2008).

In the vorinostat plus ara-C combination study, we first considered the experimental design. Based on the single experiments of inhibiting HL-60 cell line, the estimated dose–response curves of vorinostat and ara-C are

$$ \begin{array}{l}y=51.04-20.88 \log \left({X}_A-0.05\right),\kern1em {X}_A\in \left[0.1\;\upmu \mathrm{M},10\;\upmu \mathrm{M}\right],\\ {}y=9.22-10.17 \log \left({X}_B\right),\kern4.3em {X}_B\in \left[0.003\;\upmu \mathrm{M},0.6\;\upmu \mathrm{M}\right],\end{array} $$

(6)

respectively, where y is the viability (% of control). The corresponding ED ₅₀ of vorinostat and ara-C are 1.101 μM and 0.021 μM, respectively. To investigate the synergy of vorinostat and ara-C against HL-60, we used the MPD for the mixture experiments. The variance is estimated to be 1006.416 based on the pooled observations from the single drug experiments. The MPD design yields 18 combinations with five replicates at each combination (Table 1). The design has 80 % statistical power to detect at least a 15 % difference in viability between the predicted additive values and the observed values at a significance level of 5 %. Then, cells are exposed to these select combinations and the cytotoxicity of this combination is determined.

Table 1 Mixtures of vorinostat and ara-C

In the sequentially combination experiments of vorinostat with ara-C against HL-60, the dose ranges are from 0.137 to 5.005 μM for vorinostat and from 0.003 to 0.586 μM for ara-C. Of total 108 observations, the maximum viability is 78.87 % and the minimum viability is 0.027 %. The mean is 15.00 % and the standard error is 17.373. Figure 1 shows the response surface of the combination of vorinostat with ara-C against HL-60. The F-test (4) shows that we reject the null that vorinostat with ara-C against HL-60 has additive action (F _{16, 90} = 16.85, p-value < 0.0001). To explore the interaction of vorinostat with ara-C, we estimated the interaction index surface $\tau =h\left({x}_A,{x}_B\right)$ using thin plate splines (Fang et al. 2008). Figure 2 shows the contour plot of combination index surface such that when the dose of vorinostat is less than 0.4 μM or both the doses of vorinostat and ara-C are higher, the joint action is additive. The maximum synergy actions occur at the dose of vorinostat between 1.2 and 2.5 μM and the dose of ara-C between 0.003 and 0.3 μM.

Fig. 1

Response surface of the sequential combination of vorinostat (SAHA) with ara-C

Fig. 2

Contour plot of combination index surface of vorinostat (SAHA) and ara-C sequential combination. The dotted lines indicate the 95 % confidence surfaces for additive action (the combination index = 1)

Based on the preclinical results, a phase I trial was planned. In principle, modeling the toxicity interaction appropriately would add to the efficiency and result in a better trial design. However, at the time of designing the phase I protocol, none of the methods were ready for clinical trial protocol developments. We designed the phase I trial escalating the dose of vorinostat while having the fixed doses of ara-C (1–2 g/m² due to patient age) and etoposide (100 mg/m²) on days 11–14. Twenty-one patients with acute myelogenous leukemia (AML) were enrolled in the trial, and the maximum-tolerated dose (MTD) was established to be vorinostat 200 mg twice a day orally. Of 13 patients with high-risk leukemia treated at the maximum-tolerated dose of vorinostat (200 mg, orally, twice a day), six obtained a complete remission (CR) with median duration of 7 months. The relatively high CR rate in this poor-risk acute myelogenous leukemia patient group warrants further study (Gojo et al. 2013).

However, there are two missed opportunities for this study: a suboptimal clinical trial design had been used where only the dose of vorinostat was escalated, and a suboptimal study design with fixed dose of one drug for the three drug combination had been used. In the following two sections, we present both a Bayesian adaptive phase I trial design that would have been useful in identifying the maximum tolerated doses in three dimensions; and the maximal power design for combinations of three drugs that would have been utilized in the Vorinostat study had these methods been available then.

3 Bayesian Adaptive Phase I Trial Design for Drug Combinations

Different from single agent trials, the interaction effect between two drugs may have a significant impact on the joint toxicity probability of the dose combination. Independent, synergistic or antagonistic effects are the different states of interactions. The independence model implies that the two drugs have no apparent interaction with the respect to the toxicity. The Bliss independence criterion has been used in describing the joint action in toxicity for two agents (Goldoni and Johansson 2007). Its main assumption is that two or more drugs act independently from one another (Greco et al. 1995; Bliss 1939; Berenbaum 1989). Let P(A) and P(B) be the marginal toxicity probabilities of drugs A and B, respectively. If A and B are independent, the probability of no toxicity in the combination of A and B is

$$ 1-P\left(A\cup B\right)=\left\{1-P(A)\right\}\left\{1-P(B)\right\} $$

(7)

Thus, the joint probability of toxicity g(x _A , x _B ) at combination (x _A , x _B ) has the form

$$ g\left({x}_A,{x}_B\right)=1-\left\{1-g\left({x}_A,0\right)\right\}\left\{1-g\left(0,{x}_B\right)\right\}, $$

(8)

where g(x _A , 0) and g(0, x _B ) are the marginal toxicity probabilities of drug A and drug B, respectively. When $g\left({x}_A,{x}_B\right)>1-\left\{1-g\left({x}_A,0\right)\right\}\left\{1-g\left(0,{x}_B\right)\right\}$, drug A and drug B at combination (x _A , x _B ) have Bliss synergy of toxicity. For Bliss antagonism, the inequality is reversed. The Bliss antagonism results in lowering toxicity at a given drug combination. To specify the toxicity response g(x _A , x _B ), we proposed a factorial type Bliss model that allows mixed interaction profile for the combination therapy using the drugs A and B on the binary toxicity outcome. The probability of toxicity is modeled as follows:

$$ g\left({x}_A,{x}_B,\theta \right)=1- \exp {\left(-\alpha {x}_A-\beta {x}_B\right)}^{f\left({\gamma}_1,{\gamma}_2,{x}_A,{x}_B\right)}, $$

(9)

where α > 0, β > 0 and γ ₁, γ ₂ are parameters to be estimated. The function f(γ ₁, γ ₂, x _A , x _B ) is used to measure the degree of synergy or antagonism of the different dose combinations. We proposed the following form

$$ f\left({\gamma}_1,{\gamma}_2,{x}_A,{x}_B\right)= \exp \left({x}_A{x}_B\left({\gamma}_1{x}_A+{\gamma}_2{x}_B\right)\right). $$

(10)

The model satisfies the conditions that if ${x}_B=0$, then $g\left({x}_A,0\right)=1- \exp \left(-\alpha {x}_A\right)$, which is the toxicity model of single drug A. Similarly, when ${x}_A=0$, then $g\left(0,{x}_B\right)=1- \exp \left(-\beta {x}_B\right)$, the toxicity model reduces to that of the single drug B. Then, the single drug case becomes the convention exponential toxicity model. It captures antagonism when $f\left({\gamma}_1,{\gamma}_2,{x}_A,{x}_B\right)<1$, independence when $f\left({\gamma}_1,{\gamma}_2,{x}_A,{x}_B\right)=1$ and synergy when $f\left({\gamma}_1,{\gamma}_2,{x}_A,{x}_B\right)>1$. Thus, we call f(γ ₁, γ ₂, x _A , x _B ) the interaction function.

Based on the toxicity model (9), we proposed a novel method to find the maximum tolerated region (MTR) consisting of the doses that have the posterior mean toxicity probabilities below the target toxicity probability φ,

$$ \mathrm{M}\mathrm{T}\mathrm{R} = \left\{\left({x}_A,{x}_B\right):\pi \left(\left({x}_A,{x}_B\right);\theta \right)\le \varphi \right\} $$

(11)

with minimum patient number for a given target probability. The method recognizes that there may exist multiple MTDs with drug combinations and addresses the issue directly. It integrates the toxicity interaction of two drugs and Bayesian adaptive dose-finding algorithm (Yang et al. 2016). The goal is to bring the trial to dose combinations where there may be antagonistic behavior among the drugs that the patients can be safely assigned to relatively high dose of individual drugs which otherwise would not be possible in single drug scenario. Thus, patients can be exposed to doses with high efficacy without experiencing significant toxicity. Let ${x}_A=\left\{{a}_1,\dots, {a}_I\right\}$ and ${x}_B=\left\{{b}_1,\dots, {b}_J\right\}$ be the specific dose levels of drugs A and B, respectively. For given dose levels of drug A at x _A and drug B at x _B , the rescaled interaction function can be defined as:

$$ v\left({x}_A,{x}_B\right)=\frac{f\left({\gamma}_1,{\gamma}_2,{x}_A,{x}_B\right)}{f\left({\gamma}_1,{\gamma}_2,{x}_A,{x}_B\right)+1}. $$

(12)

The goal will be to locate a dose combination by minimizing a combination of the interaction function v(x _A , x _B ) and the toxicity probability g(x _A , x _B ) subject to the constraint that the toxicity probability is no more than a pre-specified value. We define the objective as a convex combination of the probability of toxicity at dose (x _A , x _B ) and the interaction v(x _A , x _B ) at that dose,

$$ {U}_{\lambda}\left({x}_A,{x}_B\right)=\lambda g\left({x}_A,{x}_B\right)+\left(1-\lambda \right)v\left({x}_A,{x}_B\right), $$

(13)

where $0<\lambda <1$. The choice of λ reflects how much emphasis one would like to put on having more allocation at antagonistic combinations. Toxicity probability and interaction function are considered jointly through the objective function with the relative contribution of each component controlled by the weight λ. Smaller values of U would indicate smaller values of the standardized interaction leading to more antagonism and smaller toxicity probability. Therefore, our Bayesian adaptive dose-finding design is developed with the goal of minimizing the objective function. The objective function is evaluated based on the measurement of the mean squared error (MSE) of the toxicity probability estimate and the amount of interaction that patients really experienced. We conducted extensive empirical studies to evaluate possible λ values over several plausible scenarios. We recommend the choice of λ = 0.5 which suggests equal contribution of toxicity probability and the allowance of interaction. The next dose combination may be chosen to minimize the posterior expectation of the objective function given the current data Z _n ,

$$ {\boldsymbol{x}}_{n+1}=\left({a}_i,{b}_j\right)= \arg \min E\left\{{U}_{\lambda}\left(\boldsymbol{x},\theta \right)\Big|{Z}_n\right\}. $$

(14)

The dose finding algorithm and simulation studies of the design properties are given in Yang et al. (2016). The simulation studies under various scenarios demonstrate that the proposed design performs satisfactorily with expected operating characteristics. In particularly, the sample size in this proposed method is more favorable than that in existing methods based on the simulation results.

4 Maximal Power Design for Three-Drug Combinations

As we have shown in Sects. 2 and 3, the preclinical experiments of the vorinostat combinations have been done pairwise, vorinostat + etoposide at a fixed ara-C dose, and vorinostat + ara-C at a fixed etoposide dose. However, the optimal design was not developed early enough to be utilized in the development of vorinostat combinations. Indeed, to our knowledge, the literature in experimental design method for three drug combinations is sparse. Tan et al. (2009) derived the potential experimental design of combinations with varying doses in all three constituent drugs determined based on the MPD by extending the method summarized in Sect. 2, and proposed a sample size formula and a MPD to detect synergy in combination studies of three drugs when each of three drugs has a log-linear dose–response. Since the design depends on the shapes of the dose–response curves of the constituent agents, combination studies based on the linear and log-linear individual dose–response curves necessitate different mathematical joint effect model and generating uniformly scattered points in a tetragon area (Tian et al. 2009). That was the first time to our knowledge that a three drug combination experiment was designed through a search of the three drug dose region.

Note that critical to the uniform design method is to be able to derive the approximate decomposition of the additive model, and this becomes more difficult with three drugs. Tan et al. (2009) has derived the MPD for combinations of common cytotoxic agents whose individual dose response is log-linear or observe the Hill model. The log-linear dose–response curve represents a wide class of drugs including antimetabolites, antibiotics, interferons, growth factors, neuropeptide Y, phorbol esters, narcotics and neuronal agonists, hepatotoxins, and cromoglycate. The combination of vorinostat combined with ara-C and etoposide against HL-60 is also considered there.

To illustrate the methods of experimental design for three-drug combination studies, we consider the experiment to determine the effects of pre-administration of vorinostat on the pharmacokinetics of ara-C and etoposide against the leukemia cell line HL-60 (Shiozawa et al. 2009). In the experiments for single agents, we have 56 observations with doses ranging from 0.1 to 10 μM for vorinostat, 56 observations with doses ranging from 0.003 to 0.6 μM of ara-C, and 64 observations with doses ranging from 0.01 to 10 μM of etoposide. Then, the single dose–response curves for ara-C, etoposide and vorinostat are estimated to be

$$ \begin{array}{rcl}y\left({X}_A\right)& =& 4.80-12.76 \log \left({X}_A\right),\\ {}y\left({X}_B\right)& =& 41.52-13.02 \log \left({X}_B\right),\\ {}y\left({X}_C\right)& =& 54.55-23.98 \log \left({X}_C\right),\end{array} $$

(15)

respectively, where y is the $100\times$ viability, and X _A , X _B  and X _C are the doses of ara-C, etoposide and vorinostat respectively. The potency of etoposide relative to ara-C is ${\rho}_0\left({X}_B\right)=0.0563{X}_B^{0.0204}$ and the potency of vorinostat relative to ara-C is ${\rho}_1\left({X}_C\right)=0.0203{X}_C^{0.8793}$, which show that these relative potencies are non-constant and depend on dose. The predicted additive model at (x _A , x _B , x _C ) is

$$ y\left({x}_A,{x}_B,{x}_C\right)=4.80-12.76 \log \left({x}_A+0.0551{\psi}^{0.0427}{x}_B+0.0203\psi {x}_C\right), $$

(16)

where ψ is determined by

$$ \psi ={\left(49.3483{x}_A+2.7204{\psi}^{0.0427}{x}_B+\psi {x}_C\right)}^{0.4679}. $$

An approximate additive model (16) is given by

$$ \begin{array}{lll}{y}_{\mathrm{appr}}\left({x}_A,{x}_B,{x}_C\right)\hfill & =\hfill & 4.80-12.76 \log \left({z}_1\right)-12.76 \log \left[\left(1-0.0563\right){z}_2+0.0563\right]\hfill \\ {}\hfill & \hfill & -12.76 \log \left[\left(1-0.3601\right){z}_3+0.3601\right],\hfill \end{array} $$

(17)

where

$$ \left\{\begin{array}{rcl}{z}_1& =& {x}_A+0.9798{\psi}^{0.0427}\left({x}_A,{x}_B,{x}_C\right){x}_B+\psi \left({x}_A,{x}_B,{x}_C\right){x}_C\\ {}{z}_2& =& {x}_A{\left[{z}_1-0.6399\psi \left({x}_A,{x}_B,{x}_C\right){x}_C\right]}^{-1}\\ {}{z}_3& =& \psi \left({x}_A,{x}_B,{x}_C\right){x}_C/{z}_1\end{array}\right. $$

(18)

To obtain MPD for testing the joint action of ara-C, etoposide and vorinostat, the dose range is chosen such that the endpoint, 100 × viability, is from 20 to 80 for ara-C. Then, the total dose ranges from 0.0028 to 0.3038 μM in ara-C. The pooled variance from the three single drug experiments is 988.422. For a meaningful difference η of 15 (100 × viability) and five replications for each mixture, with type I error rate 0.05 and power 0.80, we need study 21 mixtures in the experiment in order to detect synergy/antagonism in the combination of ara-C, etoposide and vorinostat (total 105 experiments). With the algorithm given in Tan et al. (2009), we get 21 points in domain $\left\{{\left({z}_1,\ {z}_2,\ {z}_3\right)}^T:\right.$ $\left.0.0028<{z}_1<0.3038,\ {z}_2>0,\ {z}_3>0,\ {z}_2+{z}_3<1\right\}$. According to the inverse transformation of (18), 21 mixtures of these three drugs for experiments are given in Table 2, of which the doses of etoposide and vorinostat are 16.78149(x ⁽ⁱ⁾ _B )^0.98 and 7.961724(x ⁽ⁱ⁾ _C )^0.5321 respectively, because of the total dose range according to ara-C.

Table 2 Twenty-one mixtures of ara-C, etoposide and vorinostat for combination experiment

Furthermore, the method needs to be modified to allow one or more of the individual dose response curve being not log-linear. For example, in the combination study of three anti-cancer agents PD184, HA14-1 and CEP3891 in myeloma H929 cell line. PD184 is a highly potent and selective noncompetitive MEK inhibitor. HA14-1 is a small, cell-permeable nonpeptidic ligand that binds to the Bcl-2 surface pocket and blocks its biological action. Similar studies involving three agents have been designed and analyzed in a pairwise fashion, namely, studying the combinations of any two of the three separately. This is clearly suboptimal, not only it is potentially more costly but also the analysis results are hard to interpret and may not reflect the real optimal dose of the three agents in combination (Pei et al. 2003, 2004). Fang et al. (2015) extended the MPD and derived the design and analysis in this case with mixed linear and log-linear dose response curves. The experiment of varying doses of all three drugs based on the MPD was implemented for the first time to our knowledge.

5 Multidrug Joint Response Modeling with Systems Biology

Increasing the number of agents in a combination may provide better outcomes. However, even with six drugs, each with only six doses, the number of potential combinations reaches 46,656. The exponential increase in number of combinations with the number of drugs makes laboratory testing difficult. Consequently, most work in multidrug combinations is conceptual. Calzolari et al. (2008) utilize a deterministic model and network information to develop a search algorithm. Furthermore, despite the biological advances mentioned above and the importance of multi-agent combinations, current methods are mostly topological as opposed to quantitative, and do not account for high dimensionality and proper model assumptions (Krzywinski and Altman 2014; Ashton 2015). Recently we proposed a novel two-stage procedure utilizing an initial selection by utilizing an in silico model built upon experimental data of single drugs and current systems biology information to obtain maximum likelihood estimate (Fang et al. 2016). We briefly summarize the method below and discuss its potential applications in drug development process.

Biological networks are controlled/regulated by receptors. They are comprised of connecting genes and output nodes that are implicated in determining activation of the cell death machinery. Figure 3 presents a typical example network—apoptosis related signaling from the KEGG database (hsa04210). Different nodes have various signal propagation rules. Consider a combination study of s drugs A ₁, A ₂, …, A _s , we first develop a statistical rescaling model to describe the effects of drugs on network topology. The model comprises a Hill equation for signals arriving at each receptor, a generic enzymatic rate equation to transmit signals among connecting genes, and a regression model to represent the cumulative effect of genes implicated in activation of the cell death machinery. Specifically, for a given dose-level $\boldsymbol{x}={\left({x}_1,{x}_2,\dots, {x}_s\right)}^T$ of drugs A ₁, A ₂, …, A _s , denote a _0i (x) as the signal of receptor i obtained (i = 1, 2, …, r) and a _i (x) as the signal connecting gene i obtained (i = 1, 2, …, r). Gene activity levels often exhibit a non-linear relationship to their upstream regulatory signals. Typically, a Hill equation (Weiss 1997) can be used to model the activity a _0i (x) at receptor i,

$$ {a}_{0i}\left(\boldsymbol{x}\right)=\frac{{\left({\boldsymbol{\beta}}_i^T\boldsymbol{x}\right)}^{\alpha_i}}{1+{\left({\boldsymbol{\beta}}_i^T\boldsymbol{x}\right)}^{\alpha_i}},\kern1em for\kern0.5em i=1,2,\dots, r, $$

(19)

where α _i and ${\boldsymbol{\beta}}_i={\left({\beta}_{i1},{\beta}_{i2},\cdots, {\beta}_{is}\right)}^T$ are the parameters to be estimated. To characterize the transmission of signals among connecting genes, the generic enzymatic rate equations can be used to adjust for possible feedback loops. Such equations have been motivated by various computational and biological considerations, a result of the close interaction between experimental and computational efforts (Lee et al. 2007; Ao et al. 2008). Let a _j (x) be the activity at gene j and a _(i,j)(x) the signal sending from gene j to gene i. The activity a _i (x) at gene i is defined to be the summation of all signals a _(i,j)(x) for gene j linked up gene i, and the generic enzymatic rate equation then suggests that

$$ {a}_i\left(\boldsymbol{x}\right)={\varSigma}_{j\in n(i)}{a}_{\left(i,j\right)}\left(\boldsymbol{x}\right),\ \mathrm{and}\ {a}_{\left(i,j\right)}\left(\boldsymbol{x}\right)=\frac{V_{F_j}\frac{a_j\left(\boldsymbol{x}\right)}{\omega }-{V}_{B_i}\frac{a_i\left(\boldsymbol{x}\right)}{\omega }}{\frac{V_{F_j}^2}{V_{F_j}^2+{V}_{B_i}^2}\left(1+\frac{a_j\left(\boldsymbol{x}\right)}{\omega}\right)+\frac{V_{B_i}^2}{V_{F_j}^2+{V}_{B_i}^2}\left(1+\frac{a_i\left(\boldsymbol{x}\right)}{\omega}\right)}, $$

(20)

where n(i) is the set of genes that signal to gene i, and ω is the expected steady state parameter. ${V}_{F_i}$ and ${V}_{B_i}$ are the forward and backward parameters, respectively. When the action between genes i and j is irreversible in the backward direction, ${V}_{B_i}=0$. The number of parameters ${V}_{F_i}$ and ${V}_{B_i}$ may become large if many connecting genes exist in the network. The forward and backward parameters ${V}_{F_i}$ and ${V}_{B_i}$ of connecting gene i may differ with those of connecting gene j (i ≠ j). Statistical variations typically occur when signals pass though the network because of link instability, stochastic noise inherent in the signal propagation rules, and/or chaos phenomena from the presence of loops. To model the network efficiently, it is reasonable to assume that ${V}_{F_i}$ and ${V}_{B_i}$ (i =1, 2, …) are random effects that are independently and identically distributed (i.i.d.) normal random variables with mean μ ₁ and variance σ ²₁ .

Fig. 3

The human apoptosis network extracted from the KEGG database (hsa04210). Genes are categorized as receptors (yellow circles), connecting genes (green rectangles), and the output nodes (red diamonds) that are implicated at the onset of the cell death machinery. A solid line with an arrow at the end indicates direct promotion; a dashed line with an arrow at the end indicates indirect promotion; a dashed line with a bar at the end indicates inhibition. A cross symbol between two genes indicates dissociation, in which case the two genes may be viewed as a single node (e.g., DFF45 and DFF40)

A linear model is used to represent the cumulative effect of genes implicated at activation of the cell death machinery. For a given dose-level $\boldsymbol{x}={\left({x}_1,{x}_2,\dots, {x}_s\right)}^T$ of drugs A ₁, A ₂, …, A _s , let Y(x) be the observed viability and $\mathbf{a}\left(\boldsymbol{x}\right)={\left({a}_{i_1}\left(\boldsymbol{x}\right),\dots, {a}_{i_h}\left(\boldsymbol{x}\right)\right)}^T$ be the vector of the activities at genes i ₁, …, i _h which activate the output, then we have

$$ {Y}_k\left(\boldsymbol{x}\right)={u}_0+\mathbf{a}{\left(\boldsymbol{x}\right)}^T\boldsymbol{u}+{\upepsilon}_k\left(\boldsymbol{x}\right), $$

(21)

where the subscript k is the k-th replication at dose-level $\boldsymbol{x}={\left({x}_1,{x}_2,\dots, {x}_s\right)}^T$, the measurement error is assumed to be ${\upepsilon}_k\left(\boldsymbol{x}\right)\sim N\left(0,{\left(\sigma \left(\boldsymbol{x}\right)\right)}^2\right)$, and the standard deviation σ(x) of the measurement error may depend on the dose-level $\boldsymbol{x}={\left({x}_1,{x}_2,\dots, {x}_s\right)}^T$. u ₀ is the intercept parameter, and $\boldsymbol{u}={\left({u}_1,\dots, {u}_h\right)}^T$ is the vector of regression parameters to be estimated. The positive parameter u _i indicates promotion by gene i; the negative parameter u _j indicates inhibition by gene j. Since a(x) in model (21) equals to zero when $\boldsymbol{x}=\mathbf{0}$, the intercept μ ₀ should be 100 % cell viability if there is no drug intervention on the network.

In combination studies, the data from single drug experiments are available a priori. To make models (19), (20), and (21) identifiable and estimate the multivariable dose–response, one will need some additional data on the drug combinations. Fang et al. (2016) have shown that this can be achieved with several combinations with each drug at its individual IC ₅₀. Based on the training data, the parameters in Eqs. (19), (20), and (21) can be estimated with the maximum likelihood approach. Let $\boldsymbol{\beta} ={\left({\boldsymbol{\beta}}_1^T,\dots, {\boldsymbol{\beta}}_r^T\right)}^T$, $\boldsymbol{\alpha} ={\left({\alpha}_1,\dots, {\alpha}_r\right)}^T$ and $\boldsymbol{\theta} ={\left({\boldsymbol{\beta}}^T,{\boldsymbol{\alpha}}^T,\omega, {\boldsymbol{u}}^T,{\sigma}_0^2,{\mu}_1,{\sigma}_1^2\right)}^T$ be the vector of all parameters to be estimated. Suppose that there are n distinct inputs x ₁, …, x _n , and k _i replications at each input x _i , the corresponding output is Y _ij for $j=1,2,\dots, {k}_i;$ and $i=1,2,\dots, n$. For given μ ₁, σ ²₁ and a sample ${V}_{F_i}$ and ${V}_{B_i}$ (i =1, 2, …) from the normal distribution N(μ ₁, σ ²₁ ), the ECM algorithm (Meng and Rubin 1993) can be applied to obtain the maximum likelihood estimation of β, α, ω, u, σ ²₀ . Furthermore, for given β, α, ω, u, σ ²₀ , we can obtain n samples of ${V}_{F_i}$ and ${V}_{B_i}$ using Eq. (21) with n distinct inputs x ₁, …, x _n . The estimation of μ ₁ and σ ²₁ can then be obtained.

The dose–response surface of multidrug combinations is complex and difficult to estimate adequately. To get sufficient information of drug interactions, the functional ANOVA (Sobol 1993, 2001, 2003) is employed, which similar to functional principal component analysis. Without loss of generality, assume the dose-level of s drugs A ₁, A ₂, …, A _s , $\boldsymbol{x}={\left({x}_1,{x}_2,\cdots, {x}_s\right)}^T\in {\left[0,1\right]}^s,$ and $y=g\left(\boldsymbol{x}\right)$ is the corresponding dose response. Let ${g}_0={\displaystyle {\int}_{{\left[0,1\right]}^s}g}\left(\boldsymbol{x}\right)d\boldsymbol{x}$ be the overall mean of g(x). Then, there is a unique decomposition

$$ g\left(\boldsymbol{x}\right)={g}_0+{\displaystyle \sum_{i=1}^s{g}_i}\left({x}_i\right)+{\displaystyle \sum_{i<j}{g}_{ij}}\left({x}_i,{x}_j\right)+\cdots +{g}_{1,2,\dots, s}\left({x}_1,{x}_2,\cdots, {x}_s\right), $$

(22)

which satisfies ${\displaystyle {\int}_0^1{g}_{i_1,\dots, {i}_u}}\left({x}_{i_1},\dots, {x}_{i_u}\right)d{x}_{i_k}=0,$ for any $1\le u\le s$ and $1\le k\le u;$ and the orthogonality ${\displaystyle {\int}_{{\left[0,1\right]}^s}{g}_{i_1,\dots, {i}_u}}\left({x}_{i_1},\dots, {x}_{i_u}\right){g}_{j_1,\dots, {j}_v}\left({x}_{j_1},\dots, {x}_{j_v}\right)d{x}_1\cdots d{x}_s=0$ if $\left({i}_1,\dots, {i}_u\right)\ne \left({j}_1,\dots, {j}_v\right)$. The total and partial variances can be defined by

$$ D={\displaystyle {\int}_{{\left[0,1\right]}^s}{g}^2}\left(\boldsymbol{x}\right)d\boldsymbol{x}-{g}_0^2\ \mathrm{and}\ {D}_{i_1,\dots, {i}_k}={\displaystyle {\int}_{{\left[0,1\right]}^k}{g}_{i_1,\dots, {i}_k}^2}\left({x}_{i_1},\cdots, {x}_{i_k}\right)d{x}_{i_1}\cdots d{x}_{i_k}, $$

(23)

respectively. Denote $D={\displaystyle \sum_{k=1}^s{\displaystyle \sum_{i_1<\cdots <{i}_k}{D}_{i_1,\dots, {i}_k}}}$, the ratios

$$ {R}_{i_1,\dots, {i}_k}={D}_{i_1,\dots, {i}_k}/D,{\displaystyle }1\le {i}_1<\cdots <{i}_k\le s, $$

(24)

are called global sensitivity indices (Sobol 1993, 2001, 2003). The integer k is called the order of the index. All ${R}_{i_1,\dots, {i}_k}$’s are non-negative and their sum ${\displaystyle \sum_{k=1}^s{\displaystyle \sum_{i_1<\cdots, {i}_k}{R}_{i_1,\dots, {i}_k}}}=1$. The equality ${R}_{i_1,\dots, {i}_k}=0$ implies that ${g}_{i_1,\dots, {i}_k}=0$ and so the interaction of drugs ${A}_{i_1}\cdots {A}_{i_k}$ is not significant. Significance of the interaction of drugs ${A}_{i_1}\cdots {A}_{i_k}$ decreases with decreasing ${R}_{i_1,\dots, {i}_k}$ values. Hence, the dose–response model can be reduced if we only retain the principal terms with the largest global sensitivity indices, an approach similar to principal component analysis. It is also expected that the number of terms in the dose–response functional ANOVA representation will be reduced significantly because the cumulative global sensitivity indices of the first few terms usually contribute a dominant portion (say, 80 %) of the total variation (Fang et al. 2006). To obtain the numerical values of the global sensitivity indices, the Quasi-Monte Carlo methods for approximating the integrals can be adopted. For more details, please refer to Fang et al. (2006).

We have performed two simulation experiments to investigate the effectiveness of the optimal network simulator for the discovery of multidrug interactions using the apoptosis signaling network. The first example involves a combination study of five drugs; the second example considers as many as ten drugs. The simulation with five drugs identified three terms (drugs and their interactions) making most significant contributions yielding a global sensitivity indices of 90.45 %, which is consistent with the global sensitivity indices from the true dose–response. For the simulation with 10 drugs, it has been shown that the method identified four most significant terms with a total of global sensitivity indices of 92.19 %, which is consistent with the global sensitivity indices from the true dose–response. We summarize the process of the development of drug combinations in Fig. 4.

Fig. 4

The framework for multiple drug combination study

6 Discussions and Conclusions

Cancer cells carry out their functions following appropriate responses to the extracellular and intracellular inputs to their complex network of multiple signaling pathways. Many genes that code for proteins in these pathways are controlled by regulatory proteins that up-regulate or downregulate these genes depending on the inputs to the signaling network. It is conceivable that multidrug approach can play an even greater role in cancer developmental therapeutics with the development of the systems biology. For combinations of two and three drugs, we have reviewed the MTDs and statistical modeling of joint effects for experiments performed following the MPD. For phase I clinical trial, we outlined an approach that models the drug interactions and the dose escalation algorithm based on a Bayesian model with computation performed using Markov chain Monte Carlo. However, combination studies are extremely difficult to implement in clinical trials suggesting the greater importance of preclinical testing of the combination based on cell lines and animal models that are well established.

For combinations of more than three drugs, we proposed a two-stage approach with the first stage utilizing data of single drugs (and some drug combinations) and the current network information to develop a statistical model to describe the drug effects on the network. Through these statistical models, we conducted computer experiments (in silico) to derive a global sensitivity index of each term in the functional ANOVA of dose response model by generating doses of the drugs with the Quasi Monte-Carlo method. Then, we can predict the main effects that occur with combinations of multiple drugs. Two simulation studies illustrate the superior performance of our methods. The principal global sensitivity indices generally select 3–4 terms of multidrug combinations in the functional ANOVA model if the true dose response function is smooth. Then we can develop experimental designs and statistical procedures on the few selected terms. Given the scope of this article, we will report the details of experimental design and data analysis based on these selected interaction and main effect terms in a future report.