Interactions of Bioactive Plant Metabolites: Synergism, Antagonism, and Additivity

Abstract

Drugs are commonly used in mixtures, also called cocktails, to treat disease, particularly cancer and viral infections. Any two or more drugs, or for that matter, two or more bioactive plant compounds, will either interact in some way or fail to interact. If an interaction produces an effect greater than that expected for each individual drug, the interaction is termed synergistic. If the effect is less than expected, it is termed antagonistic. If the effect is equal to the expected effect (i.e., there is no interaction), the interaction is termed additive (see Greco et al., 1995; Spelman, 2007, in Cseke et al., 2006). In most therapeutic situations, the hope is that mixtures will produce a synergistic effect, but additivity can also be useful and should not be neglected.

Our focus in this chapter is on interactions between bioactive plant compounds used in food and medicine. In particular, we are interested in plant compounds that have potential therapeutic effects, but also exhibit low systemic toxicity, and thus do not pose a high risk of producing adverse effects. Thousands of such compounds are known to exist, and more are being discovered each year. Even a single plant can contain dozens of bioactive compounds. With such a large pool to draw from, there is nearly an unlimited number of ways in which compounds can be combined, either with each other or with market-approved drugs. Clearly many opportunities exist to find mixtures that exhibit synergism or additivity.

In the following sections we explore physical models of drug interaction, discuss a mathematical model that can be used to assess interactions, and provide a number of examples of plant compounds that have been shown to interact in a synergistic fashion. In particular, we look at ways by which mixtures of plant compounds may bind to proteins and affect signaling pathways, as well as ways by which plant compounds could alter receptors indirectly by affecting the plasma membrane. The mathematical model discussed provides an accurate method to estimate interaction indices as well as to construct confidence intervals of the indices. An interaction index is of little use if it is not accompanied by confidence intervals. Technical aspects of the model are presented in order to provide a full description, but publically available software for the model can be used without a complete understanding of the mathematics involved.

These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

1 Introduction

We start this chapter by noting that most drugs approved for the market were not developed with synergism in mind. The conventional regulatory process requires that the safety and efficacy of a drug be based on the merits of the drug used alone. It is only after market approval that assessment of synergistic interactions with other approved drugs begins in earnest. The current guiding philosophy in drug development is one of targeted therapy, whereby a single drug is designed to affect a single protein target. This target could be a cell surface receptor or an intracellular protein. The goal is to develop drugs that bind with high affinity to a target, but have little affinity for off-target proteins. In this way, some adverse effects can be avoided.

When developing bioactive plant compounds as drug mixtures, nutraceuticals, or medicinal foods, an alternative philosophy is needed. The development process for a mixture of plant bioactive compounds must necessarily be different from that for a single-target drug. At least four primary differences between the two types of products stand out. First, rather than a single active constituent, there could be several or even many dozens of active constituents in a plant extract. Indeed, a given meal rich in plant compounds could contain hundreds of bioactive compounds, albeit in small doses. Second, within the class of (reasonably) nontoxic plant compounds that are the focus of this chapter, the binding affinity for known drug targets is often modest to low. Rather than using a low dose of a single high-affinity compound, the optimal clinical effect for many of these plant compounds might be seen when either a relatively high dose of a single compound is administered or, as preferred, modest doses of numerous compounds are given in a complex mixture that takes advantage of additive and synergistic effects. Third, many plant compounds are promiscuous, in that they bind with multiple targets, which may exist in different signaling pathways within the cell (Frantz, 2005; Aggarwal and Harikumar, 2009). Fourth, many bioactive plant compounds, such as flavonoids and curcuminoids, are not soluble in water, are metabolized to less-active conjugates or other products after oral administration, or in some other way exhibit nonideal absorption, distribution, metabolism, or excretion (ADME) characteristics. The fact that many bioactive plant compounds bind with relatively low or modest affinity to known targets, bind with multiple targets, and/or exhibit poor pharmacokinetics rules them out as useful drugs according to the conventional mode of thinking.

However, this does not mean that bioactive plant compounds do not and could not play a highly useful role in health and medicine. Indeed, in spite of the fact that most do not resemble a “silver bullet” drug, it is likely that such compounds are responsible for much of the disease-prevention effects seen in human populations that consume a diet rich in plant compounds (Liu, 2003). To understand how their beneficial properties can best be exploited, we must redefine, or at least expand, our definition of a model drug.

Within the class of nontoxic bioactive plant compounds, the characteristics that appear to limit their use in medicine also provide us with opportunities. The fact that a plant extract or mixture of extracts may have a large number of bioactive constituents means that there are abundant opportunities for additivity and synergism. It should be emphasized that although synergistic interactions tend to receive the most research attention, additive interactions are more common, and in large mixtures they may play an even more important role than synergism. For example, in a cytotoxicity study on doxorubicin and nine natural compounds against human lung cancer cells, Boik and Newman (2008) found that synergism in smaller mixtures could allow a tenfold reduction in the concentration of doxorubicin needed to produce a given effect level. Importantly, larger mixtures that exhibited less synergism (and more additivity) allowed a similar degree of reduction in doxorubicin concentrations. In the larger mixtures, each drug was used at a lower concentration.

Additivity and synergism arise in mixtures through their ability to affect multiple targets. The importance of affecting multiple targets cannot be overemphasized when dealing with complex diseases such as cancer, chronic inflammatory diseases, chronic viral infection, and many others. In these diseases, numerous macro homeostatic systems may be affected, as well as numerous intracellular and intercellular signaling pathways. A large number of proteins can be involved, and each may be considered as a potential target. For complex diseases, acceptance is growing in the pharmaceutical industry for the design of mixtures or drugs capable of affecting multiple targets (Tortora et al., 2004; Roth et al., 2004; Frantz, 2005; Zimmermann et al., 2007).

Taking common solid tumors as an example, multiple signaling pathways can be critically under- or overregulated (Chandran et al., 2007; Chan et al., 2008). As tumors progress, the genetic machinery of tumor cells tends to become increasingly unstable. When this occurs, more proteins can become involved and the tumor population becomes better able to adapt to drug therapy, immune attack, or other obstacles to growth. It would seem unlikely that any single drug, particularly a drug that affects a single target, would have a lasting effect on such a flexible cell population. If the drug does not kill 100% of cells, there is a reasonable chance that the surviving cells will reestablish a population resistant not only to the applied drug but also to other drugs. This characteristic is termed multidrug resistance. Cells may become resistant through gene amplification and overproduction of the target protein, production of proteins that pump the drug out of the cell, underproduction of proteins that allow the drug to enter the cell, improved repair of drug-induced DNA damage, production of proteins that inactivate the drug, and/or production of proteins that serve the same function as the target protein but that are not affected by the drug. Literally, hundreds of proteins could be involved in the progression of cancer and developed resistance to drugs.

While plants may offer a rich source of bioactive compounds for use in mixtures, do these compounds need to bind to their targets with high affinity in order to be effective? Csermely et al. (2005) have proposed that in some cases, partial inhibition of multiple signaling pathways may be more efficient than complete inhibition of a single pathway (see also Ágoston et al., 2005). This suggests that compounds that exhibit modest binding affinity could still be useful.

Could administration of multiple compounds lead to adverse effects? Of course, adverse effects are possible with any pharmacologic therapy. Thus, mixtures will need to be designed carefully. But the risk of adverse effects can be minimized through the use of relatively nontoxic compounds. In addition, keeping doses as low as possible can reduce risks. Because of additivity and synergism in a well-designed mixture, relatively low doses of individual compounds may still be capable of producing a desired effect without excessive risk of toxicity.

Pharmacokinetic issues are also important when considering the medical and health effects of plant compounds. While many otherwise interesting compounds exhibit poor ADME characteristics, a good number of these could still be useful. For example, the vehicle used to deliver the compounds could be altered to improve the pharmacokinetics. In some cases, enteric-coated tablets, emulsifiers, complexing agents, or lipid-based formulations might be useful. In addition to manipulations based on physical pharmacy, the act of using mixtures of compounds can in some cases affect ADME characteristics. For example, the ability of grapefruit juice to affect drug metabolism is now well established. This is discussed more in Chapter 14 in relation to adverse drug interactions. But beneficial effects on ADME characteristics are also possible. For example, hypericin, thought to be one of the active constituents of Hypericum perforatum (St. John’s wort), is nearly insoluble in water. Jürgenliemk and Nahrstedt (2003) showed that some phenolic constituents typical for Hypericum extracts increased the concentration of hypericin in the water phase by up to 400-fold. Butterweck et al. (2003) showed that the oral bioavailability of hypericin was increased if administered with hyperoside, also found in Hypericum extracts. In another example, Gawande et al. (2008) found that oral administration of a black grape extract along with (–)-epigallocatechin gallate (EGCG), a green tea component, increased the systemic availability of EGCG in humans.

We see then that interactions in a mixture may be due to pharmacodynamic or pharmacokinetic events (Spinella, 2002). In the former, the effects are due to two or more drugs acting on single or multiple regulatory proteins. Such interactions are often directly related to the binding affinity or membrane-altering ability of the drugs. In contrast, pharmacokinetic interactions are due to influences of a compound on another’s ADME characteristics.

2 Physical Models of Drug Interaction – Protein Binding and the Plasma Membrane

As discussed above, drugs in a mixture may interact by binding to target proteins. For convenience, we use the term drug here and in the remaining portions of this chapter to refer to both approved drugs and bioactive plant compounds. The binding of a drug to a protein may affect the function of that protein through a number of mechanisms. For example, if the protein were an enzyme that binds substrate S, the drug could bind near the active site of the enzyme, thereby sterically inhibiting the binding of S. If the enzyme contains an allosteric binding site, distant from the active site, the drug may bind to the allosteric site and thereby affect the conformation of the active site and its ability to bind S. When multiple drugs affect a single protein, the complexity of binding patterns and the mechanical alteration of protein function will influence the type of interaction produced: additive, synergistic, or antagonistic. If multiple proteins are involved, the complexity of the (signaling) network in which the target proteins exist will also influence the type of interaction produced.

As a simple example, targets could be two intracellular enzymes serially connected in a signaling pathway. In this case, inhibition of an enzyme by each drug in a binary mixture might produce additive effects. In more complex cases, drugs may affect multiple proteins in a signaling pathway that contains positive and/or negative feedback loops or drugs may affect proteins that are involved in distinct but connected signaling pathways. As the number of bound proteins and the complexity of the signaling pathways increase, there are increasing opportunities for antagonistic or synergistic interactions.

In addition to intracellular proteins, drugs may also bind to protein targets on the plasma membrane. In particular, they may bind to transmembrane receptors embedded in the lipid bilayer. Many signals that originate outside of the cell enter the cell via cell surface receptors. Growth factors, such as epidermal growth factor (EGF), are an example of an extracellular signal. Extracellular EGF binds to EGF receptors (EGFR) on the plasma membrane, and the resulting signal is propagated into the cell, eventually reaching the nucleus and causing proliferation.

Drugs can directly bind to receptor proteins on the plasma membrane, in some cases stimulating signal transduction and in other cases inhibiting it. For example, plant compounds that bind weakly with estrogenic receptors may physically block the binding of more potent ligands, thereby producing an antiestrogenic effect. Genistein, from soybean, is reported to act in part by this mechanism in some experimental models (Kogiso et al., 2006). If multiple therapeutic compounds are used and multiple receptor types are affected, downstream signaling pathways may interact in additive, antagonistic, or synergistic ways.

The interaction of drugs can be influenced by the properties of the plasma membrane that contains the receptors. Although the membrane has been described as a system driven by thermodynamic equilibrium (Aon et al., 1996), it is more accurately seen as an emergent structure consisting of highly asymmetrical structures and undergoing dynamic transitions (Perillo, 2002). Typically, mammalian cellular plasma membranes consist of about eight major classes of lipids (Simons and Vaz, 2004) and also include a variety of proteins embedded in the bilipid structure. The plasma membrane serves a number of purposes, including protection, endocytosis, signaling, and mechanical stability. It must be rigid enough to protect the cell and offer stability, but at the same time, it must be dynamic and pliable enough to allow cell deformation and promote adaptation to diverse environmental messages. Either directly or indirectly, the characteristics of the lipid membrane affect nearly every activity that occurs in a cell.

One way that the membrane affects signaling is by supporting the dynamic creation and movement of lipid rafts (also known as membrane rafts), which are clusters of proteins that horizontally “float” in the membrane. One report has identified as many as 250 proteins that exist in lipid rafts (Patra, 2008). A good number of these proteins, including ras (ras is a signal transduction protein that belongs to a large superfamily of low-molecular-weight G proteins) and EGFR, are cell surface receptors. Several authors have postulated or shown that receptors in a raft can cooperate to affect each other’s conformation, thereby coordinating the overall response to a ligand (Duke and Bray, 1999; Graham and Duke, 2005; Fuxe et al., 2008; Sourjik, 2004). For example, consider a receptor that switches probabilistically between two conformations, active and inactive, and binding of a ligand stabilizes the active state. Certain cellular responses, such as chemotaxis, in response to a chemoattractant, are most useful if they are binary. For example, it might be beneficial if a cell moves toward a weak stimulus with the same force that it moves toward a stronger stimulus. This requires that as a group, the receptors act like an on/off switch. One way to accomplish this is by allowing adjacent receptors in a raft to influence the conformation, active or inactive, of one another. Above a critical but low ligand concentration, a small percentage of receptors are bound, but these cause unbound receptors to switch to the active conformation. In a more complicated scenario, receptors may simultaneously bind two ligands, and in this case, receptor–receptor interactions may produce the equivalent of AND/OR logic gates. For example, if a cell senses both poison and chemoattractant in the same location, it is beneficial if the cell does not move toward the chemoattractant.

The switching characteristics mentioned above are dependent on receptor–receptor interactions, which in turn are dependent upon the characteristics of lipid rafts. The notion that membrane characteristics may influence the type of drug interaction (additive, antagonistic, or synergistic) begs the question of whether some drugs may act directly on the plasma membrane itself, in addition to or in contrast to protein binding. Indeed, this seems to occur for a good number of drugs and bioactive plant compounds. It is well known that hydrophobic drugs tend to interact with biological membranes (Schreier et al., 2000). At high concentrations, they can act like detergents and disrupt membranes, while at low concentrations, they tend to stabilize membranes, such as protecting red blood cells from hemolysis. Many bioactive plant products are also hydrophobic and can be expected to interact with membranes. For example, some flavonoids have been shown to affect lipid viscosity. In addition, flavonoids preferentially located in the hydrophobic portion of the bilayer have been shown to initiate the formation of raft-like domains, whereas those located in the polar interface region can fluidize membranes and have a raft-breaking effect (Tarahovsky et al., 2008). In a study on human colon cancer cells, the flavonoid quercetin was shown to induce the accumulation of cell death receptors in lipid rafts and thereby facilitate apoptosis (programmed cell death) in response to death-inducing signals (Psahoulia et al., 2007). Adachi et al. (2007) reported that EGCG from green tea inhibited the binding of EGF to EGFR and the subsequent activation of EGFR by altering membrane organization related to lipid rafts. As a last example, omega-3 fatty acids (EPA, eicosapentaenoic acid, and DHA, docosahexaenoic acid) have been shown to inhibit the proliferation of human breast cancer cells in vitro, in part by reducing EGFR levels in lipid rafts (Schley et al., 2007).

In summary, the type of interaction produced by a drug mixture is influenced by the complexity of protein binding patterns, the complexity of the network in which the bound proteins interact, the degree of receptor–receptor interactions, and the effects of drugs on the plasma membrane, particularly on the formation and composition of lipid rafts. In the interest of brevity, other mechanisms by which a drug can affect cellular function have not been discussed. For example, some drugs can bind directly to DNA molecules. Many of these drugs, however, tend to exhibit high systemic toxicity. Drugs can also affect cellular function by acting as pro- or antioxidants.

3 Quantifying Synergism Using Nonlinear Mixed-Effects Modeling

In the following discussion on mathematical models of drug interaction, we shift to a more technical tone. In particular, the reader is given a complete mathematical description of the MixLow method for assessing interaction indices developed by Boik et al. (2008). An implementation of the MixLow method is currently available in the R language (package mixlow) and can be downloaded from the CRAN web site (http://cran.r-project.org/). Although details of the model are presented here, the R package can be used without a complete understanding of the mathematics involved. For those readers who are not biostatisticians, the details given here should provide a general insight into the many issues involved in estimating an interaction index, and for those who do become users of the mixlow package or other software, the details should provide useful reference material.

3.1 Background

Over the last few decades, several mathematical methods have been proposed to assess synergism between drugs in a mixture.¹ All of these are based on some index of additivity (or null interaction). There has been disagreement on a strict mathematical definition of additivity and reviews have been published discussing the various proposals (Berenbaum, 1989; Greco et al., 1995; Merlin, 1994; Tallarida, 2001). Two indices that have gained widespread acceptance are those for Loewe additivity and Bliss independence (Greco et al., 1992). The Loewe additivity index forms the basis for the present work, as well as the basis for isobolograms (Poch, 1990), which are graphical assessments of additivity. In contrast to other indices, particularly that of Bliss independence, the Loewe index produces the intuitively reasonable result that a sham mixture, a mixture of a drug with itself, is additive. In the MixLow method, developed by Boik et al. (2008), estimation of the Loewe index is a three-step process: (1) estimate parameter values that define the shape of concentration–response curves for the mixture and its component drugs; (2) use the estimated parameters in calculating the Loewe index; and (3) generate confidence intervals of the index.

Methods to assess synergism can be distinguished not only by the interaction index used but also by the experimental design used to obtain data, the model used to estimate parameters of the concentration–response curves, and the dimensions of the resulting interaction index plot (two- or three-dimensional). Experimental designs are usually either factorial, where concentrations of each drug are crossed (or partially crossed) with concentrations of other drugs, or fixed-ratio, where concentrations of all drugs are fixed at a constant ratio. If fixed-ratio concentrations in a two-drug mixture are graphed, where each axis represents the concentration of one drug, then the plotted points fall on a straight line, or ray, extending out from the origin. With the fixed-ratio design, synergism can be assessed either along one ray or along a three-dimensional response surface based on a series of rays. The current MixLow method assesses data from a single ray.

The factorial design is used primarily when a three-dimensional response surface (additivity surface) is desired. An example of a factorial design is given by Martinez-Irujo et al. (1996). Examples of response surface methods for multiray data include those by White et al. (2003, 2004), Minto et al. (2000), and Fidler and Kern (2006). While response surface methods do provide more information than can be obtained from the analysis of single-ray experiments, they require that more data be collected. For this reason, single-ray experiments remain common.

The de facto standard for assessing synergism in single-ray experiments is the median-effect method of Chou and Talalay (1984). This method estimates concentration–response curve parameters by using log linearization and ordinary least squares. The method presented here, dubbed the MixLow (Mixed-effects Loewe) method for convenient reference, is similar to the median-effect method in that it assesses data from single-ray experiments, presents results in graphical form (as a plot of fraction affected versus combination index; see below), and uses the Loewe index to define additivity. Unlike the median-effect method, however, it employs a mixed-effects model to estimate parameters of concentration–response curves. This approach allows more accurate estimation of concentration–response curve parameters and can also produce confidence intervals with improved coverage. Coverage is the probability that a confidence interval method captures the true parameter. If the coverage differs markedly from the nominal confidence coefficient (typically 0.95), then the confidence intervals are of questionable value. Confidence intervals for the interaction index, as well as accurate parameter estimators, are vital to fully assess whether drugs in a mixture interact synergistically, antagonistically, or additively. By extension, knowledge of the coverage of these intervals is also vital. As reported in Boik et al. (2008), in a series of simulations, the MixLow method produced confidence intervals with excellent coverage properties.

The approach described here is applicable to the common situation where within-unit and between-unit measurements are available, responses follow a sigmoidal pattern,² and ratios between drugs in a mixture are fixed (that is, various dilutions of the mixture and its component drugs are tested). While such data could be generated in many types of experiments, the discussions here focus on data obtained from in vitro cytotoxicity experiments, where cancer cells are exposed to a drug for a specified length of time (typically 72 hours) and then cell viability is indirectly measured, usually via fluorescence readings after addition of a suitable dye. Such cytotoxicity assays use multiwell incubation trays, where each tray receives one drug or mixture, each column of the tray typically receives a different drug concentration, and replicate trays are tested for each drug and mixture. The experimental unit in this situation is the incubation tray.

Regardless of the drugs tested and assay employed, cytotoxicity responses occur in a nonlinear relationship with drug concentration. If a parametric model is employed, either the relationship can be linearized prior to estimation of concentration–response curve parameters, as is done in the median-effect method, or parameters can be estimated using a nonlinear model, as is done in the MixLow method. Furthermore, when using a nonlinear (or linear) model, effects can be either fixed or random. A mixed-effects model contains both fixed and random effects. Random effects are commonly associated with observations sharing the same level of a classification factor. In this paper, that factor is incubation tray. Nonlinear mixed-effects models are widely used in medicine, particularly to assess pharmacokinetic data. Their use is rare, however, with cytotoxicity data.

In the median-effect method, raw data must be preprocessed. This is accomplished by (1) scaling data by control-well means – responses are averaged across in-tray replicates and divided by average responses of in-tray control wells – and (2) taking the log of a function of the scaled data. Each of these steps can have detrimental effects on the precision of parameter estimators, and the MixLow method does not require any preprocessing.

3.2 MixLow Method

Let the random variable Fa signify the fraction of cells affected by a drug concentration. Define $\phi = {\rm E} \left[ {Fa} \right]$, where E [•] is the expected value. In some contexts, $\phi$ is estimated based on concentration–response data, and in other contexts, a concentration is estimated that results in a fixed value of $\phi$. Denote by $\psi_{d,\phi }$ the $\phi$-effective log concentration of drug d. This is the log concentration that produces a fraction affected equal to a fixed . For example, the log concentration of drug d that inhibits proliferation of a cell population by 10% relative to controls is denoted by $\psi_{d,0.1}$. By convention, $\exp (\psi_{d,0.1} )$ is referred to as the IC10 (10% inhibitory concentration).

The MixLow method fits a sigmoid curve to concentration–response data and uses the resulting parameter estimates to estimate an interaction index. The sigmoidal curve is parameterized by two constants, $\psi_{d,0.5}$ and a shape parameter denoted by $\gamma_d$. The shape parameter indexes the steepness of the concentration–response curve. At the IC50, the slope of the curve is ${{0.25\gamma_d } \mathord{\left/ {\vphantom {{0.25\gamma_d } {\exp (\psi }}} \right.} {\exp (\psi }}_{d,0.5} )$.

The Loewe index, which is used by both the median-effect and the MixLow methods, provides a measure of drug interaction. For two drugs, the Loewe index and its estimator³ are

$$L_\phi = \sum\limits_{d = 1}^2 {\frac{{\exp (m_{d,\phi } )}}{{\exp (\psi_{d,\phi } )}}} \,\,{\rm{and}}\,\,\hat L_\phi = \sum\limits_{d = 1}^2 {\frac{{\exp (\hat m_{d,\phi } )}}{{\exp (\hat \psi_{d,\phi } )}}},$$

((10.1))

respectively, where $m_{d,\phi }$ is an unknown constant signifying the log concentration of drug d in the mixture when the mixture is at its -effective log concentration and $\psi_{d,\phi }$ is the unknown -effective log concentration of drug d alone. The mixture is antagonistic, additive, or synergistic at depending on whether the value of the Loewe index is greater than 1, equal to 1, or less than 1, respectively. Chou and Talalay (1984) were apparently the first to use plots of $\hat L_\phi$ versus as a summary of drug interactions.

3.3 Basic MixLow Model

For clarity of presentation, the basic MixLow model is introduced first. Later, a modified model is discussed. The MixLow model uses a nonlinear mixed-effects framework to represent the concentration–response curve. As a skeleton description, responses are modeled as the expected mean of control wells times a sigmoidal function, plus an error term. Sigmoidal models of this type are sometimes referred to as Hill models. Formally, responses, $\left\{ {Y_{d,t,w} } \right\}$, obtained from unprocessed data are modeled as a sigmoidal function of the concentration:

$$Y_{d,t,w} = \exp \left( {\mu + b_t } \right)\left( {1 - \phi_{d,t,w} } \right) + \varepsilon_{d,t,w},$$

((10.2))

where

$$\phi_{d,t,w} = 1 - \frac{1}{{1 + \left( {\frac{{\exp (c_{d,t,w} )}}{{\exp (\psi_{d,0.5} )}}} \right)^{\gamma_d } }},$$

((10.3))

and the subscripts d, t, w refer to the dth drug, tth tray, and wth well, respectively.⁴ Here, drug d could refer to a single drug or a mixture, and $c_{d,t,w}$ refers to the log of the drug concentration for the dth, tth, and wth observation, a known constant. The expected value of $\exp \left( {\mu + b_t } \right)$ refers to the expected mean of control wells from all trays, where $b_t$ is a random deviate specific to tray t. Values $\left\{ {b_t } \right\}$ are independently distributed as $b_t \sim{\rm N} (0,\,\,\sigma_b^2 )$. The error terms in Model (10.2) are independently distributed as $\varepsilon_{d,t,w} \sim{\rm N} \left( {0,\,\,\,\,f(\sigma ^2 ,\,\,{\rm E} \left[ {Y_{d,t,w} |b_t } \right])} \right)$, where f(•) is a function discussed later.

Differences in control-well means across trays occur for a variety of reasons. For example, if one tray in a group of replicates is seeded with a higher density of cells, the responses in the tray will tend to be higher than those of the other trays. Differences can also occur because of differential handling of trays, differential growth conditions (e.g., incubating trays on different days), differential assay procedures (e.g., allowing one tray to incubate for a slightly longer time before reading results), and/or random biologic variations in cell proliferation.

Having explained the model, the procedures for calculating the Loewe index and its confidence intervals are described. An n-drug generalization of the Loewe index in (10.1) is given by

$$L_\phi = \sum\limits_{d = 1}^n {\exp \left( {m_{d,\phi } - \psi_{d,\phi } } \right)} = \sum\limits_{d = 1}^n {\tau_d \,\exp \left( {\psi_{m,\phi } - \psi_{d,\phi } } \right)},$$

((10.4))

where $\psi_{d,\phi }$ is the -effective log concentration of drug d alone, $\psi_{m,\phi }$ is the -effective log concentration of the mixture, and $\tau_d$ is the fraction of the mixture that is composed of drug d. Note that $m_{d,\phi } = \log (\tau_d ) + \psi_{m,\phi }$.

If $c_{d,t,w}$ in (10.3) is equated to the -effective log concentration, $\psi_{d,\phi }$, then $\phi_{d,t,w}$ becomes . That is,

$$\phi = 1 - \frac{1}{{1 + \left( {\frac{{\exp (\psi_{d,\phi } )}}{{\exp (\psi_{d,0.5} )}}} \right)^{\gamma_d } }}.$$

((10.5))

To write the Loewe index as an explicit function of $\psi_{d,\phi }$, first solve (10.5) for $\psi_{d,\phi }$ to obtain

$$\psi_{d,\phi } = \log \left( {\left( {\frac{\phi }{{1 - \phi }}} \right)^{\frac{1}{{\gamma_d }}} } \right) + \psi_{d,0.5}.$$

((10.6))

Second, substitute the expression for $\psi_{d,\phi }$ in (10.6) into (10.4) to obtain

$$\begin{array}{c} \hat L_\phi = \sum\limits_{d = 1}^n {\tau_d \,\exp \left( {\hat \psi_{m,\phi } - \hat \psi_{d,\phi } } \right)} \\ = \sum\limits_{d = 1}^n {\tau_d \,\exp \left( {\log \left( {\left( {\frac{\phi }{{1 - \phi }}} \right)^{\frac{1}{{\hat \gamma_m }}} } \right) + \hat \psi_{m,0.5} - \log \left( {\left( {\frac{\phi }{{1 - \phi }}} \right)^{\frac{1}{{\hat \gamma_d }}} } \right) - \hat \psi_{d,0.5} } \right)} \\ \end{array},$$

((10.7))

which is written here as the estimator of the index, where

$$\hat \psi_{d,\phi } = \log \left( {\left( {\frac{\phi }{{1 - \phi }}} \right)^{\frac{1}{{\hat \gamma_d }}} } \right) + \hat \psi_{d,0.5}.$$

((10.8))

Confidence intervals of $L_\phi$ can be based on the standard error of $\hat L_\phi$ or its log transformation ${\rm{SE}}\left( {\log (\hat L_\phi )} \right)$. The latter approach ensures that confidence interval limits are positive and this is the approach used here. The standard error of the transformed index is obtained using the Delta method as follows:

$${\rm{SE}}\left( {{\rm{log}}(\hat{\rm L}_\phi )} \right) \approx {\left( {\left( \begin{array}{l} \frac{{\partial \log (\hat L_\phi )}}{{\partial \hat \Psi }} \\ \frac{{\partial \log (\hat L_\phi )}}{{\partial {\hat {\upgamma }}}} \\ \end{array} \right)^{\rm{T}} {{\rm v}\hat{\rm a}{\rm r}} \binom{{\hat {{\uppsi }}}}{{\hat {{\upgamma }}}} \left( \begin{array}{l} \frac{{\partial \log (\hat L_\phi )}}{{\partial \hat \Psi }} \\ \frac{{\partial \log (\hat L_\phi )}}{{\partial {\hat {\upgamma }}}} \\ \end{array} \right)} \right),}^{1/2} $$

((10.9))

where the superscript T refers to transpose. The term ${{\rm v}\hat {\rm a}{\rm r}} \binom{\hat{\uppsi}}{{\hat {\upgamma}}}$ is obtained from the observed information matrix, which is produced when Model (10.2) is fit by maximizing the likelihood function. The confidence interval for the Loewe index at is calculated as

$$\exp \left( {\log (\hat L_\phi ) \pm t_{df,1 - {\alpha/2}} {\rm{\,\,\,\,\,\,SE}}\left( {\log (\hat L_\phi )} \right)} \right),$$

((10.10))

where $t_{df,1 - {\alpha/2}}.$ is a multiplier obtained from the t-distribution with df degrees of freedom and α significance level.

Within-group errors for cytotoxicity data tend to be heteroscedastic, with smaller errors at low ${\rm E} \left[ {Y_{d,t,w} |b_t } \right]$ values and larger errors at high ones. To allow for this, the error term in Model (10.2), $\varepsilon_{d,t,w}$, can be modeled as a function of $\sigma$ and ${\rm E} \left[ {Y_{d,t,w} |b_t } \right]$. Table 10.1 lists four error functions, e1–e4, which are useful for cytotoxicity data. Note that all functions listed belong to a single family of functions. For example, e4 and e3 are identical when $\beta_d$ is equal to 1.0 for all drugs. The error functions listed in Table 10.1 stem from straightforward application of variance options for the nlme function in R (http://cran.r-project.org/).

Table 10.1 Error functions

3.4 Modified MixLow Model

The MixLow model can be modified to allow for a “double” sigmoid response pattern. The extension is similar in spirit to modification of a single Hill model to a “double” one. A double-sigmoid response pattern has been observed for a number of drugs in various cell lines (Levasseur et al., 1998; Yeung et al., 1999). The modification to the basic MixLow model is straightforward. Model (10.2) is modified to represent a weighted sum of two sigmoidal curves, which is as follows:

$$Y_{d,t,w} = \,\,\left( {1 - \lambda _d } \right)\exp \left( {\mu + b_t } \right)(1 - \phi^{\prime}_{d,t,w} ) + \lambda _d \exp \left( {\mu + b_t } \right)(1 - \phi^{\prime\prime}_{d,t,w} ) + \varepsilon _{d,t,w}, $$

((10.11))

where

$$\phi^{\prime}_{d,t,w} = 1 - \frac{1}{{1 + \left( {\frac{{\exp (c_{d,t,w} )}}{{\exp (\psi^{\prime}_{d,0.5} )}}} \right)^{\gamma^{\prime}_d} }},\,\,\,\,\, \phi^{\prime\prime}_{d,t,w} = 1 - \frac{1}{{1 + \left( {\frac{{\exp (c_{d,t,w} )}}{{\exp (\psi^{\prime\prime}_{d,0.5} )}}} \right)^{\gamma^{\prime\prime}_d} }},$$

((10.12))

and $0 \le \lambda_d \le 0.5$ is a drug-dependent weight.

As illustrated in Fig. 10.1, each of the component curves and their weighted sum converge to zero as the drug concentration increases. The term $\phi^{\prime}_{d,t,w}$ refers to the expected fraction affected for the first weighted curve; the corresponding IC50 is denoted by $\exp \left( {\psi^{\prime}_{d,0.5}} \right)$ and the slope by $\gamma^{\prime}_d$. Parameters for the second curve are defined analogously. The maximal expected response of the weighted sum is ${\rm E} \left[ {\exp \left( {\mu + b_t } \right)} \right]$. The maximal expected responses for the first and second weighted curves are $(1 - \lambda_d )\,{\rm E} \left[ {\exp \left( {\mu + b_t } \right)} \right]$ and $\lambda_d \,{\rm E} \left[ {\exp \left( {\mu + b_t } \right)} \right]$, respectively. If $\lambda_d$ is equal to zero, then Model (10.11) collapses to Model (10.2).

Fig. 10.1

Example of double-sigmoid curve

Denote the fraction affected according to Model (10.11) by . That is,

$$1 - \phi_{d,t,w} = \,\,\frac{{{\rm E} \left[ {Y_{d,t,w} } \right]}}{{{\rm E} \left[ {Y_{0,t,w} } \right]}} = \left( {1 - \lambda_d } \right)(1 - \phi^{\prime}_{d,t,w} ) + \lambda_d (1 - \phi^{\prime\prime}_{d,t,w} ).$$

((10.13))

If $c_{d,t,w}$ in (10.12) is equated to the -effective log concentration $\psi_{d,\phi }$, then $\phi_{d,t,w}$ becomes and

$$(1 - \phi ) = \frac{{\left( {1 - \lambda_d } \right)\,}}{{1 + \left( {\frac{{\exp (\psi_{d,\phi } )}}{{\exp (\psi^{\prime}_{d,0.5})}}} \right)^{\gamma^{\prime}_d} }} + \frac{{\lambda_d}}{{1 + \left( {\frac{{\exp (\psi_{d,\phi } )}}{{\exp (\psi^{\prime\prime}_{d,0.5})}}} \right)^{\gamma^{\prime\prime}_d}}}.$$

((10.14))

The solution to (10.14) for $\psi_{d,\phi }$ is the root of the polynomial

$$(1 - \phi )\exp (\psi_{d,\phi } )^{\gamma^{\prime}_d + \gamma^{\prime\prime}_d} + a\,\exp (\psi_{d,\phi } )^{\gamma^{\prime}_d } + b\,\exp (\psi_{d,\phi } )^{\gamma^{\prime}_d} + c = 0,$$

((10.15))

where $a = (1 - \phi - \lambda_d )\,\exp (\psi^{\prime\prime}_{d,0.5} )^{\gamma^{\prime\prime}_d}$, $b = (\lambda_d - \phi )\,\exp (\psi^{\prime}_{d,0.5} )^{\gamma^{\prime}_d}$, and $c =-$ $\exp (\psi^{\prime\prime}_{d,0.5} )^{\gamma^{\prime\prime}_d} \exp (\psi_{\prime}{_{d,0.5}} )^{\gamma^{\prime} _d} \phi$. Equation (10.15) has at most one real root because the weighted sum of curves is monotonic decreasing by construction. To obtain an estimator of $\psi_{d,\phi }$, namely $\hat \psi_{d,\phi }$, substitute estimators $\left( {\hat \lambda_d, \hat \gamma^{\prime}_d, \hat \gamma^{\prime\prime}_d, \hat \psi^{\prime}_{d,0.5}, \hat \psi^{\prime\prime}_{d,0.5}, \hat \psi_{d,\phi}} \right)$ for parameters $\left( {\lambda_d ,\gamma^{\prime}_d ,\gamma^{\prime\prime}_d, \psi_{\prime}{d,0.5}, \psi^{\prime\prime}_{d,0.5}, \psi_{d,\phi}} \right)$ in (10.15) and label the root of (10.15) as $\hat \psi_{d,\phi}$.

The estimator $\hat \psi_{d,\phi }$ is substituted into (10.4) to obtain the Loewe index as a function of φ. Although the estimator is not in closed form, the implicit function theorem can be used to obtain derivatives for computing confidence intervals using equations similar to (10.9) and (10.10).

In practice, drug solubility or other issues can limit the concentration at which a drug can be tested. If the drug produces a double-sigmoid response, but high enough concentrations were not tested to see the full curve, then $\psi^{\prime\prime}_{d,0.5}$ is higher than the highest concentration tested and it can appear as if the drug produces a nonzero asymptotic response. This situation is not uncommon in practice. It presents a problem in that $\psi_{d,\phi}$ cannot be estimated for $\phi > 1 - \mathop {\max }\limits_d (\hat \lambda_d )$. When φ is greater than $1 - \mathop {\max }\limits_d (\hat \lambda_d )$, the data only tell us that $\hat \psi_{d,\phi }$ is not lower than the largest concentration tested. To account for this, $\psi^{\prime\prime}_{d,0.5}$ can be fixed at infinity and Model (10.11) changed to

$$Y_{d,t,w} = \left( {1 - \lambda_d } \right)\,\exp \left( {\mu + b_t } \right)(1 - \phi^{\prime}_{d,t,w} ) + \lambda_d \,\exp \left( {\mu + b_t } \right) + \varepsilon_{d,t,w},$$

((10.16))

where the Loewe index and its confidence intervals are computed only for $\phi < 1 - \mathop {\max }\limits_d (\hat \lambda_d )$. An equation analogous to (10.15) is used to obtain an expression for $\hat \psi_{d,\phi }$.

3.5 Use of the MixLow Method

While the equations given above may appear complex to some readers, the use of the MixLow method is relatively straightforward. The R package mixlow can be downloaded from the CRAN website, and a paper describing the package and its use has been submitted (Boik and Narasimhan, 2008). In brief, the following six steps are performed in sequence:

• 1. Design the experiment, collect the data, and prepare the data file. An example data file is included with the R package.

• 2. Read the data file using the function readDataFile.

• 3. Prepare and subset the data using the function prepareData.

• 4. Obtain starting values for the fixed-effects parameters of the nonlinear mixed-effects (NLME) model by using nonlinear least squares (NLS) analysis. This is done using the function doNls.

• 5. Use the estimated NLS parameters as starting values for the NLME model. The mixed-effects model is estimated using the function doNlme.

• 6. Use the estimated NLME parameters to estimate the Loewe index. This is done using the function doLoewe.

In summary, the process is: read data → prepare data → NLS → NLME → Loewe. Figure 10.2 shows the Loewe index versus fraction-affected values for an example data set. Confidence intervals of the index are shown by dotted lines. In the figure, statistically significant synergism is occurring between fraction-affected values of about 0.1–0.85 (the upper confidence interval is less than 1.0).

Fig. 10.2

Example of Loewe index versus fraction affected