Introduction

The randomized controlled trial design (RCT) is considered the gold standard for establishing the efficacy of a clinical intervention. This paper distinguishes between randomized controlled trials (RCT), where the central feature is random assignment of individuals to treatment conditions and cluster randomized controlled trials (CRCT), where clusters (i.e., groups of related individuals) are randomly assigned to treatment conditions, and wherein treatments address conditions that affect such groups as a whole. The present paper discusses the assumptions of randomization and argues for the need to consider interactions between context, treatment and outcomes in cluster randomized controlled trials of population health interventions.

Population health interventions and the cluster randomized controlled trial

Population health interventions are defined as “policies or programs within or outside of the health sector and have the potential to impact health at the population level” (Hawe and Potvin 2009). Population health interventions are thus “treatments” that are provided to a group as a whole which comprises individuals, their interrelationships and context. Treatments in population health interventions are often conceptualized as attempts through programs and policies to change social context that influence health.

When examining interventions, the strength of the RCT lies in two consequences of random assignment to treatment conditions (Fisher 1935). First, all participant background variables are balanced prior to the beginning of the experiment. Second, the treatment assignment is unrelated to all participant background variables, either hypothesized as confounding or non-confounding; the correlation between treatment exposure and any other variable that could influence treatment outcome is assumed to be null. Despite the strengths of the RCT for establishing the efficacy of a clinical intervention, there is considerable debate about the appropriateness of the RCT design for population health interventions (Bonell et al. 2011; Cousens et al. 2011; Craig et al. 2008; Macintyre 2011).

Limitations of the RCT have lead to alternative research designs for interventions with groups or populations (West et al. 2008). The CRCT has gained attention in population health intervention research, because it maintains randomization and overcomes some of the limits of the RCT. The number of publications reporting results from CRCTs has increased from approximately 10 papers published in 1995 to nearly 100 papers published in 2003 (Bland 2004). The CONSORT (Consolidated Standards of Reporting Trials) statement originally published in 1996 was extended and modified in 2004 to include CRCTs (Campbell et al. 2004).

Increasing use of the CRCT design has lead to methodological and statistical debate (Edwards et al. 1999; Eldridge et al. 2004). Limited discussion has focused on the interaction between treatment and social context in a CRTC. Specifically, if the constant effect assumption (i.e., effect homogeneity) of treatment does not hold, interactions with social context are not removed and we argue that it should become the primary object of study. An early formulation of this argument is due to Kemm (2006), “context affects both the way that an intervention operates and the outcomes. After selection for entry, the RCT aims to remove the effect of the context, being based on the assumption that comparison groups are alike in all respects apart from the intervention and that the conclusions of the trial will apply equally to all members of the group. This assumption of comparable groups is rarely justified when the unit of intervention is a community rather than an individual (p. 322).”

Causality, counterfactual and the constant effect assumption underlying the RCT

David Hume defined causality as constant conjunction between two events; the result of a comparison between what is observed following a given event and what would have occurred had this event not occurred (Hume 1739). Causality is always inferred. The concept of the counterfactual attempts to capture this inferred comparison. ‘A counterfactual is a condition that would occur if some part of the world were different than it really is. Under a counterfactual theory, causal statements are counterfactual statements (p. 7)’ (Shadish 2010). The “true” effect of a cause can never be known, it is always a plausible explanation of observed conjunctions of events under certain assumptions. Donald Rubin demonstrated that in addition to statistical assumptions requiring a sufficient number of experimental units, other assumptions are necessary to make valid causal inference based on individual or cluster randomised trials (Rubin 1974).

For Rubin (1974) there is a “fundamental problem of causal inference” under counterfactual theory as exposed in Eqs. 1–6. Each unit (or individual) u in a population U is potentially exposable to a treatment t or its absence c. For each unit u there exist a post exposure response variable Y(u) that will take the value of Y t (u) for treatment t and Y c (u) for treatment c. The effect of t on u as measured by Y and compared to c is thus given by:

$$ Y(u)\, = \,Y_{t} (u)\, - \,Y_{c} (u) $$
(1)

It is impossible to observe the value of Y t(u) AND Y c(u) on the same unit at the same time. It is, therefore, impossible to observe the effect of t relative to c on u. So the “fundamental problem of causal inference” is to find a way to estimate the value of Y c(u). In a RCT, the strategy for estimating the value of Y c(u) consists in making use of the population U from which u is selected and of statistical theory (Holland 1986).

Equations 2–6 are taken from Holland (1986) who demonstrated that in a population, the average causal effect T over all units u in a population U is the expected value of (1), which is equivalent to:

$$ E(Y_{t} \, - \,Y_{c} )\, = \,T $$
(2)
$$ T\, = \,E(Y_{t} )\, - \,E(Y_{c} ) $$
(3)

Equation 3 simply means that the effect of T in a population U is the difference of the expected value on the outcome measure Y for units exposed to t, compared to the expected value on the outcome measure Y for units exposed to c.

In a large population U, randomization works in such a way that a variable S which can take the values t or c is randomly assigned to units u leading to

$$ E(Y_{t} ) = E\left( {Y_{t} /S = t} \right) $$
(4)
$$ E(Y_{c} ) = E\left( {Y_{c} /S = c} \right) $$
(5)

Replacing the values of E(Y t ) and E(Y c ) in Eq. 3 by those provided in Eqs. 4 and 5 leads to:

$$ T = E\left( {Y_{t} /S = t} \right) - E\left( {Y_{c} /S = c} \right) $$
(6)

This last equation means that the effect T associated with treatment t compared to c is an average over all units u. Causal inference about t on any individual unit u 0 assumes constant effect, which means that the treatment t is not dependent on any attribute of the unit u.

Methods

CRCT and the violation of the constant effect assumption

There are two reasons why the constant effect assumption is difficult to meet in CRCT. One is statistical and concerns within cluster variation which applies to all randomized trials, while the other, we argue, relates specifically to how one conceives of effective treatment mechanism in population health interventions.

Within cluster variation

In a CRCT, it is not enough to assume a constant effect on units u (in this case, a cluster made of individuals i), we must also assume a constant effect across individuals i within units. As demonstrated by Murray (1998), when units u are clusters, the variance of the group mean expected in an RCT under the assumption of independence of errors is:

$$ \sigma \tfrac{2}{{y_{c} }} = \frac{{\sigma_{y}^{2} }}{mg} $$
(7)

where g is the number of clusters, m is the individuals per cluster and \( \sigma_{y}^{2} \) is the within cluster variance.

This is not the same as the variance of the group mean expected in a CRCT, where the assumption of independence of errors is violated and the intraclass correlation (ICC = fraction attributable to the unit of assignment) given by:

$$ ICC_{m:g:c} = \frac{{\sigma_{g:c}^{2} }}{{\sigma_{e}^{2} + \sigma_{g:c}^{2} }} $$
(8)

where \( \sigma_{g:c}^{2} \) is the component of variance attributable to the unit of assignment and \( \sigma_{y}^{2} = \sigma_{c}^{2} + \sigma_{g:c}^{2} \), and the variance inflation factor (1 + (m − 1)ICCm:g:c) must be applied. As a result the conditional mean in a CRCT is:

$$ \sigma \tfrac{2}{{y_{c} }} = \frac{{\sigma_{y}^{2} }}{mg}(1 + (m - 1){\text{ICC}}_{m:g:c} ) $$
(9)

For the constant effect assumption to hold the within-cluster variance (σ 2e ) cannot differ between units u (i.e., clusters), because it is plausible for t to be dependent on the within cluster variance. For example, individuals inclusion in a cluster can depend on their characteristics and on average individual characteristics could vary between clusters (Puffer et al. 2003).

Randomizing u’s is an attempt to make the constant effect assumption plausible. This is true of all randomized trials. However, in CRCT of population health interventions this assumption is less plausible. CRCT of population health interventions often rely on small samples of randomized clusters. Given these small samples, there are myriad poorly understood individual, group and social phenomena we can imagine that make the assumption of constant effect of treatment across groups less plausible than, for example, in an RCT of a drug targeting a specific biological mechanism. This failure and the nature of population health interventions mean that the effect of context is not removed and the mechanism of intervention is not solely attributable to the treatment.

Effective mechanisms in population health interventions

Contemporary social theory has shown that a model that makes firm distinctions between individuals and social context, viewing treatments (i.e., population health interventions) as external and independent to both individuals and social context is oversimplified and may not best represent reality. There is an increasing agreement that a recursive relationship exists between individual, their practices and social context, whereas contextual conditions enable and constrain individual practices, those practices reify and transform context.

Conceptualizing the effect of a CRCT of a population health intervention as constant within and between groups as a result of randomization is both statistically and theoretically debated. We argue that randomization rarely completely disconnects the intervention from the existing conditions and the composition of the clusters in which it is implemented. A population health intervention should be conceptualized as more than solely a “treatment” that comes from outside and can be isolated using randomization. Conceptualizing a population health intervention as constant is unlikely to yield results that are theoretically sound and that estimate an unbiased average treatment effect. In population health interventions there are myriad individual, group and social phenomena at play, which make the constant effect assumption less plausible. Our proposition is that researchers using CRCT to study population health interventions should examine the interaction between treatment and social phenomena (Poland et al. 2008).

Results

Examining interactions between treatment and social phenomena in CRCT of population health interventions

We propose that additional tests are needed to understand the effect of population health interventions using CRCT. From a counterfactual perspective, two important elements must be considered. First, the assumption of constant effect can be partially verified by dividing the population U into subpopulations U1, U2,….Ui and estimating the average causal effect for each subpopulation T1, T2,…Ti. If the average causal effect does not vary across T’s, it provides some evidence that the constant effect assumption is plausible (Holland 1986). Variability across T’s biases the average treatment effect estimate but is not uninformative. Social theory should be used a priori to construct plausible hypotheses that can explain this variation.

Second, a long series of comparable observations on the same units prior to the intervention should be considered whenever possible. This would aid in defining the counterfactual, which requires that the effects of causes are relative to a specific, yet unobservable alternative. Using time series prior to randomization allows for a specific counterfactual rather than a counterfactual world barren from social context. Close examination of the social processes by which randomly allocated population health interventions are appropriate locally to produce effects should be part of the research report.

Discussion

The CRCT has gained attention in population health intervention research. We argue that the assumptions of randomization often do not hold. If assumptions fail, interactions between treatment and social phenomena become the primary object of study in CRCT of population health interventions. This is critical when studying interventions that aim to effect the social and environmental determinants of health (Cronbach 1975). Potential interactions between social phenomena, treatments and the desired outcome are likely and should be defined a priori using social theory. This would strengthen the causal claims of CRCT and simultaneously improve theorization about the relationship between social phenomena and health.