Abstract
Cluster randomized controlled trials are increasingly used in population health intervention research. Through randomization, researchers attempt to isolate the treatment effect and remove all other effects, including any effects of social context. In many cases, the constant effect assumption cannot be satisfied in cluster randomized controlled trials. We argue that when studying population health interventions, the effective mechanism of intervention lies in the interaction between the treatment and social context. Researchers should be cognizant that attempts to remove the effect of social context using CRTC may fail. The interaction between the treatment and social context should be the primary object of study in population health intervention research.
Avoid common mistakes on your manuscript.
Introduction
The randomized controlled trial design (RCT) is considered the gold standard for establishing the efficacy of a clinical intervention. This paper distinguishes between randomized controlled trials (RCT), where the central feature is random assignment of individuals to treatment conditions and cluster randomized controlled trials (CRCT), where clusters (i.e., groups of related individuals) are randomly assigned to treatment conditions, and wherein treatments address conditions that affect such groups as a whole. The present paper discusses the assumptions of randomization and argues for the need to consider interactions between context, treatment and outcomes in cluster randomized controlled trials of population health interventions.
Population health interventions and the cluster randomized controlled trial
Population health interventions are defined as “policies or programs within or outside of the health sector and have the potential to impact health at the population level” (Hawe and Potvin 2009). Population health interventions are thus “treatments” that are provided to a group as a whole which comprises individuals, their interrelationships and context. Treatments in population health interventions are often conceptualized as attempts through programs and policies to change social context that influence health.
When examining interventions, the strength of the RCT lies in two consequences of random assignment to treatment conditions (Fisher 1935). First, all participant background variables are balanced prior to the beginning of the experiment. Second, the treatment assignment is unrelated to all participant background variables, either hypothesized as confounding or non-confounding; the correlation between treatment exposure and any other variable that could influence treatment outcome is assumed to be null. Despite the strengths of the RCT for establishing the efficacy of a clinical intervention, there is considerable debate about the appropriateness of the RCT design for population health interventions (Bonell et al. 2011; Cousens et al. 2011; Craig et al. 2008; Macintyre 2011).
Limitations of the RCT have lead to alternative research designs for interventions with groups or populations (West et al. 2008). The CRCT has gained attention in population health intervention research, because it maintains randomization and overcomes some of the limits of the RCT. The number of publications reporting results from CRCTs has increased from approximately 10 papers published in 1995 to nearly 100 papers published in 2003 (Bland 2004). The CONSORT (Consolidated Standards of Reporting Trials) statement originally published in 1996 was extended and modified in 2004 to include CRCTs (Campbell et al. 2004).
Increasing use of the CRCT design has lead to methodological and statistical debate (Edwards et al. 1999; Eldridge et al. 2004). Limited discussion has focused on the interaction between treatment and social context in a CRTC. Specifically, if the constant effect assumption (i.e., effect homogeneity) of treatment does not hold, interactions with social context are not removed and we argue that it should become the primary object of study. An early formulation of this argument is due to Kemm (2006), “context affects both the way that an intervention operates and the outcomes. After selection for entry, the RCT aims to remove the effect of the context, being based on the assumption that comparison groups are alike in all respects apart from the intervention and that the conclusions of the trial will apply equally to all members of the group. This assumption of comparable groups is rarely justified when the unit of intervention is a community rather than an individual (p. 322).”
Causality, counterfactual and the constant effect assumption underlying the RCT
David Hume defined causality as constant conjunction between two events; the result of a comparison between what is observed following a given event and what would have occurred had this event not occurred (Hume 1739). Causality is always inferred. The concept of the counterfactual attempts to capture this inferred comparison. ‘A counterfactual is a condition that would occur if some part of the world were different than it really is. Under a counterfactual theory, causal statements are counterfactual statements (p. 7)’ (Shadish 2010). The “true” effect of a cause can never be known, it is always a plausible explanation of observed conjunctions of events under certain assumptions. Donald Rubin demonstrated that in addition to statistical assumptions requiring a sufficient number of experimental units, other assumptions are necessary to make valid causal inference based on individual or cluster randomised trials (Rubin 1974).
For Rubin (1974) there is a “fundamental problem of causal inference” under counterfactual theory as exposed in Eqs. 1–6. Each unit (or individual) u in a population U is potentially exposable to a treatment t or its absence c. For each unit u there exist a post exposure response variable Y(u) that will take the value of Y t (u) for treatment t and Y c (u) for treatment c. The effect of t on u as measured by Y and compared to c is thus given by:
It is impossible to observe the value of Y t(u) AND Y c(u) on the same unit at the same time. It is, therefore, impossible to observe the effect of t relative to c on u. So the “fundamental problem of causal inference” is to find a way to estimate the value of Y c(u). In a RCT, the strategy for estimating the value of Y c(u) consists in making use of the population U from which u is selected and of statistical theory (Holland 1986).
Equations 2–6 are taken from Holland (1986) who demonstrated that in a population, the average causal effect T over all units u in a population U is the expected value of (1), which is equivalent to:
Equation 3 simply means that the effect of T in a population U is the difference of the expected value on the outcome measure Y for units exposed to t, compared to the expected value on the outcome measure Y for units exposed to c.
In a large population U, randomization works in such a way that a variable S which can take the values t or c is randomly assigned to units u leading to
Replacing the values of E(Y t ) and E(Y c ) in Eq. 3 by those provided in Eqs. 4 and 5 leads to:
This last equation means that the effect T associated with treatment t compared to c is an average over all units u. Causal inference about t on any individual unit u 0 assumes constant effect, which means that the treatment t is not dependent on any attribute of the unit u.
Methods
CRCT and the violation of the constant effect assumption
There are two reasons why the constant effect assumption is difficult to meet in CRCT. One is statistical and concerns within cluster variation which applies to all randomized trials, while the other, we argue, relates specifically to how one conceives of effective treatment mechanism in population health interventions.
Within cluster variation
In a CRCT, it is not enough to assume a constant effect on units u (in this case, a cluster made of individuals i), we must also assume a constant effect across individuals i within units. As demonstrated by Murray (1998), when units u are clusters, the variance of the group mean expected in an RCT under the assumption of independence of errors is:
where g is the number of clusters, m is the individuals per cluster and \( \sigma_{y}^{2} \) is the within cluster variance.
This is not the same as the variance of the group mean expected in a CRCT, where the assumption of independence of errors is violated and the intraclass correlation (ICC = fraction attributable to the unit of assignment) given by:
where \( \sigma_{g:c}^{2} \) is the component of variance attributable to the unit of assignment and \( \sigma_{y}^{2} = \sigma_{c}^{2} + \sigma_{g:c}^{2} \), and the variance inflation factor (1 + (m − 1)ICCm:g:c) must be applied. As a result the conditional mean in a CRCT is:
For the constant effect assumption to hold the within-cluster variance (σ 2e ) cannot differ between units u (i.e., clusters), because it is plausible for t to be dependent on the within cluster variance. For example, individuals inclusion in a cluster can depend on their characteristics and on average individual characteristics could vary between clusters (Puffer et al. 2003).
Randomizing u’s is an attempt to make the constant effect assumption plausible. This is true of all randomized trials. However, in CRCT of population health interventions this assumption is less plausible. CRCT of population health interventions often rely on small samples of randomized clusters. Given these small samples, there are myriad poorly understood individual, group and social phenomena we can imagine that make the assumption of constant effect of treatment across groups less plausible than, for example, in an RCT of a drug targeting a specific biological mechanism. This failure and the nature of population health interventions mean that the effect of context is not removed and the mechanism of intervention is not solely attributable to the treatment.
Effective mechanisms in population health interventions
Contemporary social theory has shown that a model that makes firm distinctions between individuals and social context, viewing treatments (i.e., population health interventions) as external and independent to both individuals and social context is oversimplified and may not best represent reality. There is an increasing agreement that a recursive relationship exists between individual, their practices and social context, whereas contextual conditions enable and constrain individual practices, those practices reify and transform context.
Conceptualizing the effect of a CRCT of a population health intervention as constant within and between groups as a result of randomization is both statistically and theoretically debated. We argue that randomization rarely completely disconnects the intervention from the existing conditions and the composition of the clusters in which it is implemented. A population health intervention should be conceptualized as more than solely a “treatment” that comes from outside and can be isolated using randomization. Conceptualizing a population health intervention as constant is unlikely to yield results that are theoretically sound and that estimate an unbiased average treatment effect. In population health interventions there are myriad individual, group and social phenomena at play, which make the constant effect assumption less plausible. Our proposition is that researchers using CRCT to study population health interventions should examine the interaction between treatment and social phenomena (Poland et al. 2008).
Results
Examining interactions between treatment and social phenomena in CRCT of population health interventions
We propose that additional tests are needed to understand the effect of population health interventions using CRCT. From a counterfactual perspective, two important elements must be considered. First, the assumption of constant effect can be partially verified by dividing the population U into subpopulations U1, U2,….Ui and estimating the average causal effect for each subpopulation T1, T2,…Ti. If the average causal effect does not vary across T’s, it provides some evidence that the constant effect assumption is plausible (Holland 1986). Variability across T’s biases the average treatment effect estimate but is not uninformative. Social theory should be used a priori to construct plausible hypotheses that can explain this variation.
Second, a long series of comparable observations on the same units prior to the intervention should be considered whenever possible. This would aid in defining the counterfactual, which requires that the effects of causes are relative to a specific, yet unobservable alternative. Using time series prior to randomization allows for a specific counterfactual rather than a counterfactual world barren from social context. Close examination of the social processes by which randomly allocated population health interventions are appropriate locally to produce effects should be part of the research report.
Discussion
The CRCT has gained attention in population health intervention research. We argue that the assumptions of randomization often do not hold. If assumptions fail, interactions between treatment and social phenomena become the primary object of study in CRCT of population health interventions. This is critical when studying interventions that aim to effect the social and environmental determinants of health (Cronbach 1975). Potential interactions between social phenomena, treatments and the desired outcome are likely and should be defined a priori using social theory. This would strengthen the causal claims of CRCT and simultaneously improve theorization about the relationship between social phenomena and health.
References
Bland JM (2004) Cluster randomized trials in the medical literature: two bibliometric surveys. BMC Med Res Methodol 4(21). doi:10.1186/1471-2288-4-21
Bonell CP, Hargreaves J, Cousens S, Ross D, Hayes R, Petticrew M, Kirkwood BR (2011) Altenatives to randomisation in the evaluation of public health interventions: design challenges and solutions. J Epidemiol Community Health 65(7):582–587
Campbell MK, Elbourne DR, Altman DG (2004) CONSORT statement: extension to cluster randomised trials. Br Med J 328(7441):702–708
Cousens S, Hargreaves J, Bonell C, Armstrong B, Thomas J, Kirkwood BR, Hayes R (2011) Alternatives to randomisation in the evaluation of public-health interventions: statistical analysis and causal inference. J Epidemiol Community Health 65(7):576–581
Craig P, Dieppe P, Macintyre S, Michie S, Nazareth I, Petticrew M (2008) Developing and evaluating complex interventions: the new Medical Research Council guidance. Br Med J 337(a1655):979–983
Cronbach LJ (1975) Beyond two disciplines of scientific psychology. Am Psychol 30:116–127
Edwards SJL, Braunholtz DA, Lilford RJ, Stevens AJ (1999) Ethical issues in the design and conduct of cluster randomised controlled trials. Br Med J 318(7195):1407–1409
Eldridge SM, Ashby D, Feder GS, Rudnicka AR, Ukoumunne OC (2004) Lessons for cluster randomized trials in the twenty-first century: a systematic review of trials in primary care. Clin Trials 1:80–90
Fisher RA (1935) The design of experiments. Oliver & Boyd, Edinburgh
Hawe P, Potvin L (2009) What is population health intervention research. Can J Public Health 100(1):I8–I14
Holland PW (1986) Statistics and causal inference (with discussion). J Am Stat Assoc 81(396):945–970
Hume D (1739) A Treatise of Human Nature. Project Gutenburg
Kemm J (2006) The limitations of ‘evidence-based’ public health. J Eval Clin Pract 12(3):319–324
Macintyre S (2011) Good intentions and received wisdom are not good enough: the need for controlled trials in public health. J Epidemiol Community Health 65(7):564–567
Murray DM (1998) Design and analysis of group-randomized trials. Oxford University Press, New York
Poland B, Frohlich KL, Cargo M (2008) Context as a fundamental dimension of health promotion evaluation. In: Potvin L, McQueen DV (eds) Health promotion evaluation practices in the Americas: values and research. Springer, New York, pp 299–317
Puffer S, Torgerson DJ, Watson J (2003) Evidence for risk of bias in cluster randomised trials: review of recent trials published in three general medical journals. Br Med J 328:785
Rubin DB (1974) Estimating causal effects of treatments in randomized and non-randomized studies. J Educ Psychol 66:688–701
Shadish WR (2010) Campbell and Rubin: a primer and comparison of their approaches to causal inference in field settings. Psychol Methods 15(1):3–17
West SG, Duan N, Pequegnat W, Gaist P, Jarlais DCD, Holtgrave D, Szapocznik J, Fishbein M, Rapkin B, Clatts M, Mullen PD (2008) Alternatives to the randomized controlled trial. Am J Public Health 98(8):1359–1366
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Fuller, D., Potvin, L. Context by treatment interactions as the primary object of study in cluster randomized controlled trials of population health interventions. Int J Public Health 57, 633–636 (2012). https://doi.org/10.1007/s00038-012-0357-x
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00038-012-0357-x