Background

The random allocation of cases to treatment and control conditions is one of the most crucial features of experimental designs (Chalmers 2001; Cook and Campbell 1979). Yet, far too often, publications based on experiments fail to describe the exact process used to select and randomize cases (Boruch 1997; Sherman 2010). The lack of transparency in the transferring of eligible cases to an equal-probability assignment algorithm entails risk of selection bias and manipulation of the sequence in relation to a pre-determined assignment. When treatment providers are aware of the random assignment ahead of time, they can even potentially change their attitudes and behavior, and ultimately affect the outcomes (Day and Altman 2000). This seems to be especially the case when treatment providers are heavily invested in the success of the studied intervention. Gartin (1991), for example, showed that advance knowledge of a random assignment sequence led to major baseline differences in one of the most highly cited trials in experimental criminology, the Minneapolis Domestic Violence Experiment (Sherman and Berk 1984). Yet many experiments fail to create blind eligibility assessments due to lack of funding for rapid random assignment dispositions, especially in 24/7 operational environments.

It is now widely accepted, for example, that the barrier between researchers and treatment providers is necessary for random assignment to avoid manipulation by operational staff (Sherman 2010). It is therefore argued that the allocation sequence should be concealed from the treatment provider (see Schulz et al. 1995), and be assigned on a case-by-case basis by researchers, who should control the random allocation of cases into experimental and control groups (see Miller and Salkind 2002: 50–56). But only an on-site research manager standing between the operations and the random assignment process is currently accepted as the means to prevent such subversions effectively, even at great cost. It is not uncommon to hire dedicated research staff to deal solely with the assignment process at a research headquarters, as well as in the field. This strategy costs less when studies require many assignments to be conducted in a relatively short period of time. But in the long-term, trickle-flow random assignments (e.g., Sherman et al. 2000), there can be long pauses between eligible cases. It could take years before all experimental slots are filled, but the research team must continue to be readily available to provide the treatment provider with the random sequence of assignment. This can become quite time-consuming and expensive.

Over time, researchers have developed more alternatives to a constant staff presence at the experimental sites. If a field manager cannot be hired, the researcher can perform the random assignment herself via telephone (landline or mobile) or email communication. In these instances, the treatment provider contacts the researcher with an eligible research case, and the researcher instructs the treatment provider how to assign this case into one of the study groups. Painstaking recordkeeping of each assignment is required (see Rezmovic et al. 1981), so that biases—such as assignment integrity, selectivity, non-equivalence between the groups, and other risks to the credibility of the study—can be quickly spotted and addressed by the research team (Barnow et al. 1980; Braucht and Reichardt 1993).

Researchers can also use a ‘sealed envelopes’ technique for the random assignment process (see Schulz and Grimes 2002). In such experiments, the treatment provider is given as many sealed envelopes as there are cases to be assigned in the study, where each envelope contains a card that informs the treatment provider what the next case allocation will be. The treatment providers, whether police officers, judges or clinicians, are then expected to maintain the integrity of the process: they undertake that no cases are excluded from the assignment process and that the treatment is delivered as assigned in the envelopes.

However, it becomes immediately apparent that such alternatives do not guarantee a bias-free allocation, when the process is out of the direct control of the researcher. There is always room for human error: envelopes get lost, assignment rosters can be confused with other lists, and treatment providers may have ‘creative interpretations’ for the way the cases need to be assigned.

We have therefore sought an alternative mechanism for the random allocation of cases, especially in field experiments, that can be both cost-effective and reliable. The mechanism we have developed can enable treatment providers to quickly and safely randomly allocate eligible cases themselves. At the same time, it can allow the researcher to be reassured that the integrity of the allocation is maintained.

With the advent of widespread access to the internet, these problems are now easily preventable. Using the tool described in this article, no experiment in criminology need ever spend money on staff costs for pure random assignment. It can provide welcome relief to researchers who go to great lengths to preserve the integrity of random assignment. Staffing for eligibility assessments will remain central to compliance with experimental protocols; however, access to random assignment need not impose any costs beyond the eligibility assessments.

“The randomizer”

This note describes and presents, for public access, a free web-based “randomizer” that allows treatment providers to do the random allocation process themselves, in a secure and user-friendly process. The research team still retains full control over the assignment procedure on a case-by-case basis. That control is exercised by software rather than a human being. Unlike other online random number sequence generators,Footnote 1, the Cambridge Randomizer is also tailored to record the identifying details of each case in each study. Finally, it is designed to allow last-minute checks for eligibility of cases, so that patently ineligible cases will not be included in the random assignment sequence. By asking for baseline information and eligibility criteria queries about each case, the Randomizer may deflect many (if not all) ineligible cases for random assignment. For example, if eligibility for a prison-based treatment requires a release date more than 180 days after random assignment, forcing the entry of the release date can prevent cases from being included that were scheduled to a release date that was ineligible.

The rest of this note provides examples of how the Randomizer works in some of the experiments in which it is used, with a list of its key features. It then considers both the advantages and disadvantages of this tool.

Procedure

The procedure for using the Cambridge Randomizer is relatively straightforward and can be explained through the following example. In an experiment designed to test the effect of police-facilitated mediation following neighborhood conflict incidents, 800 cases were randomly allocated (by trickle-flow assignment) into treatment and control groups.Footnote 2 Eligibility was established based on three major criteria: first, only individual callers to the police can participate in the study (as opposed to corporate callers); second, only those who have called the police more than once during a 12-month period are deemed eligible; and third, domestic violence cases are excluded, as they normally receive specialized care by designated police units.

The treatment in this study was comprised of follow-up police visits after the initial response to a call for service for an eligible incident. While the treatment group received the special mediation service, cases allocated to the control group were treated with ordinary police intervention (in the form of follow-up visits by neighborhood police teams or criminal action, as the case may be). Since a random assignment process was used, any differences between the groups—measured in this case by victims’ satisfaction from the process and repeat calls for service during the follow up period—were assumed to be attributed to the intervention of this study and not to other rival hypotheses.

The Randomizer sits on a password-protected online portal, so that only vetted users can log in. Each police team leader is given a unique username and password, and he or she can safely communicate information with the portal. After logging in, a series of questions and eligibility criteria queries appear. These are shown as screenshots in Appendix A. The team leader is expected to provide, for each case that meets the eligibility criteria, data on the caller (e.g., name of caller, address, phone number, etc.) as well as information about the incident (e.g., date and time of the event, details of the incident, criminal codes, etc.). The officer is then asked to review the screening process: is the caller to the police an individual (not a company)? Did the caller call in the 12-month period prior to this call? Is this a case of domestic violence? If the case meets the criteria, the Randomizer produces an assignment to either a treatment consisting of police-facilitated mediation, or a treatment consisting of ordinary police intervention. The Randomizer then automatically sends the research team a report with details of the randomized case, via a secure email server, wherever he or she is around the globe. A screenshot of such a report is shown in Appendix B. This enables the researchers to have full control over the randomization process, on a case-by-case basis. Perhaps more importantly, this allows the researcher to have a clearer registry of the cases that went through the pipeline. Once assigned, the case allocation cannot be changed by either the police team or the researcher, and the case number will continue to be associated with this allocation throughout the study.

The Randomizer is also used in the first multi-centre randomized controlled trial in the last three decades within England and Wales Her Majesty’s Prison Service: The effect of an in-house victim-awareness rehabilitation program for convicted prisoners is tested in seven prisons simultaneously.Footnote 3 A total of 800 volunteering prisoners are currently being recruited, whereas any adult male prisoner convicted of any offence except sexual or domestic violence offences is eligible to participate in the treatment.

The intervention consists of a 6-week course with restorative justice elements, including face-to-face meetings with victims. Eligible participants are randomly assigned to treatment in a multi-batch assignment procedure, with approximately 10 in each batch for each treatment cohort in each prison. Reconviction rates within two years post-release will be measured and compared with prisoners who did not participate in the program.

With each assignment date—prior to the launch of a new treatment batch in each of the participating prisons—the researcher receives a list of roughly 20 eligible prisoners from each prison, who volunteer to participate in the program. The list contains, for each prisoner, an identifying case number, prison identifying code, prisoner’s name, date of release from prison, and date of birth. These data are entered into the Randomizer, as shown in Appendix C, which then generates a batch random allocation with a 1:1 ratio of treatment to control conditions. The assignment is then reported back to the prison staff officers who in turn inform the prisoners of their assignment. This procedure is repeated, until all available slots are filled within the experimental period.

The Randomizer is particularly useful in this experiment because it enables the researcher to keep track of all cases that go through the pipeline, even though she is not physically present on site. There are two instances in which this is maintained: at the moment of random assignment and then again in the periodic report generated by the Randomizer, which lists all cases that have been randomized into treatment or control conditions.

Key features

Randomization algorithm

We use a mathematical randomization algorithm to produce non-sequential random numbers. This algorithm can provide random assignment in both unrestricted and restricted randomization procedures (see Ariel and Farrington 2010). Substantially, all kinds of random assignment procedures are possible, including trickle-flow processes, and single batch as well as repeated batch assignments. We have used the Randomizer in studies that require simple random assignment (one treatment group vs. one control group), as well as block random assignment experiments and multi-site experiments.

The algorithm is presented in its simplified version in Appendix E. We invite fellow scholars to make the necessary programming adjustments as needed after they download this code, so the Cambridge Randomizer will fit their own research settings. The algorithm can easily be adapted to accommodate batch random assignments, more complex assignment matrixes (such as experimental scenarios with three study groups or more, or with more than one treatment), or multiple sites.

The randomizer can also be modified so that other than 1:1 allocation ratios can also be used. In some instances, the researcher may want to have more cases assigned to the treatment group (e.g., 3 experimental cases for every 2 control cases), or vice versa (e.g., assignment of 3 cases to the control group versus for every 2 cases assigned in the treatment group); the algorithm can also meet such needs.

Data protection and security

Randomized controlled trials in criminology usually involve sensitive subject-level data on offenders, victims, patients, or places of crime. Therefore, concerns with data protection and security management were of special importance to us. We have introduced password-protected websites that sit on a secure University mainframe. Each end-user is provided with a unique login code and password, and any potential breaches can be monitored instantaneously. This also enables the researchers to introduce multiple users, of which all or some can be treatment providers, as opposed to relying solely on the research team.

In the West Midlands Police experiment on the victim-oriented approach depicted above, the protection of sensitive data was particularly important. Identifying details of victims were disclosed to the research team: names, addresses and telephone numbers, and general information on each case were loaded to the Randomizer, so additional attention was given to safeguarding the data. Therefore, only two sergeants were given access codes to the Randomizer—one in each of the two participating local police units. This enabled us to increase the oversight over the individuals who were permitted to enter (and view) data on the Randomizer. Second, entering data into the Randomizer was only allowed within the police station of each local police unit, using a dedicated police computer which was set to have limited access to the internet. Third, once the Cambridge team receives the information on each case, all data were immediately destroyed and not saved on the mainframe, except on Cambridge secure computers.

Inclusion and exclusion queries

Randomized controlled trials are likely to be unique in their inclusion and exclusion criteria. These must be established a priori and tested for relevance in a pilot ex ante (Claxton et al. 2002). These lists of queries can be uploaded to the Randomizer, so that the treatment provider would be able to assess each case before it is randomized into the treatment and control groups. Age, gender, criminal background, and risk factors are only some types of queries that can be used. We feel that these kinds of questions do not require a sophisticated background in research methods and can therefore be easily learned and used by treatment-providers.

An illustration of this is found in operation “Turning Point”, an experiment in offender desistance policing.Footnote 4 This field experiment is designed to test whether offenders who have not been previously been convicted at court, but whom the police would otherwise charge for prosecution, can be more cost-effectively dealt with by police-led offender management subject to a condition of the certainty of a renewed prosecution in the event of reoffending or breaking an agreed contract. All eligible cases (n = 400) undergo a rapid diagnosis meeting with a police officer as they are arrested, and are required to sign an agreed contract which includes a clear understanding that a breach of named conditions of the contract or reoffending within 4 months will automatically trigger prosecution for the original offense. These contracts involve a set of tactics, including voluntary curfew, voluntary exclusion zones, and voluntary drug and alcohol testing/treatment referral. Among the various measurements of the treatment, recidivism within a 24-month follow up period will serve as a primary outcome measurement, along with associated costs to the criminal justice agencies

This experiment is particularly interesting in the context of the Randomizer because this study incorporates a longer list of inclusion and exclusion criteria than ordinarily used in police experiments. Because the study looks specifically at people arrested and who meet the criteria for charging, the prosecution office was heavily involved in establishing which cases can and cannot participate in this study. These intense negotiations between the lead researcher and the various agencies culminated into a list of over a dozen questions (see Appendix D). Cases must satisfy the following four conditions: (1) there must be sufficient evidence to satisfy the Crown Prosecution Office Code evidential test; (2) they are not considered suitable for informal resolution, caution or conditional caution; (3) their case meets the Code threshold as being in the public interest to prosecute; and (4) they have no prior court convictions for a criminal offence. Other exclusion criteria include all drink-driving offenses, those that involve death and terrorism, or official secrets offenses, etc., as detailed in Appendix D.

Cases enter the pipeline and into the Randomizer in a two-stage process. First, a custody sergeant decides that an offender has met both the evidential and public interest test for prosecution and that they have no previous court convictions. Second, a member of the Divisional Offender Management team will assess them against a set of exclusion criteria (which seek to ensure that no serious offences are included), and then decides that they are included/excluded, at which point, if included, they are randomized to treatment or control, using the Randomizer.

Because cases can be recruited at any given time, this trickle-flow experiment required five ‘accounts’, one for each police officer who was responsible for the custody suite when a potential participant was arrested. He or she then enters the case information into the Randomizer, based on the pre-established eligibility questions, and obtains an assignment for each case. As in other experiments, the Randomizer then generates a report on the case and sends it electronically to the research team.

Control of the research team over the randomization process

Once the queries are answered and the random allocation is given to the case, the Randomizer automatically sends a report to the research team via a secure email. It can be sent to one or multiple users, as the case may be. The research team therefore has full control over this process, by auditing the random selection and random assignment on an instantaneous basis.

The random assignment information can also be sent directly to a Smartphone (e.g., Blackberry© or IPhone©), provided that additional security measures are taken to guarantee the protection of the data. In this sense, the treatment provider can also log into the Randomizer from such mobile devices, especially in experiments that require instant random assignment while being out in the field.

Each case report can then be colligated onto a spreadsheet, which can in turn be used to conduct statistical analyses. In order to maintain case-by-case control, we have introduced a manual system of colligating the data rather than an automatic aggregating script. It essentially forces the researcher to review each case, before saving it in a database. However, this can easily be adapted using Macro commands in the emailing program.

Collecting data on baseline characteristics

Collecting and recording background data on each case as it goes through the pipeline can now be done by the treatment provider (provided it actually collects such data), instead of the researcher. For example, judges in domestic violence cases can instantaneously upload baseline information to the Randomizer before allocating a case to treatment or control conditions, thus saving resources on collecting data at a later stage. Similarly, in studies with police response units, officers are normally given baseline information on each case as soon as it is assigned to them. Therefore, they can upload these data onto the Randomizer.

There is another advantage for uploading information to the randomizer as a prerequisite for obtaining a random allocation sequence. Once the experiment is completed, practitioners tend to become increasingly unavailable for the purposes of the data analysis: they might forget important information, lose the opportunity to retrieve the records, or ‘move on’ to other positions or posts. In turn, requiring the treatment provider to upload the information as soon as it is available and before random assignment mitigates recall problems and other practical challenges.

Advantages

Low error rate

To date, the Randomizer has been fully applied in several separate experiments with police forces, one judge, and a court-ordered domestic violence treatment facility. Results of these experiments will be reported elsewhere. Our present concern is the error rate of the entry of eligible cases for random assignment. We can report that in only 3 of the 586 cases that have been randomly assigned have errors been identified. We believe a 0.5% error rate is well within a reasonable range. More importantly, however, we were able to spot these mistakes, attributed to human error, immediately after random assignment.

Low costs

A major impetus for using the Randomizer is its cost saving. At an age of austerity, reducing the costs of research seems vital. While designing a bespoke website for each experiment and testing the randomization algorithm is indeed a time-consuming task, such costs are tremendously lower than employing a randomization field manager or dedicating many valuable hours for the process of random allocation. Experiments conducted with police forces, for example, can last many months or even years. Remuneration, travelling costs, and equipment can substantially increase the overall price of research, all of which can be cut down by using the Randomizer.

It is difficult to quantify the monetary savings associated with the Randomizer. However, in at least two noticeable areas, we can see great reductions in costs. First, there is no need to employ a research field manager solely for the purpose of generating the random assignment sequence. For instance, with the Turning Points experiment, some cases may enter the pipeline during out-of-office work hours, including late at night over weekends. Therefore, while having a field manager is generally recommended, at least for the purposes of random assignment, custody suite officers are able to log onto the Randomizer and enter data on their own. The field manager is instantaneously notified of these assignments, but in practice he or she is not required to interject at the moment of assignment and can potentially defer the oversight and review of each assignment to office hours. In our experience, especially with the West Midlands Police experiment, once the sergeants are trained to enter information on the Randomizer, the need to intervene in the process is minimal (we discuss the issue of error rate below).

In addition, since baseline information can be requested from the treatment provider before it obtains the case allocation, substantial time is saved as well. The Randomizer provides instantaneous reports on every eligible case in the study, so less time is required to prepare the data for analysis. For example, in studies that involve official records, it is not uncommon to have to wait a long time for the data to be collected, downloaded, and stored by the host organization. Thus, with the Randomizer various tests such as baseline analyses, assessing the randomization rigor and other types of computations cannot only be conducted at interim periods but potentially commence as soon as all cases have been assigned to treatment or control conditions. Thus, the need for research assistant hours is dramatically reduced and, by implication, the costs of the research.

Second, experienced researchers can dedicate less time worrying about the random allocation and focus more on elements of the experiment or even other research projects altogether. In trickle-flow experiments, the Randomizer provides a systematic method of entering data, which does not change from one allocation to the next. Likewise, in batch random assignments, the data-entering procedure is always the same. In both scenarios, the researcher can be reassured that no changes are occurring with the way cases are being assigned, and does not have to frequently visit the experimental site (at least for this particular reason). Therefore, whereas ordinary assignment procedures by the principal investigator can potentially demand many hours of oversight, the Randomizer can reduce the time and costs associated with supervision and control.

Lastly, the Cambridge Randomizer is currently free for any research team or professional organization looking into conducting an experimentFootnote 5, so there are no costs associated with designing and implementing this tool.

Disadvantages

Less control

Can on-site research staff be completely replaced by the Randomizer? Probably not. We can imagine a wide range of circumstances in which it will remain essential to have onsite presence: from data collection, through experimental oversight, to moral support for the treatment provider staff, human contact is terribly important in ongoing research. Even then, however, the research staff will find the Randomizer a useful tool when random allocation is necessary at 3 in the morning.

Conclusion

Given the substantial costs associated with random assignments, the Randomizer is a user-friendly, safe, and cheap platform, which enables treatment providers to conduct the allocation themselves. The integrity of the random allocation procedure can be preserved, as the researcher team maintains full control over the process at the backend of the process. The fact that we have successfully utilized the Randomizer in several instances and detected a low error rate gives additional support to our reassurance that the tool is both reliable and valid. However, we would like to see more scholars use the Randomizer, perhaps with more complex random allocation scenarios, in order to better understand how well the Randomizer can be integrated into other types of experiments.