Abstract
Using meta-analysis, we report on an investigation of the evaluator's influence in the treatment setting on criminal recidivism outcomes. Many evaluators and users of evaluation of social interventions worry that mixing of the roles of program developer and program evaluator may bias results reported in intervention studies in a positive direction. We first review the results of prior investigations of this issue across 50 prior meta-analyses, finding 12 that tested the impact of investigator influence in the treatment setting. Eleven of these reported that effect size increased positively, sometimes substantially so, when evaluators were influential or involved in the treatment setting. We followed this with a meta-analysis of 300 randomized field trials in individually focused crime reduction, also finding intervention studies in which evaluators who were greatly influential in the treatment setting report consistently and substantially larger effect sizes than other types of evaluators. We discuss two major views — the ‘cynical’ and ‘high fidelity’ theories — on why this is consistently the case, and conclude with a further agenda for research.
Article PDF
Similar content being viewed by others
Avoid common mistakes on your manuscript.
References
Alexander, M. A. (1999). Sexual offender treatment efficacy revisited. Sexual Abuse: A Journal of Research and Treatment 11(2), 101–117.
Andrews, D. A., Zinger, I., Hoge, R. D., Bonta, J., Gendreau, P. & Cullen, F. T. (1990). Does correctional treatment work? A clinically relevant and psychologically informed meta-analysis. Criminology 28(3), 369–404.
Antonowicz, D. H. & Ross R. R. (1994). Essential components of Successful rehabilitation programs for offenders. Internaltional Journal of Offender and Comparative Criminology 38(2), 97–104.
Berecochea, J. & Jaman, D. (1981). Time served in prison and parole outcome: An experimental study. Report 2. Sacramento: California Department of Corrections.
Boruch, R. F. (1997). Randomized experiments for policy and planning. Newbury Park, CA: Sage.
Boruch, R. F. & Petrosino, A. (2004). Meta-analyses, systematic reviews, and research syntheses Chapter 7. In J. Wholey, H. Hatry & K. Newcomer (Eds.), Handbook of practical program evaluation. 2nd edn. San Francisco: Jossey-Bass.
Campbell, D. T. (1969). Reforms as experiments. American Psychologist 24, 409–429.
Cleland, C. M., Pearson, F. S. & Lipton, D. S. (1996). A meta-analytic approach to the link between needs-targeted treatment and reductions in criminal offending. Paper presented at the Annual Meeting of the American Society of Criminology, Chicago, Illinois.
Cook, T. D. (2005). Why have educational evaluators chosen not to do randomized experiments? Paper presented at Random Evaluation in a Non-Routinized Environment: New Perceptions on R&D in Education and Other Social Sectors, Ecole Polytechnique Federale de Lausanne, Switzerland, April 9th.
Cox, S., Davidson, W. & Bynum, T. (1995). A meta-analytic asessment of deliquency-related outcomes of alternative education programs. Crime and Delinquency 41(2), 219–234.
Curtis, N. M., Ronan, K. R. & Borduin, C. M. (2004). Multisystemic treatment: A meta-analysis of outcome studies. Journal of Family Psychology 18(3), 411–419.
Davidson, W. S., Redner, R., Mitchell, C. M. & Amdur, R. (1990). Alternative treatments for troubled youth. New York: Plenum.
Dowden, C. & Andrews, D. A. (1999a). What works for female offenders: A meta-analytic review. Crime and Delinquency 45, 438–452.
Dowden, C. & Andrews, D. A. (1999b). What works in young offender treatment: A meta-analysis. Forum on Corrections Research 11(2), 21–24.
Dowden, C. & Andrews, D. A. (2000). Effective correctional treatment and violent reoffending: A meta-analysis. Canadian Journal of Criminology 449–467.
Dowden, C., Antonowicz, D. & Andrews, D. A. (2003). The effectiveness of relapse prevention with offenders: A meta-analysis. International Journal of Offender Therapy and Comparative Criminology 47(5), 516–528.
Drug Strategies (1999). Making the grade: A guide to school drug prevention programs. Updated and expanded. Washington, DC: Drug Strategies.
Egg, R., Pearson, F. S., Cleland, C. M. & Lipton, D. S. (2000). Evaluations of correctional treatment programs in Germany: A review and meta-analysis. Substance Use and Misuse 35, 1967–2009.
Farrington, D. & Welsh, B. (2003). Family-based prevention of offending: A meta-analysis. Australian and New Zealand Journal of Criminology 36, 127–151.
Gallagher, C. A., Wilson, D. B., Hirschfield, P., Coggeshall, M. B. & MacKenzie, D. L. (1999). A quantitative review of the effects of sex offender treatment on sexual reoffending. Corrections Management Quarterly 3, 19–29.
Garrett, C. J. (1985). Effects of residential treatment on adjudicated delinquents: A meta-analysis. Journal of Research in Crime and Delinquency 22(4), 287–308.
Gendreau, P. & Goggin, C. (1996). Principles of effective programming with offenders. Forum on Corrections Research 8(3), 38–40.
Gendreau, P., Goggin, C. & Cullen, F. (1999). The effects of prison sentences on recidivism. Ottawa, ON: Solicitor General Canada.
Gensheimer, L. K., Mayer, J. P., Gottschalk, R. & Davidson, W. S. (1986). Diverting youth from the juvenile justice system: A meta-analysis of intervention efficacy. In S. J. Apter & A. P. Goldstein (Eds.), Youth violence (39–57). New York: Pergamon Press.
Gorman, D. M. (2003). The best of practices, the worst of practices: The making of science-based primary prevention programs. Psychiatric Services 54(8), 1087–1089.
Gottschalk, R., Davidson W. S. II, Gensheimer, L. K. & Mayer, J. P. (1987). Community based interventions. In H. C. Quay (Ed.), Handbook of juvenile delinquency (266–89). New York: Wiley and Sons.
Hall, G. C. N. (1995). Sexual offender recidivism revisited: A meta-analysis of recent treatment studies. Journal of Consulting and Clinical Psychology 63(5), 802–9.
Hanson, R. K., Gordon, A., Harris, A. J. R., Marques, J. K., Murphy, W., Quinsey, V. L. & Seto, M. C. (2002). First report of the collaborative outcome data project on the effectiveness of psychological treatment for sex offenders. Sexual Abuse: Journal of Research and Treatment 14, 169–195.
Huebner, T. (2000). Theory-based evaluation: Gaining a shared understanding between school staff and evaluators. New Directions in Evaluation 87.
Izzo, R. L. & Ross, R. R. (1990). Meta-analysis of rehabilitation programs for juvenile delinquents. Criminal Justice and Behavior 17(1), 134–42.
Kaufman, P. (1986). Meta-analysis of juvenile delinquency prevention programs. Unpublished paper. Claremont, CA: Claremont Graduate School.
Latimer, J. (2001). A meta-analytic examination of youth delinquency, family and recidivism. Canadian Journal of Criminology and Criminal Justice 43(2).
Latimer, J., Dowden, C. & Muise, D. (2001). The effectiveness of restorative justice practices: A meta-analysis. Ottawa: Department of Justice.
Latimer, J., Dowden, C. & Morton, K. (2004). Treating youth in conflict with the law: A new meta-analysis. Ottawa: Research and Statistics Division, Department of Justice Canada.
Lipsey M. W. (1992). In Cook, T. C. Cooper, H. Cordray, D. S. Hartmann, H. Hedges, L. V. Light, R. L. Louis, T. A. & Mosteller F. M. (Eds.), Meta-analysis for explanation (83–127). New York: Russell Sage.
Lipsey, M. W. (1995). What do we learn from 400 research studies on the effectiveness of treatment with juvenile delinquents? In McGuire J. (Ed.), What works? Reducing reoffending. New York: Wiley.
Lipsey, M. W. (2003). The good, the bad, and the ugly: The potential confounding role of moderators in meta-analysis. Annals of the American Academy of Political and Social Science.
Lipsey, M. W. & Wilson, D. B. (1998). Effective intervention for serious juvenile offenders: A synthesis of research. In R. Loeber & D. Farrington (Eds.), Serious and violent juvenile offenders: Risk factors and successful interventions (pp. 313–345). Thousand Oaks: Sage.
Lipsey, M. & Wilson, D. (2001). Practical meta-analysis. Newbury Park, CA: Sage Publications.
Lipton D. S., Pearson, F. S., Cleland, C. M. & Yee, D. (2003). The effects of therapeutic communities and milieu therapy on recidivism: Meta-analytic findings from the correctional drug abusement treatment effectiveness (CDATE) study. In J. McGuire (Ed.), Offender rehabilitation and treatment: Effective programmes and policies to reduce reoffending (pp. 39–77). London: John Wiley.
Loesel, F. & Beelmann, A. (2003). Effects of child skills training in preventing antisocial behavior: A systematic review of randomized evaluations. Annals of the American Academy of Political and Social Science 587, 84–109.
Loesel, F. & Koferl, P. (1989). Evaluation research on correctional treatment in West Germany: A meta-analysis. In H. Wegener, F. Losel & J. Haisch (Eds.), Criminal behavior and the justice system (334–55). New York: Springer-Verlag.
Loesel, F. & Schmucker, M. (2005). The effectiveness of treatment for sexual offenders: A comprehensive meta-analysis. Journal of Experimental Criminology 1, 117–146.
Love A. J. (1991). Internal evaluation: Building organizations from within. Newbury Park, CA: Sage.
MacKenzie, D. L., Wison, D. B. & Kider, S. (2001). Effects of correctional boot camps on offending. Annals of the American Academy of Political and Social Science 578, 126–143.
Mayer, J. P., Gensheimer, L. K., Davidson, W. S. II, & Gottschalk, R. (1986). Social learning treatment within juvenile justice: A meta-analysis of impact in the natural environment. In S. J. Apter & A. P. Goldstein (Eds.), Youth violence (24–38). New York: Pergamon Press.
Pearson, F. S. & Lipton D. S. (1999). A meta-analytic review of the effectiveness of corrections-based treatments for drug abuse. The Prison Journal 79, 384–410.
Pearson, F., Lipton, D., Cleland, C. & O'Kane, J. (1995). Meta-analysis on the effectiveness of correctional treatment: Another approach and extension of the time frame to 1994 — A progress report. Presentation at the American Society of Criminology Annual Meeting, Boston, Massachusetts, November 15th.
Pearson, F. S., Lipton, D. S. & Cleland, C. M. (1996). Some preliminary findings from the CDATE Project. Presentation at the American Society of Criminology, Chicago, Illinois, November.
Petrosino, A. J. (1997). What works? Revisited again: A meta-analysis of randomized experiments in individually-focused crime reduction interventions. Ph.D. dissertation, Rutgers University. Ann Arbor, MI: University Microfilms.
Petrosino, A., Turpin-Petrosino, C. & Buehler J. (2003). Scared straight and other juvenile awareness programs for preventing juvenile deliquency: A systematic review of the randomized experimental evidence. Annals of the American Academy of Political and Social Science 589, 41–62.
Redondo, S., Sánchez-Meca, J. & Garrido, V. (2001). Treatment of offenders and recidivism: Assessment of the effectiveness of programmes applied in Europe. Psychology in Spain 5, 47–62.
Rincon, P. (2004). Secrecy penalizes cancer patients. Online at: http://news.bbc.co.uk/1/hi/sci/tech/3632882.stm (Last accessed on September 10, 2005).
Roberts, A. R. & Camasso, M. J. (1991). The effect of juvenile offender treatment programs on recidivism: A meta-analysis of 46 studies. Notre Dame Journal of Law, Ethics and Public Policy 5(2), 421–41.
Rosenthal, R. (1976). Experimenter effects in behavioral research. New York: John Wiley.
Shepherd, J. P. (2003). Explaining feast or famine in randomised field trials: Medical science and criminology compared. Evaluation Review 27, 290–315.
Sherman, L. W. (1998). Evidence-based policing. Washington, DC: Police Foundation.
Sherman, L. W. (2004). Research and policing: The infrastructure and political economy of federal funding. The Annals of the American Academy of Political and Social Science 593, 156–78.
US Bureau of Justice Assistance, Center for Program Evaluation, 2005, Online at: http://www.ojp.usdoj.gov/BJA/evaluation/glossary/glossary_m.htm (last accessed on September 10, 2005).
US National Institute of Health, (1998). NIH Policy for Data Safety Monitoring, online at http://grants.nih.gov/grants/guide/notice-files/not98-084.html (last accessed on September 10, 2005).
Visher, C. & Weisburd, D. (1997). Identifying what works: Recent trends in crime prevention strategies. Crime, Law and Social Change 28, 223–42.
Weisburd, D. L. (1996). “Preface,” In L. Greene (Ed.), Policing places with drug problems. Thousand Oaks, CA: Sage.
Weisburd, D., Sherman, L. & Petrosino, A. J. (1990). Registry of randomized experiments in criminal sanctions, 1950–1983. Los Altos, CA: Sociometrics Corporation.
Weisburd, D. L., Petrosino, A. J. & Mason, G. (1993). Design sensitivity in criminal justice experiments: Reassessing the relationship between sample size and statistical power. Crime & justice: An annual review of research (vol. 17). Chicago: University of Chicago Press.
Wells-Parker, E., Bangert-Drowns, R., McMillen, R. & Williams, M. (1995). Final results from a meta-analysis of remedial interventions with drink/drive offenders. Addiction 90, 907–926.
Wilson, D. B. (2001). Effect size determination program. Online at http://mason.gmu.edu/∼dwilsonb/downloads/es_calculator.zip (last accessed Nov. 16, 2005).
Wilson, S. J. & Lipsey, M. W. (2000). Wilderness challenge programs for delinquent youth: A meta-analysis of outcome evaluations. Evaluation and Program Planning 23, 1–12.
Wilson, D. B., Gallagher, C. A. & MacKenzie, D. L. (2000). A meta-analysis of corrections-based education, vocation, and work programs for adult offenders. Journal of Research in Crime and Delinquency 37, 347–368.
Whitehead, J. T. & Lab, S. P. (1989). A meta-analysis of juvenile correctional treatment. Journal of Research in Crime and Delinquency 26, 276–295.
Woolfenden, S. R., Williams, K. & Peat, J. (2003). Family and parenting interventions in children and adolescents with conduct disorder and delinquency aged 10–17. Cochrane Review. Oxford, UK: Update Software.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Petrosino, A., Soydan, H. The impact of program developers as evaluators on criminal recidivism: Results from meta-analyses of experimental and quasi-experimental research. J Exp Criminol 1, 435–450 (2005). https://doi.org/10.1007/s11292-005-3540-8
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11292-005-3540-8