Methodological Approach to Assessing the Evidence

Abstract

The approach taken in this book to guide authors in assessing the evidence for their respective topic areas was generated by the editors. It represents a combination of current recommendations for describing the state of public health evidence, assessing the quality of that evidence, including the suitability of the various studies reviewed to assess the effectiveness of their respective interventions, along with a good dose of practicality.

These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

The approach taken in this book to guide authors in assessing the evidence for their respective topic areas was generated by the editors. It represents a combination of current recommendations for describing the state of public health evidence, assessing the quality of that evidence, including the suitability of the various studies reviewed to assess the effectiveness of their respective interventions, along with a good dose of practicality.

It was beyond the scope of this book to conduct meta-analyses or full systematic reviews of the literature on the various topics. On the other hand, it was the intent of the editors and authors to provide a thorough and comprehensive review of the literature on select interventions designed to promote reproductive and perinatal health and to identify the role of the interventions with respect to reducing racial and ethnic disparities in related outcomes. Through this review, we expected to further our collective understanding of the strength of the evidence base for the common interventions examined and their associated outcomes, as well as the underlying assumptions of such interventions and their potential for decreasing relevant population health disparities.

Although the complexity of public health interventions is well recognized, the difficulty in assessing and evaluating the impact of population based interventions is often underappreciated and misunderstood. Public health’s focus on diverse populations in real life settings presents a significant set of challenges for evaluating and assessing impact. Understanding the effect of context on the design of interventions, their implementation and potential impacts, is central for an adequate and meaningful consideration of evidence for effectiveness. Unfortunately, fundamental information on the quality of interventions as well as critical details on the value and potential replication of such, are not usually included in most systematic reviews or evaluations of public health activities and programs.

Therefore, the guidance to authors and tools for reviewing the evidence that were developed by the editors for this book attempted to address some of these limitations (Appendix A). Specifically, authors were asked to focus on a particular intervention that has been assumed to have a positive influence on reproductive and perinatal outcomes, and to provide an overview of the theoretical and scientific basis of the intervention.

Authors were directed to include a spectrum of study designs including randomized control trials, observational studies, quasi-experimental designs, and expert reports, including both quantitative and qualitative methodologies and to summarize the reviewed studies in both tabular and narrative form. For each study, authors were asked not only to delineate the study type and provide a description of the intervention and key findings, but to also specify the characteristics of the population studied and to list major caveats or biases that may influence the outcomes or interpretation of the study’s findings, including identifiable contexts within which the intervention was designed and implemented. This information was to be included in a table which focused on the evidence for the effectiveness of the intervention with respect to major reproductive or perinatal outcomes selected by the chapter authors (see Table 2.1 template below).

Table 2.1 Major outcomes associated with studies of x intervention

Note that column eight asks for information about caveats and biases. In addition to the common use of the term caveat, some authors also used this column to provide explanations and modifying details to prevent misinterpretation and promote a more accurate understanding of the study being reviewed.

Furthermore, in an attempt to standardize the review of study quality across the variety of interventions and study designs, authors were initially asked to complete a quality checklist covering the following domains: reporting, external validity, internal validity (bias and confounding), and power. The checklist was an adaptation of the Methodological Quality Checklist developed by Downs and Black in 1998, to accommodate approaches used in most population based evaluations as opposed to clinical research. (Downs & Black, 1998) It became obvious that this checklist was not adequate for the qualitative studies that a number of authors were including in their reviews. Thus, an additional checklist was developed by the editors to provide consistency in the evaluation of study quality and evidence for qualitative studies. This checklist included specific questions related to the study’s research design, sampling, data collection, data analysis, results, as well as research value, and was adapted from existing work (Beck 1993; CASP 2002; Rychetnik & Frommer 2002; Miles & Huberman, 2002). The checklists are included in Appendix B.

Importantly, while each study reviewed by authors was given a “total quality score,” categorized as good, fair and poor, each study was also rated in terms of its respective “suitability.” For quantitative studies, suitability related to the study’s capacity to assess the effectiveness of the particular intervention, and was classified as greatest, moderate or least. This rating (Appendix A) was adopted from the Guide to Community Preventive Services (Briss et al. 1999.) Suitability of qualitative studies (Appendix B2) referred to the study’s capacity to generate knowledge, facilitate interpretation of quantitative studies, as well as illuminate factors relevant to intervention’s effectiveness. Studies were designated as having high, fair, or low value. This rating was adopted from previous work (Beck, 1993; Critical Appraisal Skills Program (CASP), 2002; Miles & Huberman, 2002; Rychetnik & Frommer, 2002). Authors were asked to tabulate the information from the quality checklists and suitability assessments (see Table 2.2 template below).

Table 2.2 Quality rating of studies associated with x intervention

In addition to individual studies, a number of the chapters also include reviews of meta-analyses and other systematic literature reviews. The importance of contextual factors that might influence the quality, strength, and external validity of the meta-analyses was noted by one of our book’s chapter authors, Mary Barger. Thus, a third table template developed by Dr. Barger was included for authors’ use in tabulating the findings of such inquiry and to facilitate discussion in the chapter narratives. However, not every meta-analysis discussed in the chapter narratives was included in such tables.

While there is no summary score for the totality of studies reviewed in relation to a particular intervention, authors were asked to provide a narrative summary of the evidence and the potential role of the intervention to reduce racial and ethnic disparities in reproductive and perinatal outcomes. In discussing the evidence summary, authors were specifically asked to address demonstrated effects as well as context and any variability in implementation of the intervention, along with the relevance of the evidence for public health practitioners. Finally, in the absence of any quantified effects or impact, authors were encouraged to speculate on reasons why the interventions continue to hold favor in public health practice.

Although efforts to standardize a quality review and discussion of the literature across the book chapters were agreed upon and embraced by authors, the actual process of reviewing the literature across the various topics did not always lend itself to such standardization. The range of intervention topics had their own set of exceptions in terms of the types of interventions and practice that were being considered, as well as the relevant studies and evaluations that had been carried out. There was also considerable variation in the availability of the desired information from the primary studies. This affected the extent to which some authors were able to address the issue of reducing racial and ethnic disparities for a particular intervention, as well as speculate on the relevance of the study findings for specific population groups or the feasibility of their replication. In addition to author preferences and prerogative, this variability is reflected in the type and number of tables included and their placement in the chapter, as well as in each chapter’s narrative discussion.

Even though each chapter is distinctive, the uniqueness of several chapters is worth noting in terms of their departure from the proposed chapter structure. Specifically, the chapters on childbirth practices, clinical interventions for preterm delivery, and screening and treatment of sexually transmitted infections and HIV, because of their focus on clinical guidelines and medical practice based on individual risk, posed challenges in terms of assessing and summarizing their relevance to population-based approaches to reducing disparities in reproductive outcomes. The chapter on family planning reviewed the evidence base for intervention strategies designed to increase access to family planning and safe abortion services (rather than reviewing the effectiveness of family planning services themselves, which is already well-established). Given the unique character of the evidence evaluated, results of this review were summarized in tables but not subjected to quality ratings. Another unique feature of some of the chapters in this book relates to those interventions (e.g., infertility treatments) which if made more available and accessible to women might potentially increase disparities in reproductive outcomes. Although the book editors were involved in extensive editing, each chapter ultimately reflects the perspective of the chapter author(s).

Overall, the chapters in this book highlight the dynamic relationship between politics and science and how social values are embedded in the scientific process of inquiry as well as in the application of “scientific” findings. Each chapter forces us to ask how and why it is that public health and medicine sometimes persist in pursuing practices and approaches that are in contradiction to solid evidence, or fail to universally adopt practices for which there is good evidence. The following chapters by Handler, and Aviles and Filc, highlight potential causes of these sometimes disconcerting approaches and the particular challenges of evidence-based public health.

Appendix A: Detailed Instructions to Substantive Chapter Authors

• 1) Each chapter is expected to be no more than 25–30 pages double-spaced including the tables. Authors will focus on a specific intervention that has been assumed to make a positive contribution to enhancing reproductive and perinatal health outcomes and examine the underlying theories and scientific basis of these assumptions. Chapters should address the following:

  • Definition of the intervention:Describe the selected intervention and provide a brief overview of its theoretical or scientific basis. Include a brief history and describe the current role of the intervention with respect to reducing racial/ethnic disparities in key reproductive/perinatal outcomes. If the studies to date have not focused on racial/ethnic disparities, state this.

  • Outcomes affected by the intervention:Provide a brief overview of the outcomes assumed to be affected by the intervention. Select no more than two outcomes which will be the focus of your review of the evidence. Typically, these outcomes should be those considered to be the “main” outcomes related to the intervention. However, if there has been a major review of the evidence of the intervention vis a vis a particular outcome, you might want to briefly summarize the findings of that review and provide readers with information about how to access that review. Then choose one of the “lesser” outcomes as one of your two outcomes for your review. For each outcome chosen, very briefly describe the overall prevalence and trends over time for the major ethnic/racial disparities. Keep this brief as this information is likely to appear in more than one chapter.

• 2) Review of the evidence

  • A. Overall instructions

Authors are requested to select research studies completed since 1985 or the last major review, if this is later. To ensure consistency between chapters, we ask that authors use the following search engines: MEDLINE, CINAHL (Cumulative Index to Nursing and Allied Health Literature), Popline, WHO Reproductive Health Library, Web of Science, Cochrane Library, OCLC First Search and Academic Search Elite. It is assumed that all authors will have access to the proposed search engines through their institutional affiliations. Some engines might require access through a university’s library portal. If problems arise in freely accessing any of the engines, please consult with your university librarian and advise the editors.

• 1. Study designs for consideration include:randomized controlled trials, observational studies (cohort and case-control, ecologic epidemiology studies, quasi-experimental designs including time series analyses), studies that have integrated qualitative and quantitative methods (if not already included in above), and expert reports. If a meta-analysis has been done, authors should include the results of the meta-analysis in the list of studies. Authors are requested to follow the paradigm for classifying study designs and determining the suitability of a study design for assessing effectiveness as presented in “Developing an Evidence-Based Guide to Community Preventive Services – Methods,” by Briss et al. The paradigm figure and suitability table are included below.

Given the hierarchy of study designs determining suitability for assessing intervention effectiveness, and to reduce author burden, it might be best to select studies hierarchically, with a focus on the methodologically strongest studies. However, if you find a series of weaker studies that tend to support the same conclusion, you will want to include these as well. In general, where there is an overwhelming amount of evidence, focus on the strongest ­evidence and comment on the amount of evidence available.

Because the focus of the book is on reducing racial/ethnic disparities, authors should if possible select studies conducted within racial/ethnic minority groups or those that directly compare the outcomes of an intervention for one or more racial/ethnic minority groups with the outcomes for European-Americans/majority culture. If a study directly addresses disparities, to the extent possible, please describe how “disparity” was defined and what determinants of disparity were included in the study. If none of the studies for this intervention are focused on racial/ethnic disparities per se, you should review the evidence at hand, and provide your own insights with respect to the potential effectiveness of the intervention for reducing racial/ethnic disparities.

Studies need not be limited to the U.S; however, for the most part studies are expected to be derived from the developed world. We are still considering devoting a separate chapter to the effectiveness of developing world interventions introduced in multiple locales in improving reproductive/perinatal outcomes.

• B. Specific Approach for Identified Studies: Reviewed studies are to be summarized in both tabular (see mock Tables 2.1 and 2.2 below) and narrative format.

  • 1. Table 2.1

For each study related to each selected health status outcome, delineate the study design according to the algorithm and identify the study type. Study type refers to where the findings and evidence were found, such as in a published article, technical report, abstract presentation, book or book chapter, unpublished manuscript, dissertation or thesis. Provide a description of the intervention (what was done, how, and where), denote the populations studied (ages, racial and ethnic categories included) and the sample size. Summarize key findings related to intervention effectiveness, list major caveats/biases, and note whether the study supports the effectiveness of the intervention and for which populations, if known.

• 2. Table 2.2

For each study, complete a set of questions (approximately 25–30) based on the Quality Checklist for RCTs and Observational Studies of Treatment Studies (used in the AHRQ study of perinatal depression and in turn, based on the Methodological Quality checklist developed by Downs & Black, 1998). This checklist (included in Appendix B) has several domains: reporting, external validity, internal validity (bias), internal validity (confounding), and power. Each domain generates a score; the scores are then summed for a total quality score. In the proposed checklist (slightly revised by the editors to accommodate approaches used in most population based evaluations as opposed to clinical research studies) scores greater than or equal to 20 are considered good studies, scores between 15 and 19 are considered fair, and scores of 14 and below are considered poor. Report the scores for each study in Table 2.2. For meta-analyses, leave columns 3–9 blank .

In Column 9, indicate the suitability of each study’s design for assessing intervention effectiveness. As noted above, this classification is taken from the Guide to Community Preventive Services. Table 2.2 will help authors in preparing a narrative summary of the evidence.

• 3) Summary of the evidence and role or potential role of the intervention in reducing racial/ethnic disparities in repro/perinatal outcomes.

Informed by the study designs, their suitability and quality, as well as the underlying theory and appropriateness of the intervention for the desired outcome, authors should use their judgment to describe and evaluate the overall state of the evidence reported. To the extent possible, authors should address: What are the demonstrated effects of the interventions with respect to reducing racial/ethnic disparities in reproductive/perinatal outcomes? Was there a great deal of variability in the implementation of the intervention? In the absence of any demonstrated effects, what might be reasons why these interventions continue to demand support and favor in public health practice? If positive effects of the intervention have been demonstrated but these effects have not been specific to reducing racial/ethnic disparities, consider the potential of this intervention for reducing racial/ethnic disparities. In doing so, be sure to consider whether (in your judgment), just simply “applying the evidence” to more populations will result in a reduction of racial or ethnic disparities, or whether other actions might need to be taken.

• 4) Relevance of evidence for practitioners:

Each chapter should provide commentary on whether the evidence to date has been well-translated into public health practice (e.g., how widespread is the intervention? where has it been implemented?). To the extent possible, discuss barriers, challenges, and solutions to translating the evidence into MCH public health practice. What can practitioners do to implement the evidence? What system/policy changes might be necessary to disseminate the evidence and to encourage its implementation?

Study Design Algorithm and Suitability Guidelines

Suitability of Study Design for Assessing Effectiveness in the Guide to Community Preventive Services

Table 3

Appendix B: Quality Checklists

B1. Quality Checklist for RCTs and Observational Studies

(used in the AHRQ study of perinatal depression and based on a Methodological Quality checklist developed by Downs & Black, 1998).

Reviewer’s initials ___________

First Author ___________ Journal: ___________________________________

Year published______

Table 4

Table 5

Table 6

Power

30. Did the study mention having conducted a power analysis to determine the sample size needed to detect a significant difference in effect size for one or more outcome measures?

Table 7

Instructions for select questions for the quality checklist for RCTs and observational studies

• 2. If the authors describe the formative research, theoretical basis(es) or constructs upon which the intervention was developed the question should be answered yes.

• 3. If the main outcomes are first mentioned in the Results section, the question should be answered no.

• 4. In cohort studies and trials, inclusion and/or exclusion criteria should be given. In case control studies, a case-definition and the source for controls should be given.

• 5. Interventions and placebo (where relevant) that are to be compared should be clearly described.

• 6. Give one point if some confounders are described and two only if most of these principal confounders are described.

• 7. Simple outcome data (including denominators and numerators) should be reported for all major findings so that the reader can check the major analyses and conclusions. (This question does not cover statistical tests that are considered below).

• 8. In non-normally distributed data the inter-quartile range of results should be reported. In normally distributed data the standard error, standard deviation or confidence intervals should be reported. If the distribution of the data is not described, it must be assumed that the estimates used were appropriate and the question should be answered yes.

• 9. This should be answered yes if the study demonstrates that there was a comprehensive attempt to measure adverse events/negative outcomes of the intervention.

• 10. This should be answered yes where there were no losses to follow-up or where losses to follow-up were so small that findings would be unaffected by their inclusion. This should be answered no where a study does not report the number of patients lost to follow-up.

• 11. The study must identify the source population for study participants and describe how the study participants were selected. Study participants would be representative if they comprised the entire source population, an unselected sample of consecutive participants, or a random sample. Random sampling is only feasible where a list of all members of the relevant population exists. Where a study does not report the proportion of the source population from which the study participants are derived, the question should be answered as unable to determine.

• 12. The proportion of those asked who agreed should be stated. Validation that the sample was representative would include demonstrating that the distribution of the main confounding factors was the same in the study sample and the source population.

• 13. For the question to be answered yes, the study should demonstrate that the intervention was representative of that in use in the source population. The question should be answered no if, for example, the intervention was undertaken in a clinically located site in which only subjects participating in clinical care might have participated in the intervention. For randomized studies where the subjects would have no way of knowing which intervention they received, this should be answered yes.

• 14. For randomized studies where the researchers would have no way of knowing which intervention subjects received, this should be answered yes.

• 15. For non-randomized studies, if methods were used to adjust for initial differences between groups, the answer should be yes.

• 16. For non-randomized studies, if the same methods were used for ascertainment of the outcome in both groups, the answer should be yes.

• 17. Any analyses that had not been planned at the outset of the study should be clearly indicated. If no retrospective unplanned subgroup analyses were reported, then answer yes.

• 18. Where follow-up was the same for all study subjects the answer should be yes. If different lengths of follow-up were adjusted for by, for example, survival analysis the answer should be yes. Studies where differences in follow-up are ignored should be answered no.

• 19. The statistical techniques used must be appropriate to the data. For example, nonparametric methods should be used for small sample sizes. Where little statistical analysis has been undertaken but where there is no evidence of bias, the question should be answered yes. If the distribution of the data (normal or not) is not described it must be assumed that the estimates used were appropriate and the question should be answered yes.

• 20. Where there was non-compliance with the allocated treatment or where there was contamination of one group, the question should be answered no. For studies where the effect of any misclassification was likely to bias any association to the null, the question should be answered yes.

• 21. For studies where the outcome measures are clearly described, the question should be answered yes. For studies which refer to other work or that demonstrates the outcome measures are accurate, the question should be answered as yes.

• 22. For example, subjects for all comparison groups should be selected from the same population. The question should be answered unable to determine for cohort and case control studies where there is no information concerning the source of subjects s included in the study.

• 23. For a study which does not specify the time period over which subjects were recruited, the question should be answered as unable to determine.

• 24. Studies which state that subjects were randomized should be answered as yes except where method of randomization would not ensure random allocation. For example, alternate allocation would score no because it is predictable.

• 25. If randomization occurred, and assignment was concealed from subjects but not from staff, it should be answered no.

• 26. If randomization did not occur and if methods used ensure that those in the intervention group and those in the comparison group were unaware of the study hypotheses, then the answer should be yes.

• 27. This question should be answered no for trials if: the main conclusions of the study were based on analyses of treatment rather than intention to treat; the distribution of known confounders in the different treatment groups was not described; or the distribution of known confounders differed between the treatment groups but was not taken into account in the analyses. In non-randomized studies if the effect of the main confounders was not investigated or confounding was demonstrated but no adjustment was made in the final analyses the question should be answered as no.

• 28. If the numbers of patients lost to follow-up are not reported, the question should be answered as unable to determine. If the proportion lost to follow-up was too small to affect the main findings, the question should be answered yes.

Source: Based on a modified version of the form from Downs & Black (1998)

Appendix B: Quality Checklists

B2. Guidelines to Evaluate the “Quality and Evidence” of Qualitative Studies

The proposed questions consider study design, study quality and consistency and address issues related to internal and external validity and reliability.

Table 8

Please identify the suitability of the qualitative study to generate knowledge, facilitate interpretation of relevant quantitative studies, and/or illuminate critical factors anticipated to influence the effectiveness of an intervention.

High value: The qualitative study addresses important research questions about the intervention and outcomes of interest (a minimal criterion for even considering it in the review) and the study design is appropriate for addressing those questions [implying that it is well-documented in the paper(s)] and the findings are credible and make a contribution to our understanding of the relationship between the intervention and the outcome that we otherwise would not have based on the quantitative studies alone.

Fair value: The study addresses important questions, is well designed, and adds support for other findings but does not contribute substantial new knowledge.

Low value: The study addresses important questions, but its contribution to our understanding of the issue is not apparent, due to lack of rigor in the study, inadequate documentation of the study design and/or findings.

Suitability: _______________________________

Adapted from: Beck (1993); Critical Appraisal Skills Program (CASP) (2002); Rychetnik and Frommer (2002); Miles & Huberman (2002).

Table 2.3 Meta-analysis table: topic area