Keywords

These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

We often hear calls to increase the rigor and relevance of education research in the United States. Many would agree that rigor has increased considerably over the past decade (National Research Council 2012; Institute of Education Sciences & National Science Foundation 2013). Improving the relevance of research has been more challenging. In part, this is because the criteria for judging relevance have not been clearly defined—relevant to whom and for what?

As we write this chapter, Congress has begun hearings to reauthorize the Education Sciences Reform Act of 2002. Research relevance is a top concern (Sparks 2013). What is unclear is to whom research should be relevant. Education—and education research by extension—has many stakeholders. The Act includes a long list of would-be research users: teachers, administrators, librarians, other practitioners, parents, policymakers, voluntary organizations, professional associations, the media, the general public, and of course the researchers themselves. Tailoring education research to the needs of so many different actors is a big lift given their wide variety of information needs. We need a clear focus on key research users and the functions research serves for their work.

Each chapter in this book presents an exciting case that builds understanding of the uses of research in education decision-making in the United States. Collectively, they cover the ways research has influenced some of the key issues of this era—the Common Core State Standards, charter schools, school vouchers, teacher merit pay, the Investing in Innovation (i3) Fund, and the No Child Left Behind Act of 2001. These authors represent a new generation of scholars working, with support from the William T. Grant Foundation, to understand the uses of research in policy and practice.

Early knowledge utilization work generated broad principles. Carol Weiss (1977) argued that research more frequently plays an “enlightenment function” influencing how policymakers orient themselves to issues rather than determining their decisions. Nathan Caplan’s (1979) “two communities” theory attributed the lack of research use to the separation of research and policy by different values, reward systems, and languages. Building on these ideas, contemporary scholars more closely examine when and how research is used. They reveal the contingent nature of research use, gleaning how research informs problem formulation in some instances, decision-making in others, and more subtle learning in still others.

In this synthesis chapter, we draw out themes from this rich body of work. We begin with a discussion of the research users and their uses of research. Then we consider what these studies suggest for building a stronger infrastructure for connecting research with policy and practice. Next, we consider what the United States might learn from some other countries about developing a more research-informed education system. We close on a hopeful note: Progress is being made. And so long as we continue to learn as we go—from this body of work and others—we may just close the notorious gaps between education research, policy, and practice.

1 The Research Users

This book focuses on a key subset of research users—the decision-makers in Congress; state and local school boards; and federal, state, and local education agencies. Focusing on these policymakers and administrators makes a great deal of sense. Over the past decade, the No Child Left Behind (NCLB) Act, Race to the Top, state accountability policies, and the Common Core State Standards pressed for greater use of research evidence in decisions about curricula, turning around low-performing schools, teacher evaluation, and improving student test scores. They sought to cultivate a diverse cadre of research users. Asen and Gurke’s chapter examines local school boards’ use of research within the context of NCLB. Daly, Finnigan, Moolenaar, and Che look at district administrators’, principals’, and teachers’ definitions and uses of evidence to improve low-performing schools, while Barnes, Goertz, and Massell focus on state education agencies. Federal actors in the Office of Management and Budget, the Department of Education, and Congress are the focus of Haskins and Margolis’ study of the use of evaluation findings in program funding decisions. By focusing on this diverse group of research users, we can better understand the various functions research serves in different decision-making contexts.

The intermediaries that sit betwixt and between research and policy are another important set of research users to consider. They are not the legislators, appointees, or agency staff who hold formal policymaking roles. Nor are they part of the traditional research community consisting of higher education and policy research organizations. Some intermediaries seek to be neutral brokers, bringing research to bear on the concerns of key decision-makers. Others are more advocacy oriented, strategically drawing on research to advance their reform agendas. Scott and her colleagues examine the ways advocacy groups, think tanks, and foundations use research to promote charter schools, vouchers, teacher merit pay, and student pay for performance. McDonnell and Weatherford describe the ways the National Governors Association and the Council of Chief State School Officers mobilized support for the Common Core State Standards by promoting them as “research- and evidence-based.” Honig and Venkateswaran describe yet another type of intermediary, which focuses less on disseminating research and more on assisting administrators in applying it to their day-to-day work.

Federal research agencies and their contracted organizations are not intermediaries per se, but they play mediating roles in bringing research to policy (Barnes et al. 2014, Chap. 8). Although the Institute of Education Sciences is primarily a research funder, its charge includes disseminating research to state and local decision-makers. For example, the Institute’s What Works Clearinghouse and Regional Education Labs and the Department of Education’s Comprehensive Assistance Centers package research for state and local decision-makers.

2 The Uses of Research

With such a diverse range of research users, it is not surprising that research is used in different ways by stakeholders with varying goals, interests, and roles. The authors complicate the common conception of research users as merely rational actors who have questions, go in search of research to answer them, and then apply it to their decisions (Nutley et al. 2007). In none of their cases does research use easily boil down to a single moment or an isolated decision. It is not a simple process whereby research “facts” are passed from researchers to research users and then applied in a linear decision-making process. Instead, research use is contingent, interactive, and iterative. It involves people individually and collectively engaging with research over time, bringing their own and their organization’s goals, motivations, routines, and political contexts with them. Research also enters the policy process at various times—as problems are defined (and redefined); ideas are generated; solutions are identified; and policies are adopted, implemented, and sometimes stalled.

The depictions of research use across the chapters vary depending on the different users and their decision-making contexts; nevertheless, certain themes emerge about the functions research serves, what research use looks like, and the ways research uptake has been encouraged and enabled.

2.1 Using Research to Frame Problems and Solutions

Asen and Gurke make the cogent point that “research evidence does not speak for itself, and even if it could speak, research evidence would not speak with one voice” (Chap. 5, pp. 53–68). Instead, political actors frame research in order to substantiate their positions and to persuade others to support them. McDonnell and Weatherford show that advocates of the Common Core State Standards promoted research showing that U.S. students rank surprisingly low in international comparisons of achievement. These advocates coupled those findings with information about national standards in higher-achieving countries to strategically frame the case for common standards across states. Scott and colleagues illustrate how advocates and think tanks on both sides of the charter school debate emphasized certain studies and not others in order to bolster their positions. When they discussed the same studies, they often argued for competing interpretations of the findings (see Jeffrey Henig’s Spin Cycle, 2009). In all these cases, research truths are not simply applied to a decision to pursue one course of action over another. Instead, these cases involve policy actors operating in a political system.

2.2 Using Research as Individual and Organizational Learning

Research use is also a learning process that involves gaining and applying knowledge over time (Nutley et al. 2007). Just as classroom learning is not simply a matter of transferring information from a teacher to a student, using research is not merely about transmitting findings from research producer to user. Instead, using research—like learning in general—is a process by which individuals revise their internal representations of the world in light of new information. It is an active and dynamic process, shaped by experience and mediated socially and cognitively.

In their chapter, Honig and Venkateswaran propose a linear progression from understanding research findings to their use in driving changes in district central offices. They are particularly interested in higher-order levels of learning—or research use—that go beyond incremental shifts and result in more profound changes in administrators’ work. While learning processes are likely to be more iterative and cyclical than a linear model might suggest, there is considerable merit in exploring these processes. This is especially important at an organizational level given that most decisions are made by groups, not by individuals. Organizational learning is more than the sum of changes in individuals’ knowledge and practices. Organizations are more complex entities with varying degrees of analytic capacity and with routines and cultures that can facilitate or obstruct learning (Fazekas and Burns 2012; Finnigan et al. 2012; Honig and Venkateswaran 2014, Chap. 4; Coburn et al. 2008). By conceptualizing research use as learning over time rather than a static event, a more complete picture of the process emerges.

2.3 Converting Research into Usable Applications

Several chapters highlight efforts to improve research uptake by converting research-based knowledge into usable applications. In some instances, research findings are embedded in tools for practice such as curricula, practice guides, observation protocols, and assessments (Coburn and Stein 2010). Barnes and her colleagues describe state education agencies’ conversion of research findings into school improvement frameworks and tools to support districts and schools. Measures of teaching consist of items and scales for assessing the instructional practices that are predictive of student outcomes. Evidence-based programs incorporate prior theory and research on strategies that improve practice. Adopting these tools for teacher evaluation, professional development, and school improvement is one way to embed research in the educational system.

McDonnell and Weatherford describe the research bases for the Common Core State Standards. Although research was not the only form of evidence used, various studies and syntheses were significant in developing the standards for what students are expected to learn in different grades. A particularly influential body of research had examined children’s learning progressions in math and English Language Arts. Other research came from faculty surveys and analyses of the relationship between student performance on admissions tests and grades in lower division coursework. The findings came from diverse sources (academic journals, books, and reports) and were published by various actors (the National Research Council, federal agencies, professional associations, Achieve, ACT, and the College Board).

At their best, these tools and other applications are not only informed by prior research; they are subject to refinement based on further research and development to improve them. Moreover, studying the choices districts, schools, and teachers make in implementing the tools can generate crucial knowledge of how to enhance the tools and to align resources and supports to ensure that they are used to maximum effect.

2.4 Tying Research to Funding Decisions

Programs, practices, and tools that are based on research can be promoted by tying incentives to their adoption. Haskins and Margolis focus on the use of evaluation findings to allocate federal dollars through the Investing in Innovation (i3) Fund. In this case, research consists of evaluation evidence aimed at determining whether intervention programs produce their desired impacts. The i3 program emphasized the application of rigorous research designs to determine “what works.” Intervention programs are arranged hierarchically according to the degree of confidence people should have in them based on the study designs used to test their impacts. In the first i3 announcement, the top-tier programs were defined as having multiple randomized controlled or quasi-experimental trials, or one randomized controlled trial in multiple sites, showing positive impacts. The second tier consisted of programs where evaluation studies were said to be less robust (e.g., quasi-experimental designs), and the bottom tier consisted of programs where evaluation studies drew only on weaker designs (e.g., pre- and post-tests). The strength of the research evidence was then used to decide on funding levels. Top-tier programs were eligible for grants of up to $50 million, the second tier for $30 million, and the third tier for $5 million.

3 Building the Infrastructure to Connect Research with Policy and Practice

In addition to illustrating the various ways research is used and promoted, the chapters point the field toward ways of better connecting research with policy and practice. Drawing on these cases and concurrent efforts in the field, we suggest four ways to shore up the infrastructure for those connections: build relationships and trust, shore up capacity, create conditions for evidence integration, and develop partnerships.

3.1 Build Relationships and Trust

All the chapters implicate personal and organizational relationships as key pathways by which policymakers and practitioners acquire research and evaluate its trustworthiness. Despite the importance of relationships, the field too infrequently leverages the power of networks as a way to enhance research dissemination and use. Barnes, Goertz, and Massell suggest that state education agencies have a cadre of people and organizations that they turn to for research. These sources include regional education boards and professional membership associations who have a history of working with the agencies, are familiar with their local context and staff, and are seen as credible. Identifying these key information brokers is a good start. The next step is forging strong ties between them and researchers to ensure that high-quality research informs the advice and technical assistance provided to agencies. Engaging with these brokers has the added benefit of exposing researchers to policymakers’ information needs—knowledge that can help them improve the relevance of their work.

Daly et al. (2014) hone in on information brokers as key leverage points. They use social network analyses to map the relationships within a school district, evaluate the strength of the various social ties, and identify where the ties are particularly weak. They find, for example, that principals of low-performing schools are the most isolated—from each other, from colleagues in higher-performing schools, and from district administrators. The educators with the greatest need for assistance, ideas, and information to support reform efforts have the least access to them. These types of analyses can help district administrators and researchers visualize the social systems they are trying to impact and target resources to the people and places where research brokerage is weakest.

A focus on relationships brings trust to the foreground and indicates a need to build greater trust to support research use. A growing body of work reveals the mistrust practitioners and local policymakers have of research and research purveyors. Decision-makers judge not only the trustworthiness of research evidence but of the people presenting it (Granger et al. 2013). Finnigan, Daly, Molenaar, and Che find a pervasive distrust of research among educators in their study. For example, practitioners believe that “research” and “evidence” are often manipulated. As one educator put it:

You can find research to support anything… People are now using research to say that all the problems are the teacher, and if you can correct the teacher, all our problems go away, which is ridiculous…. The point is research can be slanted to support many different viewpoints. It doesn’t mean it’s correct. (Daly and Finnigan 2011)

Asen and Gurke show that in high-conflict, low-trust settings, decision-makers tend to distrust any rationale other parties offer for their positions, and that includes research. In one district, for example, school board members’ distrust of the administration contaminated their perceptions of information as “spoon-fed for us from the district.” Distrust of the district administrators transferred to the researchers they cited, with the assumption that both operated with a political agenda. On the flip side, Asen and Gurke argue that higher levels of trust among decision-makers can facilitate better understanding of research and more informed uses of it. Trust is malleable, and it can be built over time with deliberate effort.

3.2 Shore Up Capacity

Using research well at school, district, state, or federal levels requires adequate time, knowledge, and skills. It takes organizational leadership as well as the cultures, structures, and resources that are conducive to research use (Coburn and Turner 2012). The evidence-based initiatives launched under the Obama administration were possible because staff in the Office of Management and Budget had expertise in research design as well as the motivation and leadership skills to make things happen. The i3 initiative could draw on the standards of evidence already developed through the What Works Clearinghouse of the Institute for Education Sciences (Haskins and Margolis 2014). Policymakers also need the political savvy to persuade others (Asen and Gurke 2014; McDonnell and Weatherford 2014). As the National Research Council report on using social science comments:

Success at promoting science depends on grasping the complexity of the policy world, and on understanding the assumptions underlying divergent policy framings, expert judgments, and consensus-building techniques, as well as standard analytic methods and approaches… [There is a need to] recognize the limits of the persuasive power of scientific reasoning, the substantial institutional barriers and cultural resistance to new scientific knowledge, and the role of moral and ethical beliefs. (National Research Council 2012, p. 6)

Capacity-building is also needed on the research side of the equation. Scott and her coauthors (2014) point to the irony that intermediaries and legislative staff view university-based research as more credible than research produced by think tanks and advocacy groups. But they also see it as too expensive to produce, not timely, and too narrow to be useful. As much as policymakers and practitioners need the capacity to interpret and use research, researchers need the knowledge, skills, and time to produce more useful work and to interact fruitfully with would-be research users.

They also require institutional and professional supports to conduct research that addresses persistent problems of policy and practice. The current academic system rewards researchers for publishing in academic journals. While the peer review process helps to ensure the scientific quality of research, it does little to address the usefulness of research to policymakers and administrators. Incentive systems could reward researchers for the impact of their work in those arenas. Moreover, if future generations of scholars are to be more apt than their predecessors at conducting relevant research and communicating it clearly, they will need better training than is currently available. They will require skills for collaborating with policymakers and practitioners in designing relevant research, writing for them, and helping them understand what existing research suggests for improving their work.

3.3 Create Conditions for Productive Evidence Integration

As the preceding chapters show, bringing the best available research evidence to the table is only the beginning. It is relatively rare for research findings to provide clear-cut solutions that can simply be adopted and implemented across a range of contexts. More often, research findings suggest a direction of travel, but specific actions are negotiated locally (see also Finnigan et al. 2013; Honig and Coburn 2008; Nelson et al. 2009). In this process, research knowledge interacts with other sources of knowledge including that from local data analyses, organizational history, and practice experience (Asen et al. 2012). Conditions must be in place so that decision-makers can weigh and integrate different types of evidence and discern their implications for the specific problems at hand.

The Common Core State Standards movement suggests ways policymakers can foster productive integration of research and other types of evidence, according to McDonnell and Weatherford. Advocates wanted the Standards to be based on research but knew that research was not sufficient in and of itself to inform the development of the Standards nor their adoption and implementation by states. They developed a process that allowed for “grafting” together research and other types of evidence (T. Lindhorst, Personal communication, July 12, 2013). For example, research on learning progressions was useful for drafting the math standards for K–2 but was not available for the upper grades. In order to develop K–12 math standards, the writers pulled in other types of evidence. They relied on researchers to provide their professional judgment on what learning progressions would look like in the upper grades—judgment that was extrapolated from their knowledge of existing studies. The Standards writers also incorporated the judgment of teachers and state education agency staff—a process that strengthened the Standards and fostered broader stakeholder support for their adoption and implementation. The American Federation of Teachers and National Education Association provided feedback on the wording of the Standards, identifying areas that would be confusing to teachers and suggesting ways to clarify them. In this case, a collaborative approach across professional specialties and interests facilitated the productive integration of research with other types of evidence.

3.4 Develop Long-Term Partnerships

Partnerships between researchers and state or local education agencies are another promising way to strengthen the production and use of research. Researchers, policymakers, and practitioners work in separate spheres with differing incentives, goals, language, demands, and time frames (Caplan 1979). They have few opportunities for sustained engagement across these worlds. Researchers and policymakers might interact after studies are completed and findings are ready for distribution. Lack of significant interaction at the outset, however, obstructs researchers’ ability to adapt study designs, measurement plans, and sampling choices so that they will address decision-makers’ information needs. Even collaborative research projects are typically quite delimited, taking the form of one-off studies or circumscribed consultations.

Coburn et al. (2013) make the case for long-term partnerships that strive for sustained, joint commitments and enable partners to tackle larger questions and explore issues in greater depth. The collaboration is maintained via frequent and regular interactions. These exchanges provide researchers with a better understanding of the problems districts face, the evolution of their system goals and work, and the constraints and opportunities for making change. For practitioners, the interactions foster greater trust that researchers will share their findings in a timely and useful fashion and help them apply the research to their work.

It is an exciting time for these education partnerships, as support grows at the federal level and organizations experiment with various approaches and strategies for fostering useful work. At the federal level, the Institute for Education Sciences has issued a Request for Applications to support research-practice partnerships, and their contracts for Regional Education Labs require working through regional research alliances. The National Science Foundation is focused on partnerships in which researchers and practitioners codesign educational innovations. A crucial need is connecting the lessons learned across these partnerships. Successful partnerships—like marriages—are not made overnight. They confront the challenges of developing research agendas that meet multiple stakeholders’ needs, navigating the different time frames between research and practice, maintaining trust even when research findings can damage districts’ public images, and preserving collaboration during frequent changes in district leadership. Some partnerships are developing smart strategies to address these challenges, but mechanisms are needed to aggregate and share these strategies broadly.

4 The United States in Comparative Perspective

While various vantage points are represented in this book, the focus is on domestic education policy. In this section, we take a look at the use of research in select nations to seek insight into factors that may otherwise be overlooked or taken for granted within the U.S. context. Understanding differences across countries can also provide fresh ideas for facilitating stronger links between research, policy, and practice.

Around the world, the United States is best known for its “what works” approach to evidence-based policy and practice and is regarded as having taken a more top-down approach to research use than have many other countries (OECD/CERI 2007; Fazekas and Burns 2012; Nutley 2013). The following is a common characterization of the United States:

The clearest and most wide-sweeping attempt to mandate the use of rational learning modes is provided by the No Child Left Behind Act of 2001 in the United States… By mandating the use of rational learning modes, the producers of such knowledge gained power and prominence and overshadowed other forms of learning. (Fazekas and Burns 2012, pp. 22 and 27)

Mason (2013) also underscores the U.S. federal government’s role in setting the course of education policy over the last decade. She describes how decisions made at the federal level have profoundly affected the demand for particular types of education research and the ways this research is supplied. In Canada, by contrast, there have been no significant federal initiatives for education, and the approach within Canadian provinces has been primarily bottom-up and facilitative rather than top-down and orchestrated (Qi and Levin 2013).

The approach in the United Kingdom has tended to fall somewhere between the United States and Canada. The United Kingdom has centrally funded many initiatives to improve the supply of education research and its use in policy and practice, but these initiatives have not always been well planned or coordinated (Gough 2013). There are signs that the United Kingdom may be traveling further in the direction of the United States in its approach to research use. In 2013, the U.K. government announced four new “what works” evidence centers on local economic growth, aging, crime, and early intervention. These centers have come together with the existing National Institute for Health and Care Excellence (NICE) and the recently formed Education Endowment Foundation (EEF) to create a “what works” network. The aim of the network is to improve the use of high-quality evidence in decision-making at national and local levels.

This effort to link what works centers in different policy areas is an interesting approach. The United States has a myriad of what works initiatives in education, crime, mental health, child welfare, violence prevention, and other areas, but they are not well connected. A “network” infrastructure to coordinate what is learned across different efforts is a promising idea. In addition, the U.K. what works centers also aim to go beyond acting as clearinghouses for evidence by helping decision-makers invest in services that can deliver the best outcomes for citizens and value for money for taxpayers. They are tasked with identifying research and capability gaps and are expected to work with partners to fill these gaps. Each center is to produce and apply a common set of standards for comparing the effectiveness of interventions. The early signs are that these standards may reflect U.S.-style hierarchies of evidence, but there is recognition that matrices of evidence might be useful given the need to answer more than just “what works” questions in order to facilitate decision-making (Nutley 2013; Nutley et al. 2013). This includes assessing the quality of evidence for addressing questions about what is important for whom and who needs to be involved in the decision-making process.

The use of a top-down versus bottom-up approach to improving research use seems to be related to the extent to which knowledge (research) mobilization is viewed as primarily about the dissemination and implementation of research or about the coproduction of knowledge at a local level. In the United States and United Kingdom, what works initiatives are mainly focused on the former. Canada and Singapore, meanwhile, seem to be more attentive to the latter (Qi and Levin 2013; Teh et al. 2013). Singapore, for example, has shifted the locus of knowledge production to schools so that they are collaborating with university researchers to coproduce their research agenda, conduct the research, and learn from it (Teh et al. 2013). In this scenario, the mobilization challenge is less concerned with the vertical dissemination of research knowledge from a central hub to peripheral locations and more intent on ensuring horizontal knowledge exchange and learning between schools and districts. This Singaporean initiative seems more consistent with recent interest in research-practice partnerships in the United States. Research and school district partners jointly determine the research agenda based on local problems of practice, and the challenge is aggregating lessons learned across localities. Further understanding these various efforts around the world is useful as nations and localities seek to balance bottom-up and top-down approaches—providing the scope to focus on local needs, synthesize lessons centrally, and share learning across communities.

5 Conclusion

As we write this chapter, the U.S. policy context that put research evidence front and center in education reform is shifting. The No Child Left Behind Act is retreating into the past. Investing in Innovation funding is being debated and it is unclear how the approach of tying program funding to evaluations will evolve. The Common Core State Standards movement is also shifting; attention is now focused on maintaining political support and ensuring strong implementation. As these political and policy contexts change, opportunities emerge for greater maturity in our efforts to improve research and uses of it by various decision-makers in the policy process.

The chapters in this book suggest some promising strategies and a few cautions as we move forward. We should avoid viewing research use in overly simplified ways. Research is not the next silver bullet for education reform, and simply mandating its use will not get us to our ultimate goals of better teaching and learning. Instead, research helps us understand problems and think about potential solutions. Research must be integrated with different types of evidence and adjudicated alongside values, interests, and local circumstances. The chapters also caution us against stereotyping the approach to research use in the United States as completely top-down and based on rational learning models. But, in so far as this stereotype has some merit, they warn us about the limitations of such an approach.

If we are committed to using research to enrich problem framing, decision-making, and individual and organization learning in education, the next decade should focus on building trust, capacity, strong relationships, and the conditions for productive evidence integration.