Keywords

Introduction

Educational design research is a genre of research in which the iterative development of solutions to complex educational problems provides the setting for scientific inquiry. The solutions that result from educational design research can be educational products (e.g., a multiuser virtual world learning game), processes (e.g., a strategy for scaffolding student learning in online courses), programs (e.g., a series of workshops intended to help teachers develop more effective questioning strategies), or policies (e.g., year-round schooling). Educational design researchers attempt to solve significant real world problems while at the same time they seek to discover new knowledge that can inform the work of others facing similar problems. This chapter summarizes arguments and evidence presented by Barab and Squire (2004), Burkhardt (2009), Reeves (2011), Schoenfeld (2009), van den Akker, Gravemeijer, McKenney, and Nieveen (2006a), and others that educational design research is an innovative and exceptionally promising approach to improving the quality and impact of educational research in general, and educational communications and technology research in particular.

Educational Design Research Origins

Design research is not defined by its methods but by the goals of those who pursue it. Design research is constituted within communities of practice that have certain characteristics of innovativeness, responsiveness to evidence, connectivity to basic science, and dedication to continual improvement. Bereiter (2002) p. 321.

What has prompted scholars around the globe sharing the above-mentioned characteristics of “innovativeness, responsiveness to evidence, connectivity to basic science, and dedication to continual improvement” to come together in the pursuit of educational design research? At least two main motives can be identified. Interestingly, both perspectives have strong historical ties to educational psychology, and both perspectives are concerned with making a contribution to educational practice. The first motive is driven more by what society needs while the second has more to do with finding adequate methods to meet those needs.

First, stemming from the notion that scientific understanding should be used to solve or at least gain a better understanding of practical problems, the call for scientific inquiry to yield what Lagemann (2002) refers to as “usable knowledge” has been present for over a century. Although this focus on demonstrable impact may be ignored by some who recommend that educational researchers should emulate the methods of the so-called hard sciences (e.g., physics) that seek knowledge without expectation of practical application, the expectation for social science research to connect fundamental understanding with applied use dates back to Münsterberg (1899) and Dewey (1900), if not earlier. Both of these former American Psychological Association presidents expressed the need for a linking science, which would use empirical insights and theoretical advancements to inform problem-solving and improvement initiatives in practice. This call has been taken up gradually within the fields of education and psychology, for example in the work of Robert Glaser (1976) who laid out the elements of a psychology of instruction and called for a science of design in education. Donald Stokes (1997), an American political scientist, provided a fresh look at the goals of science and their relation to application to real world problems, in his highly acclaimed book titled, Pasteur’s Quadrant: Basic Science and Technological Innovation. Stokes promoted more “use-inspired basic research” akin to the work of the French chemist and microbiologist, Louis Pasteur. He contrasted Pasteur’s pragmatic research approach with that of the basic science goals of Danish physicist, Niels Bohr, and the applied research aims of the American inventor, Thomas A. Edison.

Second, educational researchers have been searching for adequate methods to yield the kinds of empirical insights and theoretical advancements that could be used to address real concerns in educational practice. Acknowledging the limitations of laboratory settings, the value of relinquishing control of variables in return for increased ecological validity of the findings has been gaining support over the last 30 years. In 1992, two landmark papers were published which are often credited with launching educational design research as a specific genre of scientific inquiry. Brown’s (1992) article in the Journal of the Learning Sciences discussed tensions between laboratory studies of educational innovations and challenges inherent in integrating these innovations into real world classrooms as background to describing her own design experiments. That same year, Collins (1992) published a book chapter arguing that education should be viewed as a design science akin to aeronautics, as opposed on an analytical science similar to physics, emphasizing the fact that laboratory conditions could rarely approximate conditions in real classrooms.

By the turn of the millennium, support was increasing for innovative research approaches that might yield the kind of knowledge that can be put to use for the improvement of education. Advocates for these new approaches accepted that the kinds of knowledge needed would have to be constructed in the complex “laboratories” of everyday learning environments such as classrooms or online courses. The establishment of educational design research is growing steadily. This momentum became apparent through several special issues of highly respected journals, including Educational Researcher (2003, 31(1)), Journal of the Learning Sciences (2004, 13(1)), and Educational Psychologist (2004, 39(4)). Since then, several books have been written about educational design research. Books have focused on conceptualization (van den Akker et al., 2006a) methodological considerations (Kelly, Lesh, & Baek, 2008), and the details of conducting design studies (McKenney & Reeves, 2012) across educational fields. Related volumes have appeared specifically in the domains of literacy (Reinking & Bradley, 2008) and instructional design (Richey & Klein, 2007). In addition to special issues and books about educational design research, numerous reports of educational design research initiatives have been published in research journals such as Instructional Science (cf. Xie & Sharma, 2011), the Journal of the Learning Sciences (e.g., Schwarz & Asterhan, 2011), the Journal of Research on Technology in Education (e.g., Basham, Meyer, & Perry, 2010), and Educational Technology Research and Development (e.g., Reynolds & Caperton, 2011). In addition, doctoral dissertations using educational design research have been completed at multiple institutions such as the University of California, Berkeley (e.g., Brar, 2010), Unversity of Florida (e.g., Drexler, 2010), the University of Georgia (e.g., Oh, 2011), the Pennsylvania State University (e.g., Lee, 2009), and the University of Twente (e.g., Raval, 2010).

Today we see many sectors within education that seem to embrace educational design research, including: learning sciences, instructional design, curriculum development and teacher professional development. While educational design research is not inherently tied to any specific subject area, much of the work published so far has been related to science or mathematics, perhaps because more funding has been available for research related to STEM (science, technology, engineering, and mathematics) disciplines than for other areas (Kelly et al., 2008). However, educational design research is also being increasingly used in language and literacy research (Reinking & Bradley, 2008), as well as other disciplines. A wide variety is present across educational design study literature, a development that is partly accounted for by the methodological traditions within the various educational sectors, individual researcher preferences and the resources available for specific projects. In addition, variance across the twofold motives driving educational design research plays a large role in explaining the diversity of these kinds of studies. While pursuing both goals simultaneously remains a defining feature of educational design research, one goal may feature more prominently than the other. For example, relating more to the motive of improving practice, educational design research may be conducted primarily to:

  • Solve a problem (e.g., increase the participation of women and other minorities in engineering and science careers),

  • Put knowledge to innovative use (e.g., use the affordances of smart phones to enable mobile learning), and/or

  • Increase robustness and systematic nature of design practices (e.g., establish a set of design principles for implementing inquiry-based learning in middle school science).

Or, relating more to the motive of enhancing the quality of research findings, educational design research may be conducted primarily to:

  • Generate new knowledge (e.g., develop a theory of game-based learning),

  • Generate different types of knowledge (e.g., enhance and extend knowledge related to professional development for scaffolding strategies for math teachers), and/or

  • Increase the ecological validity of research-based knowledge (e.g., increase the likelihood that educational innovations will be used to transform educational practice).

Clarifying the Nature of Educational Design Research

What Is Educational Design Research?

While studies do differ in terms of which motives are more powerful determinants in shaping the inquiry, educational design research in general distinguishes itself from other forms of inquiry by attending to both solving problems by putting knowledge to use, and through that process, generating new knowledge. As stated elsewhere (McKenney & Reeves, 2012), educational design research is a genre of research in which the iterative development of solutions (e.g., educational products, processes, programs or policies) to practical and complex educational problems, provides the setting for scientific inquiry, and yields new knowledge that can inform the work of others. Working systematically and simultaneously toward these dual goals may be considered the most defining feature of educational design research.

Educational design research is not a methodology. It uses quantitative, qualitative and—probably most often—mixed methods to answer research questions. In so doing, educational design research is held to the same standards as other scientific work when it comes to providing transparency of the process and adequate warrants for the knowledge claims it yields (cf. Shavelson, Phillips, Towne, & Feuer, 2003). In addition to the knowledge generated, the value of educational design research is measured in terms of its ability to improve educational practice (Design-Based Research Collective, 2003).

How Does Educational Design Research Compare to Other Approaches?

While both are concerned with developing new knowledge and are connected to design processes, educational design research has commonalities but also differences from the instructional design focused design and development research described by Richey and Klein (2007, and in this volume). If considered as a Venn diagram, educational design research and design and development research would overlap in projects that are concerned with actively solving problems in educational practice (e.g., design and testing of software to help plan lessons). The area that would be unique to design and development research would be those projects that are concerned with developing tools or models to support education in the long run, but that do not function as educational interventions (e.g., retrospective analysis of how instructional designers carry out their tasks). Design research projects that would not overlap with design and development research would be those not specifically concerned with advancing the field of instructional design (e.g., design and testing of a learning sequence for early literacy).

Educational design research is also different from evaluation research (Clarke, 1999), although formative and summative evaluation methods are among the main vehicles used to study and fine-tune interventions in both cases. First, problem definition and solution design are rarely featured in evaluation research. Second, a key difference is that evaluation research is primarily concerned with evaluating and possibly improving the qualities of a particular intervention. The broader scientific orientation of generating usable knowledge (e.g., in the form of models to underpin design, theories about how teachers learn, descriptions of what engages learners, etc.) is not as overtly present in evaluation research as in educational design research.

Educational design research also entails more than research-based educational design. They are both forms of scientific inquiry, and often, each values a rational approach. They both embrace systems thinking and are both shaped by iterative, data-driven processes to reach successive approximations of a desired intervention. However, research-based educational design focuses solely on intervention development, whereas design research strives explicitly to make a scientific contribution—of value to others outside the research/design setting—in addition to the intervention development. This has important implications for the entire process. Additional information on these differences is available in (McKenney & Reeves, 2012; Oh & Reeves, 2010). Similarly, action research (cf. Mills, 2002) also lacks the emphasis on finding the kind of robust public knowledge that is a hallmark of educational design research.

Distinguishing educational design research from other forms of inquiry in education is made more difficult because it has been referenced in the literature by a number of different terms such as “design-based research” (cf. Barab & Squire, 2004), “design experiments” (cf. Brown, 1992), “development research” (cf. van den Akker, 1999), “formative experiments” (cf. Reinking & Bradley, 2008), “formative research” (cf. Newman, 1990), and simply “design research” (cf. Kelly et al., 2008). There are subtle differences in how these terms are used by various researchers as delineated in McKenney and Reeves (2012). The term “educational design research” is used in this chapter and elsewhere (cf. Plomp & Nieveen, 2009; van den Akker, Gravemeijer, McKenney, & Nieveen, 2006b) because including the word “educational” in the term helps to avoid confusion with design research as used in other fields. For example, Laurel’s (2003) book simply titled Design Research concerns the field of human computer interface design and industrial engineering rather than education.

Conducting Educational Design Research

Characteristics

Characteristics of educational design research have been offered in the literature (Kelly, 2003; Reinking & Bradley, 2008; van den Akker et al., 2006a; Wang & Hannafin, 2005). Common descriptors include: pragmatic, grounded, interventionist, iterative, collaborative, adaptive and theory-oriented. Educational design research is pragmatic because it is concerned with generating usable knowledge, and usable solutions to problems in practice. It is grounded because it uses theory, empirical findings and craft wisdom to guide the work. It is interventionist because it is undertaken to make a change in a particular educational context. Educational design research is iterative because it generally evolves through multiple cycles of design, development, testing, and revision. It is collaborative because it requires the expertise of multidisciplinary partnerships, including researchers and practitioners, but also often others (e.g., subject matter specialists, software programmers or facilitators). Educational design research is adaptive because the intervention design and sometimes also the research design are often modified in accordance with emerging insights. Finally, it is theory-oriented not only because it uses theory to ground design, but also because the design and development work is undertaken to contribute to a broader scientific understanding.

Process

There is no set process for conducting the “manifold enterprise” (Bell, 2004, p. 245) of educational design research. This approach to inquiry is rich with variation in terms of models and frameworks that describe, and in a few cases, guide the process. Across that variation, some similarities can be identified:

  • Educational design research uses scientific knowledge (and to varying degrees, also other kinds of knowledge such as craft wisdom) to ground design work

  • Educational design research produces scientific knowledge (and in some cases, also craft wisdom among the participants)

  • Though the terminology and contents differ, three phases can be distinguished in educational design research: an analysis/orientation phase; a design/development phase; and an evaluation/retrospective phase; these are often revisited in the lifespan of a project

  • Educational design research strives to develop both interventions in practice and reusable knowledge

Rich Variation

Thought-provoking differences in design research are also present. Some of the differences stem from the units of analysis, scope of implementation, nature of the subject areas addressed, or from the research domains and methodological traditions in which studies originate. As mentioned earlier, the relative emphasis on each motive (solution development, new knowledge or equally on both) can also wield strong influence on the design research process. But other differences stem from the concerns of those interpreting the concept and conducting the studies.

McKenney and Reeves (2012) surveyed models for educational design research and, in addition to highlighting similarities like those mentioned above, noted unique contributions each one has to offer. The Osmotic Model, offered by Ejersbo et al. (2008), depicts the parallels of the design cycle and the research cycle. The authors point out that both cycles originate from the problem and would ideally run simultaneously, but state that this ideal is often not the case. Bannan-Ritland and Baek (2008) developed the Integrated Learning Design Framework, which depicts four main stages and across those, 14 steps, in a combined approach to research and development. Along with the process model, guiding questions for research and examples of applicable methods for each main phase are given. Reeves (2006) presented a minimalist model that highlights four main phases of design research: problem analysis; solution development; iterative refinement; and reflection to produce design principles. He compared these phases to the four phases of predictive research. In contrast to the aforementioned three models, McKenney, van den Akker, and Nieveen (2006) offered a model which is more conceptually oriented than process-oriented. This model depicts tenets guiding a research and development cycle, situated in a particular context, yielding three main outcomes: professional development of the participants; the designed intervention; and design principles.

In addition to these visual models, Gravemeijer and Cobb (2006) described important steps in the three main phases of their work: preparing for a design experiment; conducting a design experiment; and retrospective analysis. Based on a review of literature, Wang and Hannafin (2005) delineated and argued for nine principles of design-based research. Finally, Reinking and Bradley (2008) posed six questions as a guide for conducting formative experiments, relating to: pedagogical goals; classroom intervention; factors affecting the intervention; modifications to the intervention; unpredicted effects of the intervention; and changes in the instructional environment due to the intervention.

Based on their survey and analysis of existing models and frameworks for design research, McKenney and Reeves (2012) created a generic model for design research (see Fig. 11.1). Through this basic visualization, this model shows only the core elements of a flexible process that features the three main stages described earlier, taking place in interaction with practice and yielding the dual outputs of knowledge and intervention.

Fig. 11.1
figure 00111

Generic model for conducting educational design research (McKenney & Reeves, 2012)

Full size image

Scientific Outputs

Different terms have been used to describe the kinds of theoretical knowledge that are produced by educational design research (cf. Edelson, 2002; McKenney & Reeves, 2012; van Aken, 2004; van den Akker, 1999). Descriptive, substantive or declarative knowledge is generated to describe certain phenomena (e.g., what learner behaviors are triggered by certain prompts). Prescriptive or procedural knowledge is generated to help inform interventions in practice (e.g., how to facilitate learning through the strategic use of certain prompt types under certain circumstances). Some projects may develop a research agenda more attuned to one type of knowledge over another, though eventually attending to both types seems to be the case more often than not.

Different terms have been used in literature to describe the kind of integrated procedural and declarative knowledge that comes out of design research, but design principles is probably the most prevalent (cf. Kali, 2008; Kim & Hannafin, 2008; Mishra & Koehler, 2006; Quintana, Reiser, Davis, Krajcik, Fretz, Duncan et al., 2004; van den Akker, 1999). Bell, Hoadley, and Linn (2004) describe design-principles as:

…an intermediate step between scientific findings, which must be generalized and replicable, and local experiences or examples that come up in practice. Because of the need to interpret design-principles, they are not as readily falsifiable as scientific laws. The principles are generated inductively from prior examples of success and are subject to refinement over time as others try to adapt them to their own experiences. (p. 83).

On the other hand, van den Akker (1999) suggests that the knowledge encompassed in design principles can be conveyed through heuristic statements, such as, “If you want to design intervention X [for purpose/function Y in context Z]; then you are best advised to give that intervention the characteristics C1, C2, …, Cm [substantive emphasis]; and do that via procedures P1, P2, …, Pn [procedural emphasis]; because of theoretical arguments T1, T2, …, Tp; and empirical arguments E1, E2, … Eq.” (p. 9). Complementing these perspectives on design principles, Linn and Eylon (2006) also describe design patterns, which illustrate promising instructional sequences, and may be guided or fine-tuned by design principles.

Practical Outputs

In educational design research, research and development are integrated to create educational interventions that address practical problems. In early stages, this involves analysis of the problem to be addressed. Using the findings from a needs and context analysis, together with a clarified problem statement, design work commences. Depending on the scope of the project, (re-)design work can last from several weeks to several years. Especially the revisions are fed by field investigations using a range of strategies and methods to study either the intervention itself (e.g., as a type of intervention for which guidelines or design frameworks are needed); or phenomena that are engendered by the interventions (e.g., learner reactions).

Examples

Different research reports are used here (Klopfer & Squire, 2008; Oh, 2011; Thomas, Barab, & Tuzun, 2009) to illustrate the variety of educational design research conducted within the field of educational communications and technology. One study (Thomas et al., 2009) was conducted by a research team led by Sasha Barab, one of the most highly respected senior professors in the field, with substantial funding from the National Science Foundation and other sources; one study was co-led by an at-the-time early career assistant professor, Kurt Squire, with start-up funding from Microsoft and other sources; and the last was carried out by a doctoral student, Eunjung Oh, working with one other doctoral student and a practitioner with no funding beyond a graduate teaching assistantship. For each one, the problem addressed, the primary focus of the research, the intervention that was developed, the theoretical contributions, the methods used, the scope of the intervention involved as well as its practical contribution are summarized in Table 11.1.

Table 11.1 Three examples demonstrating educational design research variation
Full size table

The three examples described here illustrate how ­different types of research reports are published as sub-components of larger educational design research projects. Published in the Journal of Educational Computing Research, Thomas et al. (2009) is one of a series of journal papers in which Barab and his colleagues have described their efforts to refine a theory of transformational play while at the same time seeking to develop advanced forms of interactive learning games. This paper summarizes the results of three qualitative studies focused on the challenges and successes involved in implementing Quest Atlantis, a 3D multiplayer virtual environment (MUVE), which serves as the primary vehicle for instantiating Barab’s transformational play learning theory and for allowing it to be refined through iterative design-based research.

Published in the Educational Technology Research and Development Journal, Klopfer and Squire (2008) describe a multi-year project to enhance student learning related to environmental science through the development and refinement of learning games that are accessed with handheld devices such as PDAs and smart phones. In addition to developing an array of learning games, the project has sought to develop and refine a theoretical framework called “augmented reality educational gaming” that can be applied by other games designers. The paper provides considerable detail about the development of the learning games using a unique “design narrative” approach. This particular paper focuses on iterative design cycles based on five case studies conducted in real high school classrooms.

Oh (2011) reports the findings of a doctoral dissertation that pursued two primary goals: (1) optimizing collaborative group work in an online graduate level course focused on “E-Learning Evaluation,” and (2) developing a refined model of group work in online courses and identifying design principles for supporting online collaborative group work among adult learners The dissertation provides a comprehensive portrayal of a 2-year design research project using what Boote and Beile (2005) called the “compilation of research articles” (p. 10) format for dissertations. The dissertation includes one published article, three submitted papers, one detailed methodology chapter, and one detailed results chapter. Oh (2011) documents how mixed methods were applied across several semester-length iterations of an online course to yield multiple distinct design principles for supporting group work by adults.

Addressing Inherent Challenges

Inspired by van den Akker’s (1999) design research challenges, this section briefly touches on several important issues that often crop up in educational design research, how they may be attended to, and areas that require further consideration.

Information Richness and Efficiency: Seeking a Productive Balance

When conducting educational design research, it is necessary to address questions about appropriate tactics for increasing the information richness and efficiency of data collection procedures and instruments without being over-whelmed with data. Design researchers should not be driven by the misconception that “more is better.” This notion is aptly conveyed by Dede (2004, p. 107) who noted in reference to a design study that “everything that moved within a 15-foot radius of the phenomenon was repeatedly interviewed, videotaped, surveyed and so-forth—this elephantine effort resulted in the birth of mouse-like insights in their contribution to educational knowledge.”

Optimizing Processes: Stacking Smaller Studies Together

Other questions arise around the linkages among design, prototyping, implementation, data collection, processing, analysis, and re-design. Managing the process of communicating evaluation findings and subsequently utilizing them for improvement of interventions is difficult. Realistic timelines must be established with allowances for flexibility. Educational design research projects must inevitably be divided into smaller, more manageable chunks. These chunks and the smaller studies involved in them can function as “bricks” in a larger structure that forms both the evolving intervention and the refined knowledge. Emerging insights can be shared through shorter (e.g., article-sized) reports of smaller chunks, whereas books or other media might be more appropriate for sharing new knowledge derived from the whole of long-term efforts. Often, the interim (i.e., smaller chunk) reporting stands on its own and does not (need to) mention the larger study; also, interim reporting for an external audience can be a timely vehicle for fostering reflection among design research team members.

Measuring Impact: Powerful Examples Needed

Ultimately, educational design researchers must address questions regarding the most relevant indicators of quality, success and impact of the interventions and knowledge advances that result from their efforts. Burkhardt (2006) writes about what is needed to bring about greater acceptance of educational design research. He describes several Nobel Prize winners for design and development in other fields and concludes that educational design research candidates should be assessed on the basis of their: impact on practice; contribution to theory and/or knowledge; and improvement in either research and/or design methodology. While it is surely too early to be expecting Nobel Prizes for educational design researchers, this approach will only gain wide acceptance when it can be shown to make the much-needed gains in demonstrating the impact educational research (cf. Kaestle, 1993).

Generalizability: Toward Uptake and Use of New Knowledge

The main conceptual vehicle through which new knowledge is transferred outside of the research context, generalizability means different things to different researchers. All researchers must seek to identify promising approaches to enable uptake and use of research findings. Because educational design research takes place in natural settings where more variables are present than can be controlled for, the findings from these studies cannot yield immutable rules, easily transferred without consideration. But they can yield useful insights to inform the work of others (design work or otherwise). For example, when designs are tested in multiple settings and under varying conditions, or when design features are systematically varied under similar conditions, theory development can occur through analytic generalization. According to Yin (1989, p. 44), analytic generalization is a process through which “the investigator is striving to generalize a particular set of results to a broader theory” which can be of use to others. Alternatively, knowledge produced through design research can be shared and used through case-to-case generalization. Firestone (1993) refers to case-to-case generalization as the transfer of ideas that takes place when a person in one setting considers adopting an intervention, or its underlying propositions and frameworks in another setting. To do this, the knowledge producer is obligated to explicate how the specific instance studied compares to other instantiations of the phenomenon. In so doing, description of salient characteristics of both the intervention and the context in which it is enacted are essential. Clearly, when it comes to putting the knowledge of design research to use, the knowledge producer must portray the work well enough. This could mean, for example, adhering to Lincoln and Guba’s (1985) criteria for naturalistic inquiry: credibility, transferability, dependability and confirmability (parallel to internal validity, external validity, reliability and objectivity, respectively). At the same time, knowledge consumers are obliged to critically assess the applicability of certain ideas for their own specific contexts.

On the Horizon

Educational design researchers and arguably all educational researchers must seek to balance rigor and relevance (Reeves, 2011). To find this balance, educational design researchers might do well to learn from sister fields. For example, engineering and product design tend to embrace creativity more than most educational researchers (e.g., Laurel, 2003). Another perspective can be found in appreciative inquiry in health care (e.g., Carter et al., 2007) that emphasizes design based on opportunity, as opposed to patching gaps uncovered by reductionist problem diagnostics.

Since the landmark design research articles in 1992, a growing appreciation for educational design research in a wide variety of contexts has been evident (Anderson & Shattuck, 2012). Gradually, the design research literature is beginning to show more consideration of factors that affect implementation. Instead of tossing innovations over the metaphorical walls of classrooms and online learning environments, educational design researchers are working hand in hand with practitioners to conduct design and research in ways that make substantive change possible. The importance of collaborative approaches and on-the-ground understanding of implementation issues, which were privileged topics of research in the 1970s (cf. Fullan & Pomfret, 1977; Hall, Wallace, & Dossett, 1973; Havelock, 1971) seem relatively new—but also quite dear—to many of those currently practicing design research. Some researchers emphasize this perspective by referring to their work as design-based implementation research (e.g., Penuel, Fishman, Cheng, & Sabelli, 2011). We embrace the surge of interest in these concerns, and express our hope for a renaissance of scholarship that brings researcher and practitioner expertise together to bear on substantial educational issues (McKenney & Reeves, 2013). Educational design research is one of several genres of inquiry that can lead the way in contributing to scientific understanding in the long term through its study of meaningful implementation in the here and now.