Keywords

1 Definitions and Scope

DBR is one of a cluster of terms used to describe various intersections between design and research, especially in the realm of academic research in either education or human–computer interaction. In this section, we attempt to define what we mean by design-based research and contrast it with other definitions.

DBR methods were originally defined (Design-Based Research Collective [DBRC] 2003; Hoadley 2002), like the earlier concept of design experiments (Brown 1992; Collins 1990, 1992), as a research method or related methodology which used a blended form of design activities and research activities to produce design-relevant, empirically supported knowledge. Designed interventions in DBR are tested iteratively in a context of use, and the iterations become settings to collect data that support or refute inferences about underlying theoretical claims. At the same time, the iterations are used for increasing the fit between the theory, the design, and the enactment or implementation so as to best test the theoretical conjectures. Unlike earlier definitions associated with design experiments (notably Brown’s 1992), DBR methods were claimed to be not merely related to hypothesis generation, but a scientific enterprise in their own right. This approach stemmed from a very practical problem described earlier by Simon (1969) in his seminal book—The Sciences of the Artificial—namely, that.

… the genuine problem is to show how empirical propositions can be made at all about systems that, given different circumstances, might be quite other than they are (p. XI).

In the case of DBR as a science of the artificial, this genuine problem concerns making empirical propositions regarding designs of learning environments that are studied while they are being created.

The notion of DBR as a research methodology contrasts with other points of connection between design and research. Specifically, Instructional design, User-centered design, and other similar terms from the fields that attempt to create educational interventions—materials, or technologies might be lumped under the terminology of research-based design (RBD) methods. In such methods, the tools of empirical research are subservient to the goal of ultimately creating a useful designed product or intervention. The main difference, thus, is that DBR uses design processes to produce research knowledge, where RBD uses research techniques to produce designs. Evaluation research of designs is similar to DBR in that at the end there is both a design and research output, but the difference is that these activities in evaluation research are by necessity separated from each other. The intervention or tool is complete at the moment in which evaluation is taking place, and the data used to inform the design are typically distinct from the data used to evaluate that design.

The terms design research or design studies are used variously in different communities, ranging from the journal Design Studies, which focuses on studies of designers and design processes, to a notion of design research which labels the learning process a designer must go through in order to connect a context to a designed solution (e.g., Laurel 2013). Another more recent term is Design-Based Implementation Research (Fishman et al. 2013; Kali et al. 2018), which can be characterized as a subset of DBR with three main distinctive characteristics: joint ownership of the research agenda by practitioners and designers/researchers; an inherent focus on designs and research questions related to the issues of scaling interventions systemically (e.g., across a large school system or a geographic region); and a linkage between micro-level design (design of a particular intervention, for instance at a classroom level) and macro-level systems change (e.g., design of an institution-wide framework for adoption) (Law et al. 2016).

We look more specifically at the issue of DBR methods in the particular sense of a research methodology which yokes the design process and research process to produce knowledge outcomes (and not just useful, validated designs). Although many have suggested more generalized definitions of DBR since its introduction (e.g., McKenney and Reeves 2012/2018), we rely on the earlier characterization from the Design-Based Research Collective (DBRC 2003) as it encapsulates more directly what critics find challenging about DBR. In this definition, DBR has five characteristics: (a) overlap between the design and research process (both temporally and intellectually); (b) iterative cycles of design, enactment in context, analysis, and redesign; (c) a goal of theory development that is relevant to practice; (d) a commitment to understanding the designs in authentic settings (as opposed to more reductionist approaches); and (e) a recognition that the design and the enactment are intertwined in producing the outcomes (i.e., that outcomes are the result of both the use of designed artifacts and the way they are used).

2 History and Development: DBR in CSCL

We believe the connection between DBR methods and the CSCL research community is not a coincidence, but rather a natural byproduct of the ways in which almost all CSCL research is contingent on shifting, culturally and technologically grounded social contexts for learning, and on theories that help encompass that social context. Various authors (e.g., Kaptelinin and Cole 2002; Koschmann 1999; Paavola et al. 2004; Stahl et al. 2006) have explored how socially contextualized theories intersect with a technology-enhanced action orientation of research. Such a design orientation for research is notable, for example, in Kaptelinin and Cole’s (2002) classic use of activity theory for analyzing the design of a collaborative learning environment. The design is conceptualized as a perturbation of activity structures, placing the scope less on a particular tool and more on how the tool, together with the designed collaboration processes support learning. As Stahl et al. (2006) point out, the intersubjective nature of learning and the challenges of intersubjectivity among researchers and analysts of human behavior influence the relationship between design and research in CSCL:

CSCL research has both analytic and design components. … To design for improved meaning making, however, requires some means of rigorously studying praxis. In this way, the relationship between analysis and design is a symbiotic one—design must be informed by analysis, but analysis also depends on design in its orientation to the analytic object. (Stahl et al. 2006, p. 11).

Challenges such as these have led to discussions and debates about DBR within the context of CSCL research and development.

In the early 2000s, a blossoming of scholarship on DBR methods yielded a number of special issues, including those published in Educational Researcher (2003), Journal of the Learning Sciences (2004), Educational Psychologist (2004), Educational Technology (2005). The articles included in these special issues helped legitimize the approach, but also proliferated alternative definitions of what constitutes DBR and how it would fit with other related concepts such as “design research,” and engaged with critiques of the method and its underlying epistemologies. Prominent critiques included a failure to contend with lack of appropriate experimental control for causal inferences (Desforges 2000), difficulty conveying in adequate detail the relevant aspects of the design and the data (Reeves 2005), being susceptible to overinterpreting and/or cherry-picking interpretations given the breadth of data collected under evolving, rather than fixed, protocols (Dede 2004, JLS), and a lack of a clear argumentative grammar (Kelly 2004).

3 State of the Art: Argumentative Grammars and Tensions Within DBR Epistemology and Ontology

As described above, one way to understand DBR is its dual goal in advancing both learning theory—explanatory evidence-based arguments on how people learn in various instructional contexts (especially in those involving CSCL), and learning design (the features and principles for environments that support such learning). When it comes to theory, we might start with a positivistic psychological or cognitive framing of what a theory is, but we can also extend the notion of learning theories much more broadly with interpretivistic sociocultural conceptions, situative understandings, humanistic theories, etc. On the other hand, design knowledge might encompass specific designed artifacts or interventions, ideas about how to instantiate particular goals through human agency, or ideas about what interventions might be possible. Unlike in traditional experimental research, design in DBR is not solely a means for the purpose of conducting research—it is a goal by and of itself, juxtaposed to its twin goal of advancing theory. Yet, there are important differences in what makes good or useful outcomes in these two arenas—theory and design. These differences create an inherent tension within DBR, which affects how we judge the worth of the processes of knowing—DBR’s epistemology, as well as the nature and types of knowledge produced—what we might term DBR’s knowledge ontology. Following Chi’s notion of ontological commitments (Chi 1992; Slotta 2011), it is worth saying that the types of knowledge produced in DBR fall into different sorts of categories which are determined in part by the ontological commitments we hold as designers and researchers. An ontology of DBR in CSCL should include different categories of knowledge, ranging from design patterns to presumed universal laws of psychology. In other words, the tension between theory and design in DBR in CSCL affects how we know things, and what kinds of knowledge are produced. In this section, we describe debates within the learning sciences and CSCL communities concerning the value of DBR, how it can best be conducted and communicated, and what its outcomes should look like. We then illustrate how these debates are in fact a result of the theory–design inherent tension within DBR.

3.1 DBRs Dual Epistemic Game

People follow rules in deciding what claims are valid in different research contexts. One term for this is epistemic games. In introducing this term, Perkins (1997) referred to patterns of inquiry, such as goals, moves, and rules, which he described as:

… woven together in a course of inquiry... [and are often] played competitively, as in the adversarial system of justice of scientific debates. (p. 52)

Another term for describing the ways in which researchers’ progress toward knowledge and understanding in a field is argumentative grammar. In the world of methodologies, the argumentative grammar determines the rules for making an argument within the coherent world of epistemology or method. Thus, epistemic games can be thought of as the language of claims and debates in a field, and argumentative grammar as the underlying structure of that language.

Within DBR, a criticism has been that it is not clear on its argumentative grammar (Kelly 2004):

What, therefore, is the logos of design studies in education? What is the grammar that cuts across the series of studies as they occur in different fields? Where is the “separable” structure that justifies collecting certain data and not other data and under what conditions? What guides the reasoning with these data to make a plausible argument? Until we can be clear about their argumentative grammar, design study methods lack a basis for warrant for their claims. (p. 119)

Such criticism objected the pluralism that DBR researchers such as Bell (2004), and later on McKenney and Reeves (2012/2018) or Bakker (2018) ascribed to DBR. Bell, for instance, already in 2004 maintained that:

At a time when many efforts that are reviewing the status of educational research seem to be operating under the working assumption that our theoretical and methodological complexity should be reduced, I argue that rigor and utility can be actively pursued through pluralism—a coordination of different theoretical views on learning and education. (Bell 2004, p. 251)

We claim that this ambiguity within DBR methodologies (even if we refer to methodologies in plural and not a single methodology) results not only from the broad range of theoretical views studied using DBR, but rather—is rooted in the epistemological tension inherently embedded in the dual goal of DBR. Consequently, the lack of a clear argumentative grammar in DBR is mainly related to lack of clear linkage between the two languages we speak (advancing theory and advancing design). That is, we (design researchers) typically play two epistemic games, and oftentimes—are not clear enough about how we switch between them.

To illustrate what we mean by a dual epistemic game, we turn to philosophical notions of design. In their seminal book, “The design way,” Nelson and Stolterman (2012) characterize the unique mode of inquiry that designers follow, by contrasting it with the one followed by scientists. While scientists, in general, strive to reason from the concreteness and complexity of the actual world, to the abstractness and simplicity of principles and laws (yellow arrow, going up the curve in Fig. 1Footnote 1), designers, they say, strive to do the opposite. That is, designers use such abstractions to create specific designs in the actual world (e.g., a specific product or policy) by making design judgments (blue arrow, going down the curve in Fig. 1). Therefore, science and design constitute quite different traditions of inquiry that encompass contrasting rules within their epistemic games.

Fig. 1
A graph of the true versus the actual. It plots a decreasing curve with abstracting generalized explanations, D B R, and particularization.

Contrasting “science” and “design” modes of inquiry (adapted from Nelson and Stolterman 2012) and the dual intertwining epistemic game we play in DBR, iterating between abstraction and particularization

Full size image

We claim though, that in DBR we play and intertwine both these traditions, iterating between abstraction and particularization (green arrow in Fig. 1). A DBR study typically begins by identifying a gap in educational theory that we (DBR researchers) aim to explore by designing and enacting an intervention within the so-called real world, what Bhaskar (1975) would call the actual world. To develop an initial design, we take into account generalized abstractions (e.g., theories, design principles), and embody them into a specific design (going down the curve). Then, we collect (messy) data regarding how learners interact with our designs in the actual world, and analyze this data (using the existing theoretical lenses, but open to refining them) to come up with new generalized conjectures about learning (going up the curve) and use them to refine the designs (down the curve), to test these conjectures (up again), and so on with as many iterations as needed to contribute to both theory and practice.

It turns out that within this abstraction–particularization tango, we constantly switch epistemic languages, and therefore it is clear why DBR is missing one agreed upon argumentative grammar.

In doing so, DBR is sometimes used within a positivistic framing to make strong, generalizable truth claims about a presumably objectively knowable world. But DBR is also sometimes used within an interpretivist framing to explore aspects of the human experience that are presumed to be knowable only through individual interpretation and which are inherently not generalizable. DBR researchers may violate some of the core tenets of either of these core epistemologies, much to the consternation of researchers hoping to fit it in with their existing epistemological commitments. Such distress is expressed in the following excerpt from an anonymous reviewer in his/her comments regarding a manuscript describing a DBR project:

[the manuscript entails] an awkward combination of qualitative and quantitative research perspectives. Symptomatic of this is the fact that you use both the word “causal” and the word “holistic” in your title! Show us where you stand. (Anonymous reviewer)

Thus, DBR sits in tension both with positivism and interpretivism, both “quantitative” and “qualitative” research, and better adheres to mixed methods (Bell 2004). Knowledge claims rely heavily on the designer’s stance and interpretation of not only the data but also their interpretation of the design context, circumstances, and goals (Tabak 2004). Therefore, such claims are presumed to be somewhat generalizable, but—using diSessa’s (1991) terminology—based on local (rather than global) sciences. Cobb and Gravemeijer (2008) refer to such generalizations as domain-specific instructional theories.

3.2 Why We Have Multiple Argumentative Grammars, and What Is Still Missing

Recently, Bakker (2018) suggested to address Kelly’s criticism by noting that we do not necessarily need one argumentative grammar, but rather, multiple grammars. This view is in line with the pluralistic view of DBR methodology described earlier (e.g., Bell 2004; McKenney and Reeves 2012/2018). In the chapter “Argumentative grammars used in design research,” Bakker lays out various solutions that have been developed in the past two decades to serve as underlying “rules” for making DBR arguments. He presents these “rules” using Toulmin (1958) general argumentation scheme which clearly distinguishes claims, evidence, and reasoning to illustrate the external structural logic of these grammars. Within these grammars he includes: (a) Proof of principle that certain learning outcomes are possible (e.g., O’Neill 2012), which requires advance setting of criteria for success and failure; (b) small changes per iteration, which enable experimental approaches for comparing learning outcomes between iterations (e.g., Kali et al. 2009); (c) building on the experience of the DBR community, as in the design principles database (Kali 2006, 2008) in which DBR researchers can use, refine, and share their own design principles, making it possible to abstract generalized explanations based on refinement of insights across studies; (d) answering the “how” question, which illustrates the logic of experimental designs that aim to develop insights regarding how a particular educational approach can support learners achieve certain educational goals (e.g., Smit et al. 2013); and (e) conjecture mapping (Sandoval 2014), which distinguishes between high-level conjectures that are derived from theory and embodied into the design of learning environments, design conjectures that define the relation between features in the environments (e.g., tools, activity structures) and the resulting mediating processes, and theoretical conjectures that focus on the learning outcomes that result from these processes (Fig. 2).

Fig. 2
An illustration depicts the D B R method with high-level conjecture, embodiment, mediating process, and intervention outcomes.

A generalized conjecture map (adapted from Sandoval 2014)

Full size image

We focus specifically on Sandoval’s (2014) conjecture mapping due to its wide acceptance and use among DBR researchers, but also, because we contend that it nicely illustrates the dual epistemological game, and the intertwining between the abstracting–generalized explanations and the particularization modes of inquiry (Fig. 1).

First, the embodiment of a high-level theoretical conjecture into design features within a learning environment clearly demonstrates a “down the curve” process of particularization. Then, characterizing the learning that occurs during enactment in terms of mediating processes represents beginning stages (typically with interpretive methods) of an “up the curve” process in seek for abstracted generalized explanations (e.g., patterns of use), which are then further substantiated in terms of theoretical conjectures (how the mediating processes support learning outcomes). But as noted by Sandoval (2014), such mapping represents only part of a trajectory of studies (multiple iterations), that together enable the development of generalized explanations in DBR. That is, conjecture maps are revised from iteration to iteration, and additional back-and-forth movements within the abstraction–particularization curve are typically conducted.

Therefore, we believe that although multiple argumentative grammars, as suggested by Bakker (2018) enable DBR researchers the flexibility in making decisions about what counts as DBR, it does not solve the dual-language issue inherent to DBR, which requires better calibration between the two epistemic games involved. Moreover, as we explain in the next section, we view the chasm between the two epistemic games as percolating from DBR epistemology into DBR knowledge ontology. Due to this chasm, researchers debate not only rigorousness of DBR methods but also the value of DBR outcomes.

3.3 How the Dual Epistemic Game Percolates into Design Ontology

The debate regarding the value of DBR outcomes was most notably expressed in a series of three “reports and reflection” articles in the Journal of the Learning Sciences. Bereiter (2014) argued that DBR researchers fail to produce outcomes that embed “know why” knowledge within “know-how” artifacts. He labeled such blended knowledge—having the potential to be useful for both researchers and practitioners in generating innovation—principled practical knowledge(PPK). Janssen et al. (2015), however, in their response article—Practicality studies: how to move from what works in principle to what works in practice—maintained that PPK, as specified by Bereiter, is too abstract to support teachers in implementing innovations developed in DBR research. They contended that the DBR community underestimates the magnitude of usability issues, and suggested an additional type of knowledge—fast and frugal heuristics—to complement PPK. This debate continued with Bereiter’s (2015) response cautioning DBR researchers from being too specific regarding how to implement the outcomes of their studies. Such specificity, he claims, may communicate a message of disrespect to teacher professionalism, and hinder teachers from venturing successfully beyond conventional practices.

This ongoing debate relates back to the dual epistemic game exemplified in Fig. 1. Is DBR trying to make truth claims within a coherent (interpretive, positivist, or other) epistemology? Sometimes, DBR produces knowledge that is contingent on context, but more actionable. In other words, sometimes DBR is more concerned with producing usable knowledge, than with producing truth claims. This tension in DBR has been referred to in various terminologies such as actionable knowledge versus knowledgeable action (Markauskaite and Goodyear 2017); generalization versus generativity (Bakker 2018); and analytical versus creative mindsets (McKenney and Reeves 2012/2018). Interestingly, all of the researchers who pointed to this tension note a detrimental bias in which the research community typically prefers the “scientific” over the “design” mode of inquiry, as indicated in standards of publication and the like. That is, actionable knowledge tends to be valued more than knowledgeable action, generalization more than generativity, and analyticality more than creativity, in conducting DBR studies.

4 The Future: Capitalizing on the Dual Epistemic Game in DBR to Spur Creativity and Innovation in Rigorous DBR Research

Up to this point we have characterized DBR as being pluralistic, accommodating of a wide range of methodologies, and have shown how this pluralism has drawn criticism, and interpreted as a lack in argumentative grammar (e.g., Kelly 2004). We also illustrated how DBR researchers have addressed such criticism with various argumentative grammars, as well as with the notion that having multiple grammars is pertinent (Bakker 2018).

However, we believe that DBR researchers need to acknowledge the duality in the epistemic game we play and that this duality is not a fair target for the criticism of lack of an argumentative grammar. Rather, we suggest that DBR be examined on the basis of the coherence of arguments across the dominant argumentative grammars as researchers intertwine the abstraction–particularization curve (Fig. 1). The next step for DBR is not only to acknowledge but also to capitalize on this epistemological and ontological duality while considering the systemic validity of the activity. That is, it is less important that the epistemic games are narrowly played, and more important that the outcomes of the research matter and make sense both in the knowledge realm and to the people involved, leading to actions and decisions that support a consequential validity of the research. To do so, in this section we draw on two frameworks: (a) methodological alignment (Hoadley 2004) and (b) design researchers’ transformative learning (DRTL, Kali 2016).

4.1 Methodological Alignment as Means for Calibrating the Theoretical and Practical Aspects of DBR

The notion of methodological alignment is essential to our understanding of rigor and research validity. It involves the ways in which researchers connect theories to hypotheses, hypotheses to interventions, interventions to data gathering, and data gathering to interpretation and application. Fifteen years ago, Hoadley (2004) argued that we tend to overemphasize certain types of validity at the expense of others. Specifically, he argued that measurement validity is often regarded as the sole or at least main indicator of rigor. That is, the efforts of ensuring that the means of data collection accurately align with what is being measured predominates our view of well-designed research. DBR, he claimed—with its unique research design—affords three other types of validity: (a) treatment validity—ensuring that the treatments we create accurately align with the theories we are examining, (b) systemic validity—that the inferences we make to prove our claims are aligned with these theories, and (c) consequential validity—that these theories are applicable to decisions based on the research.

We view these three types of validity measures for reaching methodological alignment as principles for calibrating methodological moves in DBR, aiming at both theoretical and practical advancements. That is, the multiple iterations in DBR—each involving back-and-forth movements within the abstraction-particularization curve, between scientific and design modes of inquiry (Fig. 1)—afford DBR researchers with multiple opportunities to reach higher degrees of treatment, systemic and consequential validity. In this way, methodological alignment principles can serve DBR in achieving a unique type of rigor, which traditional research methods in education may fail to afford.

At the same time, these calibration principles can address the ontological debate and assist in producing PPK. Traditional education research believes that the knowledge (or what Nelson and Stolterman (2012) refer to as “the true”) lives in the abstracted generalized explanations that are typically expressed in journal articles. Traditional design believes the knowledge lives in the designed artifacts—curricula, technology-enhanced learning environments, etc. (what designers add to “the actual”—according to Nelson and Stolterman (2012)). In DBR, because we have this different ontological status of knowledge, it lives in neither and both. If we follow the CSCL way of seeing knowledge as contextualized, distributed, culturally embedded, and constantly negotiated by real human beings using information communication technologies, we need to understand that PPK doesn’t live in a research article or a designed learning environment alone. It lives in humans who must negotiate the ontological tensions we have outlined, and this demands personal transformation.

4.2 Transforming Ourselves as a Prerequisite for Transforming Others

In the “Design Researchers’ Transformative Learning” (DRTL) framework, Kali (2016) claimed that DBR provides an especially fertile ground for transformative learning among those who conduct it. DRTL builds on Mezirow’s (1996) transformative learning theory, in which such learning is characterized as “the process of using a prior interpretation to construe a new or revised interpretation of the meaning of one’s experience in order to guide future action” (p. 162). That is, transformative learning results not so much in a learners’ recognition of new facts about matters under study. Rather, these are personal “aha moments” that bring learners to reorganize the ways of looking at, thinking about, and acting on those matters. Kali (2016) claimed that in DBR, such personal “aha moments” often expose researchers to flaws in their earlier conceptualization (which is one of the reasons DBR researchers tend to keep these parts of their research behind the scenes). But more importantly, the transformative learning enables design researchers not only to develop new conceptualizations for how to continue their research but also for how they position themselves as actors within the situation they are exploring. In this personal positioning aspect, DRTL differs from the three aspects of learning described by Edelson (2002) in his “what we learn when we engage in design” article, which are domain theories, design frameworks, and design methodologies, which do not include the more personally experienced notion of design knowledge.

We claim that what makes DBR such a potentially fertile ground for DRTL is the methodological alignment it affords, and the careful, iterative calibration between the pursuit of advancing theory and design both in terms of DBR epistemology and ontology. Figure 3 illustrates DRTL as part of the model we suggest for calibrating DBR epistemologies and ontologies. We claim that DRTL that results from following the principles of methodological alignment described above leads to what McKenney and Reeves (2012, 2018) describe as blending of analytical and creative mindsets, which is crucial in developing CSCL innovation. It is worth noting, as we exemplify in the case study below, that such iterative calibration within both our epistemologies and ontologies requires a somewhat adventurous attitude to research. It also often involves developing unconventional types of knowledge that may be difficult to judge and share through traditional forms of knowledge dissemination (e.g., academic publishing, see Kali 2016) and valuing (e.g., peer review, tenure processes, etc.).

Fig. 3
A plot of the D B R model. It plots a decreasing curve that provides data for the epistemological and ontological calibrations with abstract generalized explanations and particularization.

Model for calibrating DBR epistemologies and ontologies in DBR

Full size image

4.3 Methodological Alignment and DRTL: A CSCL Case Study

This case focuses on a DBR study conducted in the context a large-scale undergraduate level, semester-long course in biology. In addition to a quick summary of the story already told (described in detail in Sagy et al. 2019; Sagy et al. 2018; Tsaushu et al. 2012), the following sections present the story behind the scenes of this DBR study. Specifically, it illustrates how the back-and-forth movements within the abstraction–particularization curve enabled the DBR team to reach higher degrees of methodological alignment, calibrating between the two modes of inquiry, and how this eventually brought to their transformative learning, and the development of PPK (Fig. 3).

4.3.1 Story Already Told—Part 1: Redesigning an Undergraduate Biology Course

The motivation for this project came from the course instructors—two biology professors who have been teaching the course for many years in traditional ways. A DBR team was initiated, which included the instructors, two science education researchers, and two CSCL researchers. The research was conducted by gradually intervening within the course. In each of the 3 years of the study, a more advanced stage of the intervention was enacted with a new cohort of about 300 students. All three stages involved the use of a website that the team designed to go along with the course, which was used differently at the three stages of the intervention (Sagy et al. 2018). At the first stage (and first year of the study), the course was taught as it had been taught for years, through lecturing in a large hall. The only difference was that students could use the course website to review the contents taught in lectures. At the second stage, the instructors still gave lectures, but students were required to use the website. At the third stage of the intervention, the course website replaced the lectures. In addition, the instructor served as facilitator in weekly “mini-conference” meetings, each time on a different topic of the course with a different group of about 30 students. To prepare for this, students used team websites designed for this purpose, which included content resources as well as process scaffolds for developing team knowledge artifacts to share and discuss in the “mini-conference.”

4.3.2 Untold Story: Dilemma in Research Highlighting the Need for Methodological Alignment

The DBR team’s initial assumption was that within each stage of the intervention they will be able to find relationships between students’ patterns of use of the course website and their understanding of the scientific content. They also assumed that they will find improvement in learning outcomes and attitudes toward biology learning as the stage of the intervention became more advanced. (This represents a “scientific” mode of inquiry aspect of this DBR endeavor—going up the abstraction-particularization curve.)

However, following design, enactment, and data analysis (representing a “design,” or particularization mode of inquiry—going down the curve), both assumptions were refuted. That is, no meaningful or interesting findings were found using what seemed straightforward means of analysis (e.g., comparing students’ achievements in the course test between iterations, and seeking relationships between students’ use of the website and their learning outcomes within each iteration using learning analytics techniques). While interview data seemed to hint at deeper learning as the intervention advanced, the processes that supported student learning (mediating processes, in Sandoval’s 2014 terminology) were not clear, nor were the design features supporting them. Eventually, further back-and-forth movements within the abstraction–particularization curve enabled identification of a gap between the values that guided students in their learning process and the instructors’ perceptions about these values (Sagy et al. 2019).

4.3.3 Story Already Told—Part 2: The Culture of Learning Continuum as a Conceptual Lens

This new lens, which the DBR team called “the culture of learning continuum (CLC)” (Sagy et al. 2018), indicated that students who learned in more advanced versions of the course referred to course features with higher degrees of what was described in the CLC as internal values. Specifically, students were more likely to seek personal growth, appreciate the formative nature of assessment, make efforts to learn (and not only succeed in the test), negotiate meaning with peers (rather than seek the “right” answer for the test), and take ownership of their own learning process.

4.3.4 Retrospective Analysis of Relationships Between Methodological Alignment, DRTL, and PPK

Retrospectively, the difficulty to explain the intervention outcomes in terms of mediating processes at preliminary stages of the project, eventually, improved the teams’ methodological alignment. Changing what was measured (culture of learning instead of students’ patterns of use of the website) and how it was measured (measurement validity), transformed the DBR researchers conception about the intervention. That is, they developed a renewed understanding of what the intervention represented from a theoretical point of view (treatment validity). As a result, they developed a renewed view of their role as researchers and designers within the study. They took on a role that focused more on exploration within the unknown—being open to “build the plane while flying it”—discover the means of analysis while conducting the research, which required the blending of analytical and creative mindsets (McKenney and Reeves 2012/2018).

But there was also a shift in the ontological work being conducted—as designers, they understood that their role is to develop PPK in the form of not only the course’s website with its various digital resources but also the social activity structures that can support them and the cultural lens for explaining the rationale behind them (the principled aspect of the practical tools). These turned out to be crucial for continued implementation after the research was already over (evidence exists that the instructors continued to implement the advanced versions of the course for many years). This long-lasting effect was possible due to the transformative learning of the instructors too, who were part of the research team, who adopted to their professional identity a role of cultivating a culture of learning (Tsaushu et al. 2012).

4.4 Concluding Remark

The literal meaning of DBR (design-based research) is that we are nudging both the epistemology and ontology to follow scientific as well as design modes of inquiry and knowledge outcomes. This unusual property of the ontology, as pertained in PPK, calls for an unusual epistemology. At the same time, the dual epistemic game of advancing theory while advancing design helps holding the knowledge accountable. The eclecticism of DBR relates to the many ways we can intertwine scientific and design modes of inquiry, going back-and-forth the abstraction–particularization curve (Fig. 1). What unifies these activities is moving toward increased coherence, and therefore systemic validity. Over 15 years ago, Hoadley noted that “the promise of having better alignment in research—certain and sure links from theories to hypotheses to interventions to data gathering activities to interpretation and application—should be a strong incentive to continue to pursue the design-based research approach” (p. 211). The model we suggest for calibrating DBR epistemologies and ontologies (Fig. 3) can assist in capitalizing on the dual epistemic and ontologic game inherent to DBR, to spur creativity and innovation in rigorous research.

Thus, we claim that DBR, while accommodating multiple epistemic games, is not simply a laundry list of ways to make knowledge. Rather, our flexibility in DBR’s epistemic games should be driven by, and accountable to calibration between these games. In particular, we believe that the inherently embedded and contextualized nature of CSCL, as well as its design orientation, demands a set of knowledge activities which seek to use treatment, systemic, and consequential validity of research as the principles for moving between different epistemic framings, and indeed—different knowledge ontologies. By doing so, we transform not only the types of knowledge produced but also the knowers themselves, reshaping the role and perspective of students, teachers, and DBR researchers.