Keywords

These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Introduction

the Process of Designing Effective Learning Materials…?>In this chapter, the role of interpretation in research about learning is demonstrated by a research example using a specific methodology known as design-based research. This approach supports the design of educational interventions and learning materials to improve learning. In what follows, first the reasons to choose for this particular approach are explained referring to the main characteristics and procedure of this research methodology. Next, the invaluable contribution of this research approach is illustrated by a report of a study concerning the design of effective educational materials about the risks on social network sites. The research project described gives more insight in the total process of the design-based research methodology and approach. Finally, we describe the conclusions that are drawn and we discuss what is happening in terms of interpretation during design-based research in general and during the design and evaluation of educational materials about the risks on social network sites in particular. The specific advantages of this research approach are presented, but we also discuss the accompanying disadvantages and challenges of design-based research in an educational setting.

The Design-Based Research Approach

What Is This Approach About?

The design-based research methodology is a well-used research approach in the learning sciences (Barab and Squire 2004; Brown 1992; The Design-based Research Collective 2003) and relies on multiple sources of evidence, both quantitative and qualitative, which are triangulated (Cohen 2011). Yet, although a design-based research approach includes several well-established research methods and is based on existing norms for sampling, data collection, and data analysis (McKenney and Reeves 2013), the approach as a whole is fairly recent and evolved only near the beginning of the twenty-first century (Anderson and Shattuck 2012). The method mostly stands out because of the goals it puts forth (McKenney and Reeves 2013): it wants to bridge theoretical research and educational practice (Vanderlinde and van Braak 2010), thereby resulting in both an increase of theoretical knowledge and a societal contribution, such as school programs (Reeves 2006). The methodology has been defined by Wang and Hannafin (2005) as

A systematic but flexible methodology aimed to improve educational practices through iterative analysis, design, development, and implementation, based on collaboration among researchers and practitioners in real world settings, and leading to contextually-sensitive design principles and theories. (p. 6–7)

This definition includes different important characteristics of design-based research that were described by several authors and summarized by Anderson and Shattuck (2012). First of all, it focuses on the design and testing of a significant intervention. It therefore starts from problems that are both scientifically and practically significant, as is revealed in an initial problem analysis (Edelson 2002; McKenney and Reeves 2013). Second, it involves multiple iterations of testing and refining of problems, solutions, methods and design principles (Phillips et al. 2012). Third, throughout all phases of the design-based research, it is involving a collaborative partnership between researchers and practitioners. Fourth, the research needs to be conducted in real educational contexts, and not in labo-settings. Fifth, next to the development of practical solutions, it results in design principles, or “prototheories,” that help communicate relevant findings towards other researchers and practitioners (The Design-based Research Collective 2003). Finally, another characteristic that is not explicitly apparent in the given definition is the fact that the approach makes use of mixed methods, including a variety of research tools and techniques, as integrative research with varying methods is necessary to meet new needs and issues that emerge during the process (Wang and Hannafin 2005).

Following these characteristics, the procedure of design-based research, depicted in Fig. 1, iteratively involves four sequential steps (Reeves 2006): (1) the analysis of practical problems, (2) the development of solutions based on existing knowledge, (3) evaluation research of the solutions in practice, and (4) reflection to produce design principles.

Fig. 1
figure 1

Iterative steps of design-based research, based on Reeves (2006)

Full size image

Why Choosing a Design-Based Research Approach?

The design-based research approach is partly originated in reaction to the lack of theoretical base in designing and developing interventions to improve learning, the lack of theoretical implications of intervention research, and the lack of evaluation studies in authentic settings (Phillips et al. 2012; The Design-based Research Collective 2003). Since the methodology eliminates the boundary between design and research (Edelson 2002) and results in both theoretical contributions and practical solutions, this research approach is appropriate for research about the design of new educational learning materials.

Several advantages of design-based research have been described in literature. Edelson (2002) summarized the three most important reasons why someone should choose to use a design-based research approach. First of all, it provides a productive perspective for theory development, as it starts from a fully specified theory, shows inconsistencies of this theory by evaluating the design that was based on it, and ends in context-specific guidelines. The goal-oriented nature of the design-based research guides this theory development (Edelson 2002).

A second advantage of design-based research that was described by Edelson (2002) is the usefulness of the results. He states that in the past, practitioners often complained that they did not know how to implement the results that were found in research in their daily practice. Design-based research not only results in practical solutions that can be used immediately in the learning context, it also delivers design guidelines that can be used easily to develop similar interventions.

The third reason to use a design-based research approach, following Edelson (2002), is the fact that design-based research directly involves researchers in the improvement of education. Whereas previously, the design was often in the hands of publishers and practitioners, the expertise and knowledge of researchers now directly influences the development process, making innovative designs based on recent educational studies possible.

Next to these three advantages described by Edelson (2002), several other advantages have been described in design-based research literature. One of these is the fact that the use of real-life settings, in contrast to labo-settings, ensures the ecologic validity (Phillips et al. 2012). This aim for generalizability is highly valued, as it ensures the usability of materials in the classroom. Another advantage that is described is that it fulfills the norms of good research in general, including the articulation of clear goals and research questions, the cumulative and systematic nature of gathering evidence, and the use of methodologies that are appropriate to the research goals (Phillips et al. 2012).

Applying Design-Based Research: A Research Example

In this section, we describe a design-based research project that has been conducted from 2011 to 2013, as part of the Security and Privacy in Online Social Networks project, further referred to as SPION project. This project has received funding from the Strategic Basic Research (SBO) Program of the Flemish Agency for Innovation by Science and Technology (IWT). The main goal of the SPION project was to counter responsabilization (i.e., the process where the user of a social network site is responsible for its own safety and privacy) and to redirect responsibilities towards other institutions (e.g., service providers, schools, government, etc.). One of the subgoals was therefore to develop educational materials that can be used in secondary schools, to teach teenagers about the risks on social network sites.

IWT financed a 4-year PhD track, thereby allowing a multiyear project to be set up. By including researchers in the development of materials, the expertise necessary to do evaluation studies was ensured as well. The research group that was involved in this studies, a division of the Department of Education of Ghent University, already had built an expertise in doing design-based research and evaluation studies in secondary education (Raes et al. 2012; Schellens and Valcke 2004).

Since one of IWT’s conditions for funding was that the project would result in both solutions for practice and in scientific progress, the advantages and characteristics of design-based research were ideally suited.

Focus of the Research

This research project focused on a form of learning situated in the field of media literacy. Traditionally, media literacy refers to the ability to analyze and appreciate literature, but this meaning has been enlarged the moment computers became prominent in society (Brown 1998). With the rise of Web 2.0, the meaning of media literacy has evolved even more, as it covers not only interactive exploration of the Internet but also the critical use of social media and social network sites. Since social media give an excellent opportunity to create online content, the development of new skills is necessary. Livingstone (2004a) therefore describes media literacy in terms of four skills, this is the ability to access, analyze, evaluate, and create messages across a variety of contexts. Previous research shows that children are good at accessing and finding things on the Internet, but they are not as good in avoiding some of the risks posed to them by the Internet (Livingstone 2004b).

In this respect, schools are put forth as ideally placed to provide media literacy education to all children and teenagers (Livingstone and Haddon 2009; Marwick et al. 2010; Patchin and Hinduja 2010). In the current research, it was aimed to develop effective educational materials to teach children of secondary education (aged 12–19 years) how to behave safe on social network sites (i.e., to increase awareness and to change unsafe attitudes and behavior), and to describe critical design guidelines for the development of these materials.

Method and Results

As stated before, the procedure of design-based research iteratively involves four sequential steps as depicted in Fig. 1 (Reeves 2006): (1) analysis of practical problems, (2) development of solutions based on existing knowledge, (3) evaluation research of the solutions in practice, and (4) reflection to produce design principles. In the following, the methods used in this research throughout the sequential steps are described in detail, together with a short summary of the results.

  • Step 1: Analysis of Practical Problems

In a first step, the practical problem needs to be analyzed and a theoretical framework has to be articulated, including initial design guidelines to proceed to the next step (development). To analyze the practical problem, three important resources can be consulted: previous literature, shared experiences of researchers and practitioners, and one or more pilot studies (Reeves 2006).

In this research, the lead to answer the questions about the nature of the problem was taken from previous literature: do teenagers care about their privacy, are they behaving risky on social network sites, are they aware of the existing risks on social network sites, and what is the role of school education? The results of this literature study were extended with three pilot studies. First, an observation study of Facebook profiles was conducted, to find out what teenagers are doing on Facebook and whether they show risky information. Second, a theoretical evaluation of existing educational packages about safety on social network sites showed the gaps and challenges to develop new materials. And third, a survey study showed the impact of school attention for the topic of safety on social network sites on students’ attitudes and behavior. Finally, the experiences of practitioners were taken into account by organizing a focus group with educational stakeholders (i.e., teachers, developers of educational materials, educational counselors).

Following the results of the literature study and the three pilot studies, some conclusions were particularly important to guide the decisions in the next stage, that is, the development of solutions. In the literature study, it was found that the risks teenagers face on social network sites can be divided into three main categories: content risks, contact risks, and commercial risks (DeMoor et al. 2008). The first one includes encountering provocative or wrong content on your social network site, such as hate messages or gossip, respectively. The contact risks find their source in the fact that social network sites are made to communicate and have contact with others. Examples of contact risks are cyberbullying, sexual solicitation, and all kinds of privacy risks (De Moor et al. 2008; Livingstone et al. 2011). The third category of risks contains the commercial risks. These include the commercial misuse of personal data: information can be shared with third companies via applications, and user behavior can be tracked in order to provide targeted advertisements and social advertisement (Debatin et al. 2009). In the first pilot study that we conducted, the observation study, we found indeed that teenagers face a significant amount of risks (Vanderhoven et al. 2014e). In the organized focus group, it was found that cyberbullying and privacy risks are the most encountered risks by educational stakeholders such as teachers. These risks may form a threat, since research indicates that a significant amount of teenagers experience harm after exposure to online risks (Livingstone et al. 2011; Mcgivern and Noret 2011).

Further literature study revealed that a variety of prevention campaigns and awareness-raising interventions has been developed to account for the rising concerns about the new risks children face when using the increasingly popular social network sites (e.g., for an overview of European packages, see Insafe 2014). However, a systematic review showed that almost none of these interventions has been empirically evaluated (Mishna et al. 2010). The results of our second pilot study confirmed that most of the existing packages were developed without any theoretical consideration, nor with regard to the cause of the problem that is tackled, nor with regard to the intervention that is developed (Vanderhoven et al. 2014a). Moreover, the few evaluation studies that were conducted only show an impact of the interventions on Internet safety knowledge, but not on pupils’ behavior (Mishna et al. 2010). This is in line with the results from quantitative intervention studies about media literacy education in general that typically show an increase in knowledge about the specific topic of the course, but lack a measurement of attitudes and behavior (Martens 2010). If measured, it is found that attitudinal and behavioral changes are much harder to obtain (Cantor and Wilson 2003) or not found (Duran et al. 2008; Steinke et al. 2007). Still, in our third pilot study, a survey study, we found that school attention for the topic of online safety has a positive influence on pupils’ attitudes and behavior (Vanderhoven et al. 2013a).

As stated before, another typical characteristic of this first step of design-based research is the articulation of an initial framework (Reeves 2006). Therefore, next to the studies that were conducted to analyze the practical problem, initial design guidelines and predictors of effective materials were formulated. We took into account both general principles that are shown to be important in prevention campaigns (Nation et al. 2003) and more specific instructional design principles that follow out of the leading theory in education, that is, constructivism. Furthermore, because of the goal of our materials (i.e., not only changing awareness but also changing unsafe attitudes and behavior), theories of behavior change are taken into account as well. More specifically the study was based on the transtheoretical model of behavior change (Prochaska et al. 1992) and the theory of planned behavior (Ajzen 1991).

  • Step 2: The Development of Solutions

Based on the results of the first phase, educational materials were developed. A detailed design was created, and explicit goals about the outcome of these materials were put forth: an increase in awareness about risks on social network sites and a decrease of unsafe attitudes and behavior on social network sites. There was a special focus on contact risks as these were of most concern to the educational stakeholders in our focus group (privacy risks, cyberbullying, and sexual solicitation; DeMoor et al. 2008). The package consisted of a syllabus for the pupils and a manual for the teacher. Every course lasted 1 h, trying to satisfy the need of teachers to limit the duration of the lessons and the work load (Vanderhoven et al. 2014a). The different criteria that were put forth in our theoretical framework were taken into account during the development of the materials (Vanderhoven et al. 2014). All courses followed the same structure:

  1. 1.

    Introduction. The subject is introduced to the pupils, using the summary of risks (De Moor et al. 2008).

  2. 2.

    Two-by-two exercise. Students receive a simulated social network site profile on paper and have to fill in questions about the profile together with a peer. These questions were scaffolding the pupils towards the different existing risks on the profile.

  3. 3.

    Class discussion. Answers of the exercise are discussed, guided by the teacher.

  4. 4.

    Voting cards. Different statements with regard to the different contact risks are given. Students agree or disagree using green and red cards. Answers are discussed, guided by the teacher.

  5. 5.

    Theory. Some real-life examples are discussed. All the necessary information is summarized.

  • Step 3: Evaluation Research of the Solutions in Practice

The materials that were developed were implemented in authentic classroom settings in secondary education, and the impact of the materials on the awareness, attitudes, and behavior of the pupils that were involved during the intervention was measured. Based on the results, materials have been refined. These revised materials were implemented again. In total, there were five iterations of development, evaluation, and refinement. The methodology was mostly equal for the five different intervention studies. However, some small changes have occurred. This is a typical characteristic of design-based research, where integrative research with varying methods is necessary to meet new needs and issues that emerge during the process (Wang and Hannafin 2005).

The materials were implemented in classes in secondary schools. In the first intervention study 1,035 pupils participated, in the second intervention study 1,487 pupils were involved, and in the third intervention study 156 pupils followed the course. In all these studies, the pupils were on average 15 years old. In the last two intervention studies, slightly younger students were involved, because in these studies the importance of parental involvement was tested, which is particularly important in lower grades. The mean age of the 146 pupils that were involved in the fourth intervention and of the 205 pupils in the fifth intervention was therefore 13 years.

A pretest–posttest design was used in all intervention studies. This means that in all conditions, in all studies, pupils had to fill in an online pretest survey before the intervention took place. Afterwards, they followed the intervention, which was different in all studies. Finally, they filled in a posttest survey. In all intervention studies, a specific experimental intervention was compared with a control group. In the first two studies, no intervention took place in this control group, and pupils only had to fill in the surveys. In the last three studies, the intervention out of a previous phase was given to the control group, so that comparisons with the experimental group indicated the added value of the revised materials.

The survey measured pupils’ awareness, attitudes, and behavior towards contact risks on social network sites. This survey was developed based on the contact risks as described by DeMoor et al. (2008). In the first two studies, three different scales were developed, one for awareness, one for attitudes, and one for behavior, all built on the base of the means of six or more items. They all had a satisfactory reliability as measured by Cronbach’s alpha. In the last three studies, the survey was shortened, because pupils and teachers reported that it was too long and time consuming. Therefore, a new and shorter survey was developed with less items on the awareness scale and with attitudes, intention, and behavior measured based on the theory of planned behavior following the manual of Fishbein and Ajzen (2009).

In all studies, an open question asked pupils about what they had learned during the intervention, to have a direct measure of increased awareness. Moreover, a direct binary measure of behavioral change was conducted by the question; “Did you change anything on your profile since the first questionnaire?” If the latter was answered affirmatively, an open question about what they changed exactly gave us more qualitative insight in the type of behavioral change.

Multilevel modeling (MLM) with the software package MLwiN was used to analyze the data. Since our data clearly have a hierarchical structure, that is, pupils in classes, the obtained data from pupils out of the same class might be dependent and might so break the assumptions of simple regression analysis. In this respect MLM is suggested as an alternative and adequate statistical approach. Consequently, since a significant between-class variance could indeed be observed in the first two studies, a two-level structure is used: pupils (level 1) are nested within classes (level 2). The impact of the intervention on different posttest scores—when controlling for the pretestscores—is evaluated by comparing the control condition with the experimental conditions. Bonferroni corrections were used to control for multiple testing. However, in the last three studies, no significant between-class variance could be observed, so there was no need to use MLM. Therefore, a multivariate repeated measure approach has been used, using the software package SPSS.

The different iterations of implementation, evaluation, and revision were previously described in detail (Vanderhoven et al. 2014) and are summarized in Fig. 2. In the first intervention study, the originally developed materials (see step 2) were implemented and the impact was compared with a control condition where no intervention took place (Vanderhoven et al. 2014b). It was found that while there was an impact on pupils’ awareness about the risks on social network sites, only limited impact on behavior could be found. Based on these results, observations in the classrooms and remarks of teachers and pupils, and the theoretical framework of behavioral changes that was put forth in step 1, the materials were revised: peer influences during the course were reduced by decreasing moments of collaborative learning, making place for more time for individual reflection. In the second intervention, it was found that this new intervention had more impact on attitudes and behavior, while the impact on awareness was still the same (Vanderhoven et al. 2012a). Since there is always room for improvements in the design (Anderson and Shattuck 2012), the materials were revised again. Based on remarks of pupils and teachers, the authentic setting that was used in the materials was made even more authentic by including the own social network site profile in the course. It was however found that this manipulation did not increase the impact of the intervention (Vanderhoven et al. 2013b). In the final two studies, the importance of parental involvement was tested by revising the materials so that parents were included. First, a parental evening was organized next to the course that was given to the pupils. In the fourth evaluation study, it was found that this was not enough to involve all parents (Vanderhoven et al. 2014d). Second, the materials were revised so that parents were involved in a homework task. This appeared to increase the impact that the intervention had on the behavior of the pupils, especially for boys (Vanderhoven et al. 2014c).

Fig. 2
figure 2

Iterative process of implementation, evaluation, and revision of the designed materials, based on Vanderhoven et al. (2014)

Full size image

  • Step 4: Reflection to Produce Design Principles

Summarizing the results of the previous step, it can be stated that materials need to include time for individual reflection and that involving parents in the intervention is beneficial, especially for boys. However, it needs to be taken into account that involving parents using an information evening might not be enough to include all parents. Involving parents as partners, using a homework task, is put forth as a good alternative. Considering the authentic context, exercises with simulated profiles are just as good as real online profiles to obtain the goals that were set. Taking into account all these findings, a final practical solution has been developed that effectively has an impact on both awareness and unsafe behavior.

However, design-based research results not only in practical solutions but also in a theoretical contribution. Therefore, the last step of design-based research includes a reflection of the total research procedure and all findings, resulting in both practical solutions and improved theoretical understandings (Reeves 2006). A the start of this research, different theoretical frameworks were put forth, such as the general principles that are shown to be important in prevention campaigns (Nation et al. 2003), more specific instructional design principles out of constructivism, and the theory of planned behavior (Ajzen 1991). In the light of the results found in this design-based research project, these frameworks needed to be reinterpreted and contextually sensitive design principles and theories were put forth. For example, collaborative learning, which was proposed as an important instructional strategy following constructivist theories (Duffy and Cunningham 1996), appears to be less effective in the case of reputation-related behavior like unsafe behavior on social network sites. In the same way, the importance of authentic learning and of parental involvement was put into perspective. Following these results, context-specific guidelines were formulated (for more details, see Vanderhoven et al. 2014).

Presenting Results

As stated, the results of this research are important for both researchers and practitioners, such as developers of new e-safety materials. Therefore, different formats of presenting these results were chosen. First of all, to reach researchers, the results of this research were disseminated by means of academic publications. Every single study of all stages of the design-based research was presented separately and in detail (Vanderhoven et al. in press, 2013a, b, 2014a–e). Moreover, the design-based research as a whole has been presented in detail in another academic manuscript (Vanderhoven et al. 2014). Second, to reach both researchers and practitioners, the results have been presented at several academic and nonacademic conferences (e.g., EARLI, AERA, IAMCR, Media and Learning Conference, etc.). Finally, to reach practitioners, articles have been published in nonacademic journals (e.g., Vanderhoven and Schellens 2012), presentations were given at training seminars (e.g., Insafe training meeting), and workshops were organized.

Evaluating the Design-Based Research Approach

To overcome problems of previous research, in this study it was chosen to use a design-based research approach. So far, due to limited financial recourses and expertise, the existing educational materials about the risks on social network sites were built without a strong theoretical base, and no research was carried out to evaluate the possible impact of these packages (Vanderhoven et al. 2014a). Since IWT financed a multiyear project involving researchers with expertise in design-based research, these problems could be overcome in the SPION project. Moreover, the advantages and specific characteristics of design-based research were well suited to fill in the gaps that existed in the literature and research about e-safety interventions. First, based on the fact that the originally developed materials did not obtain all the goals that were put forth (i.e., they did not change unsafe behavior), the initial design principles drawn from the initial framework during the first step of the research were adapted, and the design-based research project did not only result in the development of effective materials but also in contextually sensitive design principles. Second, the typical collaboration among researchers and practitioners in this type of research helped us to find a balance between the teachers’ needs and the guidelines based on previous research. For example, while previous research about effective prevention campaigns shows that interventions should be sufficiently dosed (Nation et al. 2003), a short-term intervention would be more satisfying for teachers who reported a high workload (Vanderhoven et al. 2014a). In the design-based research, it was found that a short-term intervention appears to be enough to have an impact. This also maximizes the possibilities for dissemination and usefulness in practice. This is especially important given the conditions that were put forth by the financing institute IWT. Third, design-based research directly involves researchers in the improvement of education. Whereas, previously, the design and development of educational materials was often in the hands of publishers and practitioners, the expertise and knowledge of researchers now directly influences the design. For example, as stated before, previous e-safety interventions were often not evaluated, due to a lack of expertise of the designers. In our research example, it is the conjunction of the experiences of the practitioners, and the knowledge and theoretical background of the researcher that made several evaluation studies possible which led to an effective course that could change both risk awareness and unsafe behavior.

Yet, although most literature focuses on the invaluable contribution and advantages of design-based research, some of the pitfalls that are inherent to this research approach also need to be mentioned (Anderson and Shattuck 2012; Barab and Squire 2004; McKenney and Reeves 2013). For example, while generalizability and ecological validity are often argued to be positive aspects of design-based research, the fact that design principles are context-specific might also jeopardize the external validity of the implications. In our research example, the research only assures that formulated design principles are applicable in the context of teaching pupils about the risks on social network sites and how to behave safely. However, these guidelines might also be applicable on the design of interventions about different behaviors that are typically tackled in other prevention campaigns, such as smoking, drug abuse, or aggressive behavior. However, further research is necessary to prove this generalizability.

A second challenge that is described in the literature, is the fact that it is difficult to know when (or if ever) the research program is completed. The multiple iterations ascertain cumulative knowledge and an improvement of the design, but as stated before, there is always room for upgrading (Anderson and Shattuck 2012). When can one decide a design is good enough to finalize the research? In the research example above, five iterations of development, implementation, and evaluation have been conducted. However, several more iterations could have been conducted, possibly even increasing the impact of the intervention. Most of the time, the end of funding means the end of research, independent of whether this happens after one or five iterations (Anderson and Shattuck 2012).

These time limits are a third disadvantage of design-based research: the total research procedure is very time consuming, considering the different iterative phases that need to be completed. It often needs a multiyear project to finish a design-based research (Anderson and Shattuck 2012). The research example described above indeed took about 3 years, with every step of the process lasting several months. The choice to conduct this time-consuming research was possible since a larger research agenda was financed, which is most often not the case. However, it also has more negative consequences for the research itself, such as the fact that only a short-term impact of the intervention is measured. Since conclusions of one study lead to the next step of the research, it is difficult to include long-term impact. If a long-term impact would be measured, several further steps of the research process would already have been started or even finished. Given the raising importance of sustainable learning, additional research using a longitudinal approach might be interesting, not only to find out if the materials have a delayed impact but also to find out whether the impact of the intervention is persistent over time.

Finally, it should be noted that although we elaborated on the advantages and disadvantages of design-based research for our research project, it is difficult to evaluate the impact of this research method in general. Anderson and Shattuck (2012) reviewed the impact of several design-based research interventions, concluding that it might be meeting its promised benefits, but McKenney and Reeves (2013) reacted that next to the scientific impact, which is easy to find in academic articles, there is also a practical impact, which is much harder to identify and therefore to evaluate.

The Role of Interpretation in DBR

In every research, interpretation plays a undeniable role. It is the interpretation of the context by the researcher that determines which research method will be used. It is the interpretation of the researcher that decides which measurement tools will be used. Participants interpret the questions asked in these measurement tools, which again has an impact on the results. Further, the results need to be interpreted by the researcher again: how will he or she analyze the gathered data. Finally, the researcher decides what is interesting for whom, when determining which results he or she will report, how and where.

In design-based research in particular, it is important to acknowledge the role of interpretation. One of the reasons is the close involvement of the researcher in the design process, and the bias that this involvement may cause. Some authors state that the results of the research must be biased because of the interpretations of the researcher, while others claim that these researchers with their biases, insights, and understanding are the best research tool (Anderson and Shattuck 2012; Barab and Squire 2004). Following the involvement of the researcher in the designing process, ethical issues are raised as well (Barab and Squire 2004). When observing problems in school, do they intervene, or do they minimize their impact in the classroom?

In addition, it should be noted that while the influence of interpretation is important in one single study, the accumulation of these interpretations and decisions in the different steps and studies of a design-based research even increases this influence. As different studies are sequentially conducted, with the results of each study influencing the setup of the following study, the interpretation of the results has a very big impact on the progress of the study as a whole and the final results. Moreover, as stated before, it is argued that because the researcher is closely involved in all the research steps, including the implementation of materials in real-life classroom settings, “researcher bias” is even larger when using this methodology (Barab and Squire 2004). However, as is shown in the different chapters of this book, interpretation is inherent to every research and should not paralyze us or prevent us to do any research at all.

In the following paragraphs, we will repeat the different steps of our research, thereby indicating the role of interpretation in every phase. The decisions and interpretations that we made as a research team are only examples of the interpretations any researcher needs to make when conducting design-based research.

  • Step 1: Analysis of Practical Problems

As explained in the methodology section, there are three important resources to describe the problem: previous literature, shared experience of researchers and practitioners, and one or more pilot studies (Reeves 2006). A first decision a researcher needs to make is what he or she will do to analyze the problem, and to what extent. It could be decided only to focus on previous research or to have one focus group with practitioners to have an idea of the state of the art. It could also be decided to conduct multiple pilot studies, making a broad idea of the state of the art possible. Several aspects, such as time constraints, expertise, and practical opportunities, influence these decisions.

In our research, it was chosen to complete an extensive needs analysis, including three exploratory studies, next to the literature study, and one focus group with practitioners. Concerning the literature study, it is clear that the interpretation of the researcher is of very big importance. One example is the search for information about privacy care with teenagers (Vanderhoven et al. 2013a). In this search, we found that some authors reported that teenagers care about their privacy, while others reported the opposite, depending on the exact measure of privacy care in their study, the age of the respondents, and other methodological differences. These kinds of contradictions are often found in literature and should be taken into account when making a state of the art during this first needs-analysis phase.

Next to the interpretation of previous literature, different decisions needed to be made about the method, the data collection, the measures, the data analysis, and the reporting of every single pilot study. As an example, we analyzed the different interpretations made in the observational study of Facebook profiles (Vanderhoven et al. 2014e). We chose to use the method of observation, to overcome problems that are inherent to the self-report methods that are mostly used to study teenagers’ behavior on social network sites, such as social desirability (Phillips and Clancy 1972). With this, we wanted to eliminate the amount of variation caused by the interpretation by the participants of the questions in a survey. However, this does not mean that observation is free of interpretation. A detailed codebook was developed to code the information that was observed on the different pages. It was tried to be as exhaustive as possible when composing this codebook, but there is always information that is excluded, depending on the choices of the researchers. Moreover, we chose to use research assistants to collect the information. A total of 179 research assistants coded the information on the Facebook profiles of their friends and friends of friends. By including so many researchers, it was aimed to randomize the researcher bias, a method rarely used in social sciences. While most of the time, researcher bias is tried to be eliminated (although it can be argued that this is quite impossible, hence the focus of this book), we tried to randomize the impact of the observer, thereby eliminating the importance of the different interpretations for the overall research results. Finally, the results of this study were also impossible to report without any interpretation. To give an example: 34 % of the minors in the study were tagged in pictures in which they were drinking alcohol. This is a fact, a number, that can be interpreted in several ways. Is it a risk? Is it a problem? Is 34 % a significant amount, enough to put effort in preventing it? In our research team, we concluded that indeed a significant amount of teenagers show risky behavior (of course, there were also other risk indicators that we found to be threatening), and we based our further intervention on these interpretations. It is important to note that this is a decision and that others might feel that the risks teenagers face are not important enough to put so much effort in prevention campaigns.

  • Step 2: The Development of Solutions

Design is a sequence of decisions made to balance goals and constraints that can be divided in three sets of decisions: how the design process will proceed, what needs and opportunities the design will address, and what form the resulting design will take (Edelson 2002). These decisions are guided by the results that are found in the previous step of the research, the needs analysis and other context variables, such as the financing resources. In our research example, the development of materials is guided by the results of the needs analysis but also by some context variables. As stated before, a research team funded by the Flemish Agency for Innovation by Science and Technology (IWT), more specifically in the context of a Strategic Basic Research (SBO) Program, conducted this research. These kinds of projects have an important focus on valorization of the research results and on the value of the research for society. The Flemish government showed special interest in the development of educational materials, because they believe that media literacy is very important to teach children how to behave safe on social network sites. This implicates that, as a researcher in this project, the stakes, norms, and values were colored, that is, the project started from the fact that a security problem was existing regardless of how the user felt about this (as was found in the first step, the needs analysis). While this is not detrimental by nature, and although the development of materials was still primarily based on the results of our extended needs analysis, it is important to keep in mind these norms and values that were part of our research from the start.

During the development process, interpretation is also involved when the researcher must decide about the goals he or she wants to accomplish with the educational materials. In our research example, we have put forth that our materials aim to raise awareness and to change unsafe behavior on social network sites. Putting forward the goal of changing behavior cannot be done without any consideration. Indeed, trying to change behavior can be seen as paternalistic and undemocratic (Kelman 2001). While it can be argued that teenagers deserve to be informed, so that they can make informed decisions when using social network sites, it can also be argued that it is unethical to decide how they actually should behave. One should keep in mind that developing educational materials always includes the developer’s expectations of desirable attitudes and behavior. This is not always in line with the goals and expectations of the pupils. It can be argued that every individual has the right not to care about certain risks and to choose to behave “unsafe” on social network sites if that is what he or she wants, given the benefits this entails, such as communication (Pruulmann-Vengerfeldt and Runnel 2012) and identity formation (Hum et al. 2011; Madden and Smith 2010). However, it is generally believed that schools have a broad educational agenda, including the enhancement of pupils’ character, health, and civic engagement (Greenberg et al. 2003). School education needs to enable pupils to participate fully in public life (Cazden et al. 1996). It can be argued that in the twenty-first century, this means that schools have a responsibility to teach teenagers how to behave safe on social network sites. In this line of thought, putting forward the goal of attitudinal and behavioral change next to raising awareness seemed appropriate.

  • Step 3: Evaluation Research of the Solutions in Practice

As mentioned before, the results of each study of the design-based research influence the setup of the following study. This is especially the case in the third step of the research: the iterative implementation, evaluation, and revision of the materials. For every cycle, the revisions are based on quantitative and qualitative results (sometimes contradicting each other, making an interpretation by the researcher necessary to proceed in the research), observations in the classroom, collaboration with practitioners, and a theoretical framework. It is the conjunction of all these different aspects, which guides the decision to change specific aspects of the materials and to improve the impact that these materials have on the pupils. It goes without saying that this amount of information can lead to different decisions, making the interpretation and the decisions of the researcher at the moment of revisions of materials of tremendous importance for the final results.

To demonstrate this importance, we analyze the decisions of our research team during the first revision of materials (after the first intervention study), described by Vanderhoven et al. (in press). The materials were changed so that moments of individual reflection were increased during the intervention, while moments of collaborative learning were decreased, trying to minimize peer pressure during the course. This decision was based on different pieces of information: the observation that popular kids raised their voice during the course to influence their peers, the quantitative and qualitative results of the first study indicating that there was no impact on unsafe behavior (Vanderhoven et al. 2014b), the theory of planned behavior stating that the social norm has a significant impact on people’s behavior (Ajzen 1991), and theories about peer pressure in adolescence stating that teenagers are especially vulnerable for peer pressure (Sumter et al. 2009). Of course, there were other observations as well that might have influenced the impact of the intervention but that were not chosen for revision. For example, maybe students did not have the technical skills to act safer and more attention should have been given to the training of specific skills.

  • Step 4: Reflection to Produce Design Principles

The design guidelines that are formulated are based on the results of the previous steps of the research and therefore again dependent on the interpretation of the researcher. Moreover, it is dependent on the amount of iterations, the choices made about the revisions of the materials, and so on. In our research example, the design guidelines that were put forth are not exhaustive, as time constraints limited the amount of iterations to five. More design guidelines might have been revealed in other iterations.

Finally, the interpretation of the researchers about the dissemination of the results is vital. For example, to reach practitioners we formulated simple rules of thumb in the final materials, derived from the design principles. This enabled teachers to use these guidelines during their courses. The rules of thumb are, by definition, a simplification of the conclusions of the results of the total design-based research and therefore dependent on the interpretation of the researcher with regard to what is most important in the conclusions of this research.

Conclusion

In this chapter, it was argued that a design-based research approach can eliminate the boundary between design and research (Edelson 2002), which can result in both theoretical contributions and practical solutions. Although this research approach is appropriate for studying the design of new educational learning materials, it must be concluded that interpretation plays an undeniable role in all research, which accumulates throughout the different studies in design-based research.

It is therefore of significant importance to acknowledge the presence of interpretations in this approach. In that sense, design-based research can be seen as a story, which can be told as objective as possible, but which is undoubtedly colored by the interpretations of the storyteller. Nevertheless, we hope that the methodology described in this chapter can inspire other researchers to write their own story based on their own interpretations.