Introduction

In cyber society, new participatory communication platforms are currently rapidly evolving. Social network sites (SNSs) are an expression of these new communication technologies. SNSs are basically Internet communities which allow individuals to interact online through profiles representing their identities and their networks of connections (Acquisti and Gross 2006). Although many authors have emphasized the numerous opportunities which SNSs offer, many risks have been reported as well (Christofides et al. 2012). This gives cause for concern with researchers and policy makers (Safer Internet Programme 2009; Walrave and Heirman 2013; Watson et al. 2006).

Although these researchers and governments agree that educating children about online safety is an important future challenge, the initiatives coupled with the organization of this type of education are not evidence-based (Mishna et al. 2010). Hence the impact of these initiatives remains unclear and little is known about the conditions in which they are effective (Livingstone and Bulger 2013).

The current article aims at proposing validated design principles which can be adopted by future developers. To pursue this goal, a design-based research approach is presented which fills the gaps found in previous studies on this topic. As design-based research is a recently developed methodology and given the extensiveness of the research procedure, only a limited number of studies is available reporting on the full design-based research process (Anderson and Shattuck 2012). Researchers consequently argue that more reports of the full design process should be available in literature (McKenney and Reeves 2012). This full research example therefore also contributes to the design-based research literature by making the full process more transparent.

Education on safety on SNSs

As stated above, many risks which accompany the use of SNSs have been reported (Christofides et al. 2012). Risks related to the communicative aspect of SNS platforms, the so-called contact risks, form one of the main categories (De Moor et al. 2008; Livingstone et al. 2011). SNSs are for instance very popular media used for cyberbullying (Livingstone et al. 2011). Children with a SNS profile are more likely to be bullied than children without such a profile (Lenhart 2007). Moreover, SNSs are used to send sexual messages (Livingstone et al. 2011). Especially girls report having experienced online contact which makes them feel uncomfortable (Smith 2007). Teenagers also face privacy risks when posting personal information online while not defining privacy settings (Livingstone et al. 2011).

In light of the increasing concerns about these new risks, safer Internet for children is high on the international policy agenda, as reflected in the Digital Agenda for Europe which puts forward the goals of “scaling up awareness and empowerment including teaching of digital literacy and online safety in all EU schools” (European Commission 2012). Moreover, the European Commission and U.S. Homeland Security have signed a joint declaration to “work collectively and in partnership to reduce the risks and maximize the benefits of the Internet for children.” (Department of Homeland Security and the European Commission - Joint declaration 2012). Worldwide support for this digital agenda clearly shows through the international support for Safer Internet Day, which is annually organized by 103 countries across the world to promote a safer and more responsible use of online technology, especially by young users (Safer Internet Day, 2013).

In light of that policy agenda, the role of schools and education in general is to teach about online safety. This has been emphasized by different stakeholders, including teenagers themselves, besides parents, teachers, policy makers and researchers (Livingstone and Haddon 2009; Marwick et al. 2010; Safer Internet Programme 2009; Tejedor and Pulido 2012). The role of schools and education also fits their broader educational agenda, including contributing to the pupils’ health and civic engagement and enabling pupils to participate fully in public life (Greenberg et al. 2003; The New London Group 1996). Therefore, it can be argued that in the 21st century schools have a major responsibility to teach teenagers how to act safely on SNSs. Consequently online safety has been formally included in school curricula in many European countries to contribute to broader media literacy or technology programs (Safer Internet Programme 2009). An analysis of international technology curricula shows that they are based on the rationale that all children must become digital literates in order to face the knowledge-based society (Aesaert et al. 2013). Since teenagers lack the skills to avoid the risks of the Internet (Livingstone 2004), a critical and safe use of SNSs is a crucial component of media literacy education to be taught in schools.

However, the implementation of the curriculum topic—online safety—in the classroom appears to be inconsistent (Safer Internet Programme, 2009). Studies in England show that 42 % of the teachers never teach about online safety. So, in order to help the implementation of the curriculum in the classroom, there has been an enormous increase in the number of prevention campaigns and awareness-raising interventions (See Insafe 2014, for an overview of European packages). However, a recent systematic review shows that almost none of these packages has been empirically evaluated (Mishna et al. 2010), so that it remains unclear whether or not they have had any impact on the teenagers’ awareness, attitude or behavior. Packages which have been evaluated do result in an increase in Internet safety knowledge but not in safety behavior (Mishna et al. 2010). This is in line with the findings on media literacy education in general, where intervention studies systematically reveal that media literacy education increases the knowledge about the specific topic of the course, although attitudinal and behavioral changes usually fail to occur (Duran et al. 2008; Steinke et al. 2007). Up until now, it is unclear which characteristics can guarantee that interventions and prevention campaigns may effectively change awareness and unsafe behavior and under which circumstances that change is accomplished (Livingstone and Bulger 2013).

In this research we elaborate on the results of previous studies and try to fill the aforementioned gaps by developing evidence-based educational materials about risks on SNSs and by proposing validated theoretical design principles to help future practitioners and developers. In order to achieve this goal, a design-based research approach is proposed.

Design-based research

Since a design-based research approach partly originated as a reaction to a lack of evaluation studies in authentic settings and a lack of theoretical implications of intervention research (Phillips et al. 2012; The Design-based Research Collective, 2003)—which are also the aforementioned gaps in the education about safety on SNSs—the methodology is ideally suited for the current research. The design-based research methodology is a well-used research approach in the Learning Sciences (Barab and Squire 2004; Brown 1992; The Design-based Research Collective 2003) which triangulates multiple sources of evidence, including quantitative and qualitative data (Cohen 2011). However, although the approach makes use of several well-established research methods (McKenney and Reeves 2013), it has evolved only recently (Anderson and Shattuck 2012).

Design-based research is defined by Wang and Hannafin (2005) as “a systematic, but flexible methodology aimed to improve educational practices through iterative analysis, design, development, and implementation, based on collaboration among researchers and practitioners in real world settings, and leading to contextually-sensitive design principles and theories” (pp. 6–7). Therefore, the output of design-based research does not only include an increase in theoretical knowledge, but also adds a societal and educational contribution (Edelson 2002; McKenney and Reeves 2013; Reeves 2006; Vanderlinde and van Braak 2010). Although few studies report on the entire design-based research procedure (McKenney and Reeves 2013), a review reveals promising results for this methodology (Anderson and Shattuck 2012).

Reeves (2006) states that a design-based approach involves four sequential steps: (1) analysis of practical problems, (2) development of solutions based on existing knowledge, (3) evaluation research of the solutions in practice, and (4) reflection on the produced design principles (see Fig. 1). Previous literature reporting design-based research studies often only reports on one of these steps (e.g., Hakkarainen 2007; Land and Zimmerman 2015; Lee and Thomas 2011). In this study, the entire design-based research approach has been used to develop effective educational materials to teach children in secondary education (aged 12–19 years) how to act safely on SNSs and to describe critical design principles for the development of these materials. Therefore, this paper demonstrates the useful contribution of design-based research as it delivers interesting output for researchers, developers and practitioners.

Fig. 1
figure 1

Design-based research, based on Reeves (2006)

Full size image

Method

In the next paragraphs it is described how the different steps as defined by Reeves (2006) are rolled out in this study. As it is typical of design-based research, the researchers—who have expertise in instructional design—were in close cooperation with practitioners, such as educational publishers, educational developers and other educational stakeholders (teachers, pupil guidance centers, headmasters,..) during the full process of the design-based research.

Step 1: Analysis of practical problems

As can be seen in Fig. 1, the analysis of the problem is based on different types of input data. In this study the results of previous research have been extended with three exploratory studies all conducted in the first year of the design-based research. Firstly, an observation study of Facebook profiles led to a better insight in teenagers’ SNS behavior. Secondly, a theoretical evaluation of existing educational packages dealing with safety on SNSs and a focus group with educational stakeholders showed which gaps and challenges needed to be taken into account when developing new material. Thirdly, a survey study showed the impact of the attention schools give to the topic of safety on SNSs. More detailed information about the methodology of these three studies can be found in Table 1.

Table 1 Goals and methodology of the three studies in the first step of the design-based research procedure
Full size table

Finally, to conclude the first phase of this research, a framework was described based on existing literature, which formulates initial design principles and predictors of effective material.

Step 2: The development of solutions

In the second year of the research, educational materials were developed based on the results of the first phase. A detailed design was created, and explicit goals for the outcome of these materials were set. More precisely, an increased awareness when it comes to risks on SNSs and a decrease of unsafe attitudes and behavior on SNSs were targeted.

Step 3: Evaluation research of the solutions in practice

Immediately after development, the materials were implemented in authentic, Belgian classrooms in secondary education. The awareness, attitude and behavior of the pupils who were involved during the intervention were measured in the course of the week before and the week after the intervention. It should be noted that only self-reported measures were used, as we felt that it would be unethical to teach teenagers about privacy and the importance of securing online information, while at the same time asking for their consent to observe their personal information and their actual behavior on SNSs.

Based on the impact results, the materials were revised and implemented again in different classrooms. In total, there were five iterations of development, evaluation and refinement, equally spread over the second and third years of the research. The methodology was generally identical for the five different intervention studies, save for some minor changes in the surveys used (see Table 2 for more details), which is a typical characteristic of design-based research as it is necessary to be flexible when it comes to the used methods in order to meet new needs and issues revealed during the process (Wang and Hannafin 2005).

Table 2 Goals and methodology for the five studies in the third step of the design-based research procedure
Full size table

Step 4: Reflection to produce design principles

It is important that design-based research goes beyond designing and testing particular interventions. It should lead to sharable “proto-theories”, which help to communicate relevant implications to practitioners and educational developers (The Design-based Research Collective 2003). Therefore, to obtain the final goal of this research we reflect on the overall research procedure and all findings resulting in the proposal of context-specific theoretical design principles in the final step of design-based research procedure.

Results

Step 1: Analysis of Practical Problems

Exploratory studies

Observational study

In this first study, it was found that most young people post pictures (100 %), interests (95 %) and some basic personal information (90 %) on their profile. Some of them manage their privacy settings so that this information is confined to friends’ eyes only, but much information is still accessible on the ‘friends-of-friends’ pages (e.g. 94 % show pictures, 64 % show date of birth). Detailed analyses show that teenagers face a significant risk, for example by posting risky information such as sensitive pictures (18 %) or by joining hateful group pages (36 %). Moreover, older teenagers and girls post more (risky) information although there are no differences in privacy settings (for more detailed results, see Vanderhoven et al. 2014).

Theoretical evaluation of existing educational material and focus group

Main results show that the evaluated packages are attractive and ready to use, although some weaknesses were exposed. Most packages deal with safety on the Internet in general and do not only focus on SNSs. Therefore, some typical aspects of SNSs are often overlooked, such as the impact of hate messages and the risks of identity- shaping content such as messages revealing alcohol abuse or negative attitudes toward school or superiors (Vanderhoven et al. 2014c). A more comprehensive approach concerning the different risks is preferred. Moreover, the focus group showed that the most important risks educational stakeholders face with their pupils are cyberbullying and privacy risks. They also report that educational packages should be clear and concise (not time-consuming), attractive, with varying teaching methods and a teacher manual (which ideally contains learning goals). They prefer the formula by which the material can be treated by the teacher in one course (Vanderhoven et al. 2014c).

Survey study

The results of the survey study reveal that teenagers do not care much about their privacy, and, moreover, that a lack of privacy care leads to unsafe behavior on SNSs (Vanderhoven et al. 2013). However, results also showed that school education has a positive influence on privacy care, and that, by raising privacy care, it also has an indirect positive influence on the safety of pupils’ behavior (confirmed by a bootstrapping method, Vanderhoven et al. 2013). In conclusion, the results suggest that a heightened effort for school education about safer use of SNSs is important, especially since the attention for the topic in schools is found to be still extremely limited and not laid down in the curriculum.

Proposing an initial theoretical framework

To complete the first step of the design-based research, an initial theoretical framework is described. The current research is theoretically founded on two levels. Firstly, it starts from initial theoretical design principles, which are based on findings in general prevention research (Nation et al. 2003), and specific instructional design principles drawn from constructivism, which is the leading theory in the field of learning sciences (Gordon 2008). Secondly, the separate studies in this design-based research are supported by several specific theories, such as the theory of planned behavior (Ajzen 1991).

The initial design principles derived from prevention research are described by Nation et al. (2003). In a review of reviews, they identified nine general principles of effective prevention campaigns which transcend specific content areas. Important program characteristics are that they need to be comprehensive, make use of varied teaching methods, be well dosed, be theory driven and need to encourage positive relationships. Furthermore, the program needs to match the target audience, which entails it should be appropriately timed and socio-culturally relevant. Lastly, the implementation and evaluation of the program are important as well. This also necessitates relying upon a well-trained staff and an outcome evaluation.

The specific instructional design principles drawn from the field of learning science are based on the dominant theory in the past decades: constructivism (Gordon 2008). In light of that theory, the following principles were proposed: active learning (Duffy and Cunningham 1996), situated learning (realistic and authentic settings, Snowman et al. 2008), and collaborative learning (Duffy and Cunningham 1996).

To guide the revisions and iterations of the developed materials, theories about behavioral changes are taken into account, as the goal of the intervention is not only to heighten awareness, but also to improve unsafe attitudes and behavior. Different theories predict that attitudes precede behavior. The trans-theoretical model of behavior change (Prochaska et al. 1992) states that a contemplation phase in which the problem is recognized, precedes the preparation phase and action phase in which behavior is changed. The same prediction can be deduced from the theory of planned behavior (Ajzen 1991), which states that attitudinal beliefs, combined with subjective norms and perceived behavioral control, predict behavioral intentions and thus behavior change. Meta-analytic reviews show that both theories have been confirmed by numerous empirical studies (Armitage and Conner 2001; Prochaska et al. 1994).

Step 2: development of initial solutions

In the second phase of this design-based research project, an educational package has been developed to be used in secondary education with a focus on contact risks as these prove to be the biggest concern for the educational stakeholders in our focus group (e.g., privacy risks and cyberbullying, De Moor et al. 2008). This package aims at heightening awareness and stimulating safe behavior. It consisted of a syllabus for the pupils and a manual for the teacher. Every course lasted 1 h, trying to satisfy the need of the teachers to limit the duration of the lessons as well as the work load. Table 3 shows how the different criteria put forward in our theoretical framework were taken into account when developing the materials.

Table 3 Design principles and how they were applied on the developed materials (Vanderhoven et al. 2014b)
Full size table

All courses followed the same structure:

  1. 1)

    Introduction. The subject is introduced to the pupils, using the summary of risks (De Moor et al. 2008).

  2. 2)

    Two-by-two exercise. Students are handed a simulated ‘worst case scenario’ SNS profile on paper and have to fill out questions about the profile together with a peer. These questions are scaffolding the pupils toward the different existing risks on the profile.

  3. 3)

    Class discussion. Answers of the exercise are discussed, guided by the teacher.

  4. 4)

    Voting cards. Different statements corresponding to the different contact risks are given. Students agree or disagree by raising green or red cards. Answers are discussed, guided by the teacher.

  5. 5)

    Theory. Some real-life examples are discussed. All the necessary information is summarized.

Step 3: Evaluation research

In a third phase, five different implementation, evaluation and revision cycles were completed to ensure that the packages reach the intended goals. The results of these five implementation studies are described in the following paragraphs and summarized in Table 4.

Table 4 Summary of the design, the analysis and the quantitative results of the five intervention studies in the second step of the design-based research procedure
Full size table

A summary of how the sequence of the intervention evolved based on these results is presented in Fig. 2.

Fig. 2
figure 2

Summary of revisions of materials in different iterations

Full size image

Study 0: Implementation and evaluation of the materials’ first version

Quantitative results showed that the awareness of contact risks on SNSs increased when the pupils were involved in the course, compared with those of the control group (see Table 4, Vanderhoven et al. 2014b). Although the courses did not influence their attitudes, nor their behavior, differences could still be found in the answers to the open questions: in the experimental condition significantly more pupils reported having changed something on their profile (17 vs. 7 % in the control condition), mostly their privacy settings (75 %) and personal information (14 %; including contact information). However, it should be noted that 83 % of the teenagers reported not having changed anything (Vanderhoven et al. 2014b). It was therefore concluded that the initially developed course has a significant impact on the awareness of the risks on SNSs, but that it did not influence the attitudes and only had a limited impact on behavior.

First revision of materials

Because of the lack of influence on attitudes and behavior, several aspects of the intervention have been examined more closely. Observations made during the courses were analyzed and comments of teachers on the intervention were taken into account. For example, it was remarkable that during class discussions and the voting game, ‘popular’ kids raised their voices and claimed that there is nothing wrong with posting risky information. As a result of this observation, one of the theories of behavioral change which was put forward in the theoretical framework was studied more closely. The theory of planned behavior (Ajzen 1991) states that behavior is determined by the intention to execute this behavior, which is, in turn, determined by the social norm—described as the social pressure people experience to behave in a particular way –, the perceived behavioral control and the attitude toward the behavior. Following this theory and the fact that teenagers are particularly sensitive to peer pressure (Sumter et al. 2009), it could be hypothesized that the ‘social norm’ can have an important impact on pupils’ behavior. Because of the opportunities SNSs offer when sharing information—e.g., communicating (Pruulmann-Vengerfeldt and Runnel 2012) and creating an online identity (Hum et al. 2011; Madden and Smith 2010)—risky behavior could be stimulated between peers and peer pressure could prevent behavioral change after the intervention.

Because of this observation, the educational materials were adapted. By increasing the time for individual reflection during the intervention and by decreasing the ‘peer time’ in which pupils can be influenced by their classmates, a larger impact on attitudes and behavior was targeted. Specifically, the two-by-two exercise was replaced by an individual task, where pupils had to answer the questions accompanying the simulated profile on their own. Afterwards, there was a class discussion. The same adaptation was applied to the voting game. While previously pupils had to raise cards, so that everyone could see each other’s cards, pupils now had to reflect on the statements individually, before discussing them with the whole class.

Study 1: Implementation and evaluation of the revised materials

The revised materials were implemented in an authentic classroom setting. Both the course with individual reflection as well as the course with collaborative learning had a positive impact on the awareness of contact risks, compared to the control condition. However, with regard to teenagers’ attitudes and behavior, a change could only be observed when the course gave the opportunity of individual reflection, compared to the control group. The course with an emphasis on collaborative learning did not influence the experimental group any differently from the control group when it came to attitudes or behavior (see Table 4, Vanderhoven et al. in press).

When analyzing the answers to the open questions, it was found that in both experimental conditions more pupils changed something on their profiles after the intervention than in the control condition (13 and 17 vs. 7 %). Changing privacy settings and modifying the personal information on the profile page were the most commonly reported changes (57 and 32 % respectively). To conclude the second evaluation study, it can be stated that more time for individual reflection is beneficial to the outcome of the intervention.

Second revision of materials

Although the results of the second implementation and evaluation study had already proven to be more promising than those of the first study, there was still room for improvement. Once more, the intervention and the remarks of teachers, pupils and observers were submitted to a more in-depth analysis. It was striking to find that the simulated worst-case scenario profile in the course contained so many risks that it proved to be less realistic. Pupils seemed to dissociate themselves from the profile and could therefore have felt as if the risks could not be applied to their own profiles. Therefore it can be argued that the exercise does not meet the standards of an authentic setting as described in our theoretical framework (see Table 1). To counter this argument, the materials were adapted. The exercise where pupils had to answer questions about the simulated profile on paper, was changed so that they had to answer the questions about their own profiles on a computer. However, this also implies that the profile held fewer risks than on the ‘worst-case scenario’ profile, making scaffolding toward all risks more difficult. For practical reasons (the availability of computers and several remarks of teachers about the narrowly timed course), the exercise with their own profiles was given as a homework task that needed to be completed right before the course took place and after the pretest was filled out.

Study 2: Implementation and evaluation of the revised materials

It was found that although the simulated profile was indeed judged as a non-realistic profile, the quantitative results of the study showed that there was no difference in impact on the awareness, attitudes and behavior of the pupils between the two conditions and therefore that there was no added value of making the context more authentic by using their own profiles (see Table 4, Vanderhoven et al. 2015). Qualitative results even showed that in the control condition more students reported to have learned something about sexual solicitation, one of the contact risks tackled in the course, probably because it was easier to scaffold toward this risk (Vanderhoven et al. 2015). Therefore, it was opted to keep the simulated profile in the package.

Third revision of materials

The Theory of planned behavior (Ajzen 1991) supported a further improvement of the materials. In the second study, it was found that the social norm has an important impact on the behavior of the pupils involved, as shown by an increased influence when the possibilities of peer pressure were decreased. In light of these results, it is interesting to note that next to peers, parents play an important role in the lives of adolescents. Parents are often thought to be primarily responsible for the moral socialization of the child (Maccoby 2007) and are seen as important actors in the education about online risks (Marwick et al. 2010; Pasquier et al. 2012; Safer Internet Programme 2009). Moreover, encouraging positive relationships, for example between parents and children, is one of the characteristics of effective prevention campaigns put forward in our theoretical framework (Nation et al. 2003). Therefore, while peer pressure negatively influences the effectiveness of the intervention, parental involvement in school interventions could have a positive influence on the effectiveness of the intervention.

Consequently materials were adapted in order to increase parental involvement. One of the possibilities to involve parents, as described by Berkowitz and Bier (2005), is to involve parents as clients, so that the school operates as a resource for the parents by organizing trainings in the topics of interest. Because of the rapid development of SNSs, it is found that many parents lack the skills to guide and support their children’s Internet use (Livingstone and Bober 2004). Therefore, training in Internet-related skills and literacy is necessary, not only for teenagers but also for parents. That is why an information evening for parents was coupled to the materials, involving the parents in the intervention as clients. The parental evening took place in the week before the course was given to the pupils and after the pretest had been filled out.

Study 3: Implementation and evaluation of the revised materials

Since only 50 parents showed up at the parental evenings, and only 19 of their children filled out both the pretest and posttest, it was difficult to interpret quantitative impact results (see Table 4). Still, the qualitative results of the study show that the information evening was effective in increasing the skills and literacy of the parents. Moreover, 53 % of the parents who attended the information evening gave their children some information about the risks on SNSs and how to behave more safely after the information session (Vanderhoven et al. 2014d). Therefore, it can be concluded that involving parents is effective at least to some extent. However, since only 15 % of the invited parents attended the information evening, simply organizing information evenings may not be the best way to involve all parents. Although the attending parents were satisfied with the information, and indicated to have learned a lot, there is no way to measure the awareness, the Internet literacy and skills of the parents who did not attend the evening. Analyzing the characteristics of the attending parents points to one of the main challenges of increasing parental involvement, namely involving all parents and not only those who are already involved (Reynolds, 2005).

Fourth revision of the materials

Following the third study, other methods to involve parents were considered. As stated by Berkowitz and Bier (2005), involving parents as clients by giving workshops or information is not the only way in which they can be involved. The authors also discuss the possibility to involve parents as partners in education. This active approach may be better suited to reach all parents, and may have a more positive influence on teenagers’ behavior on SNSs. Therefore, the materials were adapted, so that the homework task had to be done together with the parents. That is, while pupils answered their questions individually, the same questions needed to be answered by the parents and a few extra questions needed to be answered by the pupils together with their parents, such as: On which questions did you have the same answer? Where did you/did you not agree? This way, all parents would be actively involved in the intervention.

Study 4: Implementation and evaluation of the revised materials

Results show that both interventions had a significant impact on the awareness about contact risks on SNSs. While there was no difference between conditions with regard to the impact on attitudes, boys behaved more safely in the experimental condition. Although the girls of both conditions posted less personal information on their SNS profiles after the course, only the intervention where parents were involved showed a beneficial impact on boys (Vanderhoven et al. 2014a).

Step 4: Reflection to produce design principles

When starting the current study, several theoretical frameworks were put forward. In light of the results found in the intervention studies, these frameworks needed to be reinterpreted. To obtain our research goal, we deduced four contextually sensitive design principles for developing educational materials about the risks on SNSs. In Table 5 it is summarized how these principles are reinterpretations or adaptations of the initially proposed design principles.

Table 5 Initial design principles compared to the revised design principles
Full size table

Principle 1: Time for individual reflection is critical

Individual reflection during the course decreases the possible negative impact of peers promoting risky behavior. It gives pupils the time to reflect upon how they feel about things themselves, and to reflect upon possible negative future consequences of their actions, before being influenced by their peers. This time for individual reflection appears to be critical, to assure an impact of the intervention on unsafe behavior on SNSs.

Principle 2: simulated digital environments are sufficient

Although previous research shows that it is important that the learning context maintains the complexity of the authentic context (e.g., Snowman et al. 2008), educational materials about the risks on SNSs do not need to involve technology nor real SNS profiles to obtain the goals of increasing risk awareness and changing unsafe behavior. As the integration of technology in interventions is often still challenging (Hohlfeld et al. 2008) and as it is easier to control the progress of the course with fixed simulated materials, this lightens the burden for teachers.

Principle 3: Involving parents in the intervention is beneficial

Efforts to involve parents in the intervention have a significant added value. However, it is critical to find a way to involve all parents, and not only those who are already involved in an active way in the intervention. Therefore, a homework task that needs to be completed in collaboration with the parents, is proposed as a suitable solution.

Principle 4: A short-term intervention is sufficient

Teachers often complain about the workload they experience. While media literacy and online safety are often part of the compulsory program (Safer Internet Programme 2009), teachers often feel as if this is yet another ‘extra’ that is added to their already fully loaded teaching program. It is therefore interesting that an intervention including a homework task and a 1-h course appears to be sufficient to obtain the proposed goals.

Conclusion and discussion

During the first phase of the research, it was found that while teenagers face certain risks on SNSs, and school attention for the topic is beneficial, existing materials needed to be adapted and evaluated. Consequently, new materials have been developed in a second phase of the study, based on different theoretical principles. Thirdly, five iterative cycles of implementation, evaluation and revision revealed important characteristics of effective educational materials that both increase the awareness about risks on SNSs and decrease unsafe behavior. Based on these five iterations a final effective practical solution was developed and four context-sensitive design principles were formulated.

When evaluating the design-based research approach, several typical characteristics were shown to be important. For example, it was the iterative nature that enhanced the effectiveness of the materials, while the collaboration among researchers and practitioners made it possible to find a balance between the teachers’ needs and the theoretical design principles (i.e., a short-term intervention which satisfies needs of teachers appears strong enough to have an impact), which maximizes the possibilities for dissemination.

Moreover, the theoretical design principles that were formulated have a predictive value to inform future designers. This predictive theoretical contribution is often lacking after the development of interventions (McKenney and Reeves 2013). We should however acknowledge the sensitivity to the specific context of the design principles, which jeopardizes the external validity of the implications. Therefore, future research can shed light on the specific contexts in which these design principles are valuable. For example, it has been stated that individual reflection is especially important in this context, since the undesirable unsafe behavior on SNSs is strongly reputation-related. This means that the same principles may also work for different types of behavior, typically tackled in other prevention campaigns, such as smoking, drug abuse or aggressive behavior. Ideally, an experimental manipulation comparing two of these interventions, with and without collaborative learning, would demonstrate this. The same holds true for all other context-specific design principles that were put forward: future research should reveal the generalizability of these principles. This further research can lead to a sound theoretical base for the development of a broader intervention field, such as prevention or online safety, a result of design-based research that has been shown before in the field of computer-supported collaborative learning (McKenney and Reeves 2013).

A second challenge in design-based research is the fact that it is difficult to know when (or if ever) the research program is completed. In the current research, it was chosen to conduct five iterations to cumulate knowledge and to improve the design, although there is always room for upgrading (Anderson and Shattuck 2012). Several more iterations could have been conducted, possibly even increasing the impact of the intervention. But when can one decide a design is good enough for the research to be finalized? In this case, the question whether the design was sufficiently effective got even more complicated given the nature of the intervention: the goal was to increase awareness and to stimulate safer behavior, which is not an easily measurable outcome with an easily defined threshold. Yet, as we observed a significant increase in awareness and safer behavior was obtained with a significant amount of pupils, we decided the intervention was sufficiently effective to finalize the design-based research.

This leads to another limitation of design-based research, namely that it is very time consuming. The current research, including all eight studies, was conducted during three consecutive years. That is why only a short-term impact has been measured in the different studies of this research. This holds important consequences for the interpretation of the results. For example, the interventions may have a delayed impact on attitudes and behavior, making it impossible to completely observe the impact in the posttest scores measured immediately after the intervention. While the observed immediate impact on attitudes and behavior is desirable, future research using a longitudinal approach could be interesting, not only to find out if the materials have a delayed impact but also to find out whether or not the impact of the intervention will last.

The nature of design-based research also entails a sensitivity to culture which has not yet been discussed in this paper. It needs to be taken into account that the series of studies have all been set up in the same cultural and educational context. Although the children who took part in our studies are average users of SNSs regarding the risks they face and the coping strategies they use (d’Haenens and Vandoninck 2012), the impact of SNSs on teenagers should be linked to cultural differences within and between societies. Furthermore, this study chiefly resulted from a formal education perspective. Non-formal and informal educational contexts (youth organizations, cultural settings, …) could be included in future studies. Since the perceptions and behavior of teenagers can change rapidly, future studies should adopt a longitudinal and more intercultural perspective.

Finally, it should be noted that it is difficult to evaluate the impact of the design-based research method in general. Although the method effectively led to practical solutions and design principles, confirming the promising results of the review of Anderson and Shattuck (2012), McKenney and Reeves (2013) already emphasized that it is hard to evaluate the final practical impact of these materials. Still, the contribution of the current design-based research strengthens the theory that the research approach effectively leads to both theoretical and practical solutions.

In conclusion, this research has successfully led to four validated design principles for the future development of educational materials about the risks on SNSs. Moreover, by illustrating our complete design-based research procedure and by explaining how the eight reported studies are related to one another and influence each other, helping to pursue a common goal, the added value of the design-based research procedure has been demonstrated.