Keywords

1 Introduction

1.1 Introduction: Science, Technology, Engineering and Math (STEM) Skills Deficits

Transformed by advances in recent decades in computer and Internet communications technologies (ICT) , our interconnected and networked world is dependent upon knowledge in science, technology, engineering, and math (STEM) disciplines. Developments in ICT technologies have also precipitated significant change in the processes and systems of non-STEM workplaces. There is an increasing, yet unmet, demand for workers with expertise in STEM content knowledge and competencies, particularly those associated with creativity, invention, and complex problem-solving (Gmür & Schwab, 2014; United States Department of Commerce, 2012a; U.S. Department of Commerce, 2012b).

There is global concern about the deficit of skilled STEM workers, a perplexing problem because knowledge and activities in STEM fields are directly linked to nations’ capacities to compete (Bosworth, Lyonette, Wilson, Bayliss, & Fathers, 2013). Numerous studies over the past decade have underscored the essential nature of STEM skills for U.S. competitiveness and innovation , especially in the context of a global marketplace (Congressional Commission on the Advancement of Women and Minorities in Science & Engineering & Technology Development, 2000; U.S. Department of Commerce, 2012b).

The shortfall of skilled STEM workers is a major concern for the United States. It is the subject of national debate and study to determine the causes of the deficit with the aim to understand, and ultimately to create, a solution. According to national studies, the problem begins in the preparation of potential workers, in the so-called STEM pipeline, and includes issues of quality, access as well as student motivation and engagement:

Despite the clear demand for STEM talent by domestic employers , the U.S. is failing to produce an ample supply of workers to meet the growing needs of both STEM and non-STEM employers. The existing STEM pipeline leaves too many students without access to quality STEM education, and without the interest and ability to obtain a degree or work in STEM. (U.S. Congress Joint Economic Committee, 2012a, p. 3)

The reasons for the deficits in STEM workers are complex and varied, which impedes a quick resolution. Particularly concerning, is the lack of diversity of workers in STEM fields of practice. Substantial and persistent achievement gaps in STEM and other critical areas for some underserved youth perpetuate this problem. The achievement gaps of Black and Hispanic students, in particular, must be ameliorated for increased minority participation in the STEM workforce (Gonzalez & Kuenzi, 2012; Ito et al., 2013; U.S. Congress Joint Economic Committee, 2012a).

A concern with far-reaching repercussions, especially for women and minorities, is the lack of student engagement, associated with greater academic achievement, in STEM activities. Underdeveloped student characteristics such as persistence-at-task, motivation and the effective use of metacognition (Fredericks, Blumenfeld, & Paris, 2004) as well as access to high quality science education, compound the problem. These concerns, along with the lack of successful role models in STEM careers, has impaired the critical formation of STEM identities for many learners. Without this essential component, that is, students’ beliefs and self-perceptions of their own capacities for success in STEM areas, students’ pursuit of education in STEM disciplines, and ultimately, the pursuit of STEM careers is adversely impacted. For young children, discovery learning is so important; children learn through exploring and testing ideas. Using digital badges as a pedagogy, educators could foster similar learning strategies in youth by fostering the ideas of learning through trial and error, persistence and productive failure.

1.2 Why Digital Badges?

Learning is an any-time, anywhere activity, occurring spontaneously in the context of a digitally-mediated and facilitated world (Fontichiaro & Elkordy, 2013a). Digital badges have been proposed as a system to recognize and communicate achievement in a variety of learning contexts, particularly informal frameworks. As such, digital badges have the capacity to bridge formal and informal learning environments and to make learning in each context visible.

The development of the concept of digital badges is an outcome of a convergence of forces: a changing global work force, an evolving educational landscape and the rise of online learning resources, particularly open resources and open education. One of the most compelling reasons the idea of digital badging and micro-credentialing is gaining traction is the need for new knowledge and skill sets to be somehow quantified and communicated. As the workplace evolves, new methods to assess, measure and competencies and transferable skills have been necessitated by the need for workers to participate in on-going professional development and to periodically retool as skill sets become obsolete. This kind of job-related learning , often occurring in informal contexts, has neither been measured nor communicated systematically. However, the proliferation of the culture and practices of life-long, life-wide learning in both formal and informal contexts compels a solution to acknowledging, scaffolding and communicating this knowledge. It may be possible for digital badges, functioning as micro-credentials, to bridge learning contexts, recognizing and communicating competencies acquired in both formal and informal environments.

Digital badges are aligned with the idea of competencies or skills-based learning and the measurement of informal learning which can be particularly effective for STEM content knowledge as well as practices. The increase of informal, out of school learning experiences for pre-college students is recommended, particularly for women and minority students who remain underrepresented in STEM disciplines (Congressional Commission on the Advancement of Women and Minorities in Science & Engineering & Technology Development, 2000; Democratic Staff of the House Committee on Science, Space, and Technology, n.d.; National Science Foundation, n.d.). Furthermore, digital badges support recent recommendations to support evidence-based approaches in STEM education (Federal Coordination in STEM Education Task Force, Committee on STEM Education, & National Science and Technology Council, 2012).

Gibson, Ostashewski, Flintoff, Grant, and Knight (2013) called for a research agenda on digital badges examining “several new affordances for education that need additional research…and the impact of digital badges in education on the psychology of learning” (p. 7). They voiced a concern articulated by badge skeptics, specifically about the possibility of digital badges to replace “intrinsic motivation to learn.” They pose the question “…would that be a bad thing if they did?” (Gibson et al., 2013, p. 7). For educators working with youth learners of secondary school age, these affordances can be leveraged to open the possibility of connecting student learning in informal and formal contexts and to personalize learning through differentiation of learning processes and products.

In addition to acknowledging self-directed and self-motivated informal student learning , when purposefully designed and implemented, digital badges can be powerful pedagogical tools to promote student engagement, motivation in formal learning contexts. By making learning visible through criteria which clearly articulate learning targets, competencies can be effectively scaffolded and become clear to the learner. These attributes of instructional design reflect best practices in assessment (Stiggins & Chappuis, 2011).

Digital badges can be effective in competency-based models of education. They are flexible tools which can be used to promote higher order thinking skills and facilitate the assessment of discrete skill sets or competencies. The evidence-based model of digital badges is particularly suitable for promoting experiential learning and performance assessment , critical to STEM education and outcomes. When learning targets are clear and the products to demonstrate understanding or competency in objectives are flexible, students are empowered to take ownership of their own learning and critical components of metacognition, such as self-regulation, are fostered. Digital badges may be particularly effective with middle and high school students because they intuitively understand the social capital and idea of currency of digital badges.

Could digital badges effectively scaffold learning in STEM content? Would students find them motivating? How can digital badge learning trajectories be implemented in a formal learning context to incorporate out of school learning? This research study explored the careful application of rigorously designed digital badge learning trajectories for STEM learning in an underserved population to explore these questions.

2 Literature Review: STEM Practices

This abridged literature review section focuses upon: 1) STEM learning and assessment in a digital age 2) digital badge for learning in formal and informal contexts, 3) motivation and learning, and 4) digital badges and motivation.

2.1 STEM Learning and Assessment in a Digital Age

The idea that learning or meaning is constructed through and within social contexts was initially proposed by Vygotsky. He theorized that learning occurs when individuals internalize concepts mediated through spoken language. Vygotsky (1967) postulated that individuals create meaning through the processes of social discourse by internalizing language as individual thought. Since then, the social constructivist learning theory has been modified and adapted by educational theorists including Jerome Bruner (2006), Brown and Adler (2008), and Etienne Wenger (2000). Learning is considered as an individual, cognitively-based activity which is socially-mediated; meaning is made through socio-cultural contexts and interactions with others.

We increasingly live in an age of convergent media, where production, sharing, and participation are the norm and expectation, at least for our youth. There is fluid group formation and cognitive, social and linguistic complexity, all embedded in popular culture (Gee, 2010, p. 14). Various theorists have written about the role of language, learning and cognition. Within these socio-cultural contexts, when learning occurs, it is contextual. In terms of the theory of situated cognition, learning is embodied, and knowledge and intelligence are contextual and distributed across tools, technologies and groups (Gee, 2010). Situated cognition emphasizes practices of collaboration, using tools and technologies. The concept of situated cognition is consistent with Social Constructivist theories of learning, which postulate that meaning is constructed by individuals within a larger social context, and that meaning is interpreted using memory and existing schema. In response to socio-cultural changes, the Connected Learning Model (CLM) has been proposed by researcher Mimo Ito and others, in order to describe how learning occurs in these connected learning environments (Ito et al., 2013). The CLM builds upon earlier models of social learning in its emphasis upon participation, shared purpose and peer culture while adding aspects of digitally mediated culture such as openly networked and interest.

In practice, practitioners and learning theorists have integrated the principles of social-constructivist learning theory through strategies which include class discussions, collaborative learning, or reciprocal teaching (Brown, Collins, & Duguid, 1989). The result is the acquisition of new learning, either directly or vicariously (Bandura & McClelland, 1977). Etienne Wenger (2000) called groups of learners collaborating and working together Communities of Practice and described “Learning [as] the engine of practice, and practice is the history of that learning” (p. 96). This concept of learning within a community of practice, which leverages experiential learning as well as the premise that learners move along a trajectory from novices to experts, is highly compatible with the shifting view of STEM pedagogy. The notion that learners must approach STEM content as practitioners, first as novices, to authentically experience science, is the foundation of the NGSS performance standards and cross-cutting concepts. The learner-as-practitioner model is a distinct shift from the inquiry-based learning model which has been pervasive in formal educational contexts. In informal learning contexts, the learner-as-doer and maker-of-trials as long been an effective learning model which fosters curiosity and normalizes failure as an integral aspect of design and problem solving.

2.2 Digital Badges for Learning in Formal and Informal Contexts

Core concepts of the new digital badge movement are ideas of equity and transparency as well as recognition learning in diverse contexts. In many ways, these concept mirror, and are inspired by, the entrepreneurial and open spirit of the Internet itself. Much of this learning occurs in informal contexts and is currently neither recognized nor communicated effectively.

A report published by the European Centre for the Development of Vocational Training (2001), which reviewed various European initiatives to quantify and communicate the outcomes of informal learning, is representative of the growing, world-wide interest in the premise. In Making Learning Visible: Identification, Assessment and Recognition of Non-Formal Learning in Europe the author discuss the importance of this issue (Bjornavald, 2001). It is necessary to make learning, which takes place outside formal education and training institutions, more visible. Non-formal learning is far more difficult to detect, evaluate and communicate. This invisibility, is increasingly perceived as a problem, impacting competence development at all levels, from the individual to society as a whole (Bjornavald, 2001, p. 11). Furthermore, the author urges that “…competencies have to be made visible if they are to be fully integrated into a broader strategy for knowledge reproduction and renewal” (Bjornavald, 2001, p. 21).

The use of a system to assess and promote learning in STEM knowledge and practices has potential for a variety of reasons. Despite their importance, many of these skills remain untaught, or they go unmeasured through the current, often standardized, processes of assessment in formal educational environments . Furthermore, the persistent lack of alignment between the goals and outcomes of educational systems and the requirements of the workplace has contributed to the paucity of skills in some areas, and overabundance in others. A new, more effective way of assessing learning is essential for twenty-first century learners.

Making competencies visible: Boundary objects. In his joint report with the OECD, Werquin (2010) asserted that “Recognition generates four different types of benefits” (Werquin & Organisation for Economic Co-operation and Development, p. 8) in recognizing skills learned in informal environments: shortened time for acquisition of qualifications; more effective deployment of human capital; and increased coordination between employment and individual employee talents. Life-long learning increases educational and social benefits for the learner, fostering equity and improved access to education and employment, particularly for disadvantaged groups. Life-long learning provides a “…psychological boost to individuals by making them aware of their capabilities” (Werquin & Organisation for Economic Co-operation & Development, 2010, p. 9).

To meet the demand for new knowledge, new learning and assessment paradigms must be developed in socio-cultural contexts . The use of digital badges for scaffolding, assessing, and communicating learning, within connected contexts, is one possible solution. As such, digital badges can function as boundary objects, i.e. objects which exist in different contexts and have context-specific properties, but share enough of a framework to be useful as a construct which traverses these limits or boundaries (Rughinis, 2013; Star & Griesemer, 1989). Wenger (2000) description of a boundary object, a way of translating the practices and social capital of one community to other, dissimilar communities, suggests digital badges are almost ideal for this purpose (as cited in Halavais, 2012, p. 367).

2.3 Motivation

“Motivation is a theoretical construct used to explain the initiation, direction, intensity, persistence and quality of a behavior, especially goal-directed behavior” (Brophy, 2010, p. 3). In K-12 environments the behavioral view has proliferated; it is visible in attempts to modify behaviors through reward systems, grading, strategies to gain student compliance, and negative consequences for breaking rules or failing to comply with targeted behaviors.

Extrinsic motivation can be a major concern for educators. Misapplied, extrinsic motivators can act to demotivate learners and create false expectations of reward which may impair intrinsic motivation (Hattie & Timperly, 2007). Motivation is a factor associated with self-concept and academic achievement. It is an important factor for minority students including Arab Americans and African Americans in self-esteem and positive identity formation (Kovach & Hillman, 2002). Malone and Lepper (1987) have proposed a taxonomy of intrinsic motivations, which they suggest “make learning fun” (p. 223). The concepts, including curiosity, control, and challenge, are often incorporated into game-based learning, where they function powerfully to engage learners to the point of flow, an optimal state of intrinsic motivation when participants are motivated and engaged (Csíkszentmihályi, 1990).

2.4 Badges and Motivation

The idea of using badges in education remains controversial, with advocates and detractors having strong opinions on either side; some commentators are concerned that badges are an extrinsically motivating behaviorist strategy to reward learning, which will lead to badge acquisition as the goal, versus the learning goals themselves. Dr. David Goldberg’s response below acknowledges that this may superficially and sporadically transpire, but that in the process of learning and badge acquisition, intrinsic motivations do occur:

In the Kantian vein, then, we could conclude that badges without effective learning would be empty, even useless; while learning without a badging system that embeds an assessment capacity capable of motivating further learning—both more and deeper—would be missing an opportunity to draw into the lure of learning some, if not many, of those we otherwise are in peril of losing. And that’s a good, perhaps even in itself. (Goldberg, 2012)

Regarding the idea of motivation itself, Professor Goldberg, cofounder of the HASTAC organization, and co-sponsor of Digital Media and Learning Competition, continued:

…the deeper point about badges is that where they work, they work always within contexts that socially support them and where their users are invested in their significance. They do not work for everyone, as motivations or modes of recognition (2012).

Digital media expert and cultural commentator, Henry Jenkins expressed concern that youth learning informally may be alienated by the formalistic processes of badge acquisition, before they have a chance to exert ownership over the knowledge they are acquiring. Furthermore, he noted that this issue would grow when the system of digital badges moves into a global phenomenon, when cultural contexts will mediate the meaning and value of badges (Jenkins, 2012).

One of the major concerns and advantages of using digital badges to recognize learning is the pivotal issue of motivation which is closely associated with engagement and academic achievement (Steinmayer & Spinath, 2009). Skeptics are concerned that badges are a purely extrinsic reward system, which will result in learners working hard to collect badges as rewards (equivalent to good grades or gold stars), rather than learning.

Digital badges, however, are an educational intervention adapted and derived from the world of online gaming where they are a widely recognized symbol of achievement. According to Ostashewski and Reid (2015), “Emerging from the intersection of games cultural, visuals on the Internet, and the traditional and historical uses of badges and medals, the digital badge is an online visual representation of an accomplishment, skill or award” (p. 187). Educators have been interested in understanding how digital game elements engage and motivate participants so effectively to persist since the work of early commentators such as Marc Prensky (2001) and researcher James Gee (2003). Digital badges are at the intersection of “gamification” (using game elements) and use of the underlying mechanisms of the games. According to Deterding (2012):

Recently, the lessons to be learned from good video games have been extended beyond the literal design and use of games for learning to the use of game design principles to conceive of a different way to organize instruction, turning formal education itself into a game-like experience. (as cited in Fishman et al., 2013)

2.5 Summary

Conceptually, the idea of awarding badges as an outcome, or in combination with a performance assessment in an open, potentially socially mediated and authenticated system to assess, guide and recognize informal learning is deeply grounded in current theories of how people learn, including situated cognition and motivation. For example, the fact that the performance benchmarks are readily available propagates self-regulated learning and fosters the development of metacognition on the individual level. It also facilitates discussion and inquiry which are the foundation of participatory culture and at the heart of knowledge making in a social constructivist manner. The idea of badging systems for assessment is aligned with the concept of participatory cultures. It is also powerfully aligned with theories of motivation in learning. Digital badges leverage many of the strengths of digital media, participatory cultures, ICT as well as foster mastery learning and the formation of positive STEM identities.

3 Case Study Details

3.1 Overview of the Study

A mixed methods study was conducted to assess the impact of a digital badge intervention for STEM learning in a formal secondary learning context. The study explored the perceptions and attitudes of participants regarding the use of digital badges and their learning trajectories for learning, including pedagogical aspects used in implementation such as teaching strategies and feedback practices. An exploratory approach is appropriate because of the emergent nature of research in the use of digital badges in formal education contexts. “Exploratory studies are quite valuable in social science research. They’re essential whenever a researcher is breaking new ground, and they almost always yield new insights into a topic for research” (Babbie, 2010, p. 93). A mixed methods design was selected due to “a major advantage of mixed methods research is that it enables the researcher to simultaneously ask confirmatory and exploratory questions and therefore confirm and generate theory in the same study” (Teddlie & Tashakkori, 2009, p. 20). The objective of the research was to explore how digital badges, used as an educational intervention with specific pedagogical practices, may impact the learning of STEM content and practices in the secondary school sample of underserved students.

3.2 Significance of the Study

Although it has been widely assumed that the use of digital badges impact learning, both positively and negatively, there is a lack of empirical data to measure effects. Essentially research “related to incentives, motivation, and learning on badge-based learning …in its infancy” (Bowen & Thomas, 2014, p. 25). In particular, the premise that digital badges will affect participant motivation has been repeatedly asserted, but “there is little research that examines how badges interact with student motivation” (Abramovich, Schunn, & Higashi, 2013, p. 218).

The findings of this study contribute to the emerging knowledge base about the use of digital badges systems for learning in secondary learning contexts. This research also contributes to the practical aspects of designing learning trajectories, which incorporate sound, research-based principles of teaching, learning, and motivation. In addition, the use of digital badges may provide scaffolding, tools for flexible assessment and may propagate the deep learning of key STEM concepts in connected learning contexts.

The ultimate goal of this work was to inform educational practitioners and policy-makers in addressing authentic problems of practice–to enhance learning of STEM knowledge, concepts, and practices to all youth, particularly learners in underserved communities.

3.3 Rationale and Purpose

The objective of this research was to explore the use of digital badges as an educational intervention in the learning of STEM competencies aligned with NGSS Practices, in a specific, secondary school context. Student characteristics important to effective learning and a positive STEM identity including motivation, persistence, self-efficacy, and task value were measured. The digital badges designed for the study were standards-aligned with robust learning trajectories articulated through badge criteria, and suggested assessments of learning. They were designed to scaffold the acquisition of STEM content knowledge, practices and habits of mind. Data describing the learning environments, which could affect program implementation were collected through both quantitative and qualitative measures and then analyzed. These data also included teacher and leadership factors.

Although digital badges have been used successfully in other technology-mediated instructional systems such as educational games, understanding how (if) digital badges function as an intervention for learning and instruction is largely unexplored. The “nascent nature of STEM badges ,” and in light of the fact “to date, few journal articles focus specifically on badges,” “the potential efficacy and methods of application of digital badges in K-12 populations are currently unknown” (Riconscente, Kamarainen, & Honey, 2013, p. 2). Funded by the National Science Foundation to explore “Badge-based STEM Assessment,” Riconscente et al. (2013) reported that there are “novel affordances badges bring to the current context of STEM learning,” with “potential …for supporting deeper student engagement, substantive opportunities for learning STEM content, and a greater transparency of underlying assessment criteria,” (Riconscente et al., 2013).

3.4 School Context and Learners

The digital badge intervention programs were implemented over a course of 3–6 weeks during the 2013–2014 school year, in a charter school system (publically funded, independent schools) in the Midwest with a large English as a Second Language (ESL) population; Arabic is the primary language spoken in the home. The site was invited to participate in the study because of its previous adoptions of innovative instructional practices, and its student population which is socio-economically, linguistically and culturally diverse. According to Fall 2013 data, the school has a free and reduced school lunch rate of 87 % (Center for Educational Performance and Information, n.d.).

The total number of student participants was 72, with 20 students in 7th grade, 32 in 10th grade, 2 in 11th grade, and 18 in 12th grade; data for the 11th grade students was excluded from the analysis because of the small number of participants. Two teachers successfully completed the entire digital badge study. The units of analysis for the study were: 1) individuals (students and teachers), and 2) groups of individuals interacting in learning contexts (e.g. classes or groups of students). The digital badge interventions took place in a social studies class (7th grade) and in two high school math courses (algebra and Business math).

3.5 Methodology

3.5.1 Research Design

There are a wide variety of mixed methods designs. They are often categorized according to the purpose of the research, the methodological emphasis, or the sequence of methodological integration. An evolving field, mixed methods does not yet have an established nomenclature (Teddlie & Tashakkori, 2009). This study was a concurrent or parallel (Teddlie & Tashakkori, 2009), or concurrent triangulation (Creswell & Plano Clark, 2007) mixed methods design. The study comprised quantitative analysis of survey data and thematic analysis of qualitative data, collected from a variety of sources. Qualitative data were collected from a post-program, semi-structured interview (focus group), personal communications, open-text survey questions, and artifact analysis of student work. This design is used to confirm and corroborate findings, with the data being integrated during the interpretation phase (Creswell & Plano Clark, 2007).

A mixed methods research design, which combines survey data with qualitative methods, is consistent with strategies advocated by researchers working with mixed methods research (Andres, 2012; Creswell, 2008; Creswell & Plano Clark, 2010; Creswell & Plano Clark, 2007; Plowright, 2011; Teddlie & Tashakkori, 2009). This design is appropriate for studying the attitudes and student characteristics of participants.

3.5.2 Research Questions

A main focus of this research was the question:

  1. 1.

    How does the use of a digital badge intervention for STEM learning impact student: motivation, task value, learning goal orientation, self-efficacy, learning behaviors and strategies, including self-regulation and persistence-at-task?

3.5.3 Measures

Students responded to surveys before and after the digital badge program on their attitudes and opinions regarding STEM learning and the digital badge program. In addition, data were collected about student learning behaviors as well as their ICT and digital media use. In order to measure the construct of motivation in learning STEM skills, competencies, and knowledge, several sub-scales from the Students’ Adaptive Learning Engagement in Science Learning (SALES) (Velayutham, Aldridge, & Fraser, 2011) scale were modified and implemented as intensity scales with values of 1–20. The following SALES subscales, consisting of 4–8 items, were used as measures in both the pre- and post-treatment: self-efficacy, (learning) goal orientation, task value, self-regulation and learning behaviors. In the pre treatment, students were asked about their learning in STEM content whereas in the post treatment, students were asked about their learning in the digital badge program (which focused upon STEM content).

The pre-program questionnaire was comprised of 40 questions. In addition to the SALES sub scales, the instrument included ranking and interval items to measure ICT use, digital media use, and learning behaviors. The student post-program questionnaire was comprised of 33 questions. In addition to the SALES sub scales, items were included to assess student attitudes about the digital badges and learning behaviors used during the program. The post program student survey included additional intensity measures (from 1 to 20), ranking, interval and open text questions.

3.6 Procedures

3.6.1 Study Preparation: Digital Badge Intervention

Three digital badge series were designed to scaffold learning, provide criteria for measurement, and to establish guidelines for assessment and learning in select STEM concepts and practices. Digital badge learning targets were aligned with the NGSS standards articulated by the National Academy of Sciences in A Framework for K-12 Science Education: Practices, Crosscutting Concepts, and Core Ideas (National Research Council, 2012). The core ideas are organized into three dimensions which are recommended for integration into K-12 STEM curricula and instruction. The specific digital badge learning targets, performance tasks, and assessment criteria were developed in collaboration with a middle school science and math teacher and reviewed by several others.

The National Research Council (NRC) ’s Framework is divided into three dimensions: Practices, Crosscutting Concepts, and Disciplinary Core Ideas. Scientific and Engineering Practices (Dimension 1) requires significant proficiency in higher order thinking skills: analysis, evaluation, synthesis, and the application of tacit concepts and ideas. “The NRC uses the term practices instead of a term like ‘skills’ to emphasize that engaging in scientific investigation requires not only skills but also knowledge that is specific to each practice” (NGSS Lead States, 2013). The practices require opportunities to apply knowledge and to ultimately gain the kind of tacit professional knowledge acquired through professional practice in the field. Mastery of the practices is consistent with the idea of epistemic frames which “are described as the ways of knowing, of deciding what is worth knowing, and of adding to the collective body of knowledge and understanding of a community of practice” (Shaffer, 2006, p. 223). This is also consistent with the view of the learning as situated cognition, occurring in communities of practice (Brown et al., 1989; Lave & Wenger, 1991; Wenger, 2000). It also supports the view of “Science as a Process of Participation in the Culture of Scientific Practices” (Duschl et al., 2007, p. 29).

Eight science and engineering practices are described and defined in the NGSS framework. Due to their inherent complexity, mastery of these practices is difficult to assess in traditional, formal learning contexts. Formal learning contexts rely heavily upon standardized testing measures (Gilmer, Sherdan, Oosterhof, Rohani, & Rouby, 2011). The Practices provide suitable competencies and learning objectives for the pilot digital badge intervention. The badge criteria can include performance tasks which require mastery of concepts as demonstrated through diverse products of learning.

Three digital badge series with five levels were designed to spiral and scaffold skills development for the study: InfoMaker, Data Whiz and Data Hacker. They are aligned with the following:

  • select Next Generation Science Standards Dimension 1 Practices

  • Common Core State Standards (CCSS) in Math (Common Core State Standards Initiative, 2014b)

  • English Language Arts (Common Core State Standards Initiative, 2014a)

  • National Educational Technology Standards for Students (NETS-S) (International Society for Technology in Education (ISTE), 2007)

  • Partnership for 21st Century Learning (P21) (Partnership for 21st Century Skills, 2009) standards.

Intentionally, some competencies did not align specifically with standards because existing frameworks are not applicable and therefore alignment would result in restrictions on learning products. In particular, these include competency in higher order thinking skills, or digital media consumption and creation. The design of the badge curriculum framework incorporates the idea of spiraled curriculum (Bruner, 1976). It also includes Gagne’s theory of varieties of learning, and articulation of learning outcomes, as instructional objectives and practices (Aronson & Briggs, 1983). Furthermore, the digital badge design incorporates theory-based practices of mastery learning. The design also incorporates feedback to assess the student learning process and to gauge effectiveness of instruction (Guskey, 1996). The badge learning targets and criteria were designed by working backwards from learning targets (Wiggins & McTighe, 2005). The learning targets represented steps along a learning path or trajectory, consistent with the premise of instructional design for effective learning (Reigeluth, 1983; Smith & Ragan, 1999).

The badge learning trajectories were presented as either curriculum documentation for school administrators and teachers, or as stories in the Makewaves learning management system and digital badging platform. The stories or blog posts, were written in language accessible to target youth participants. The curriculum documentation included the following: badge overview, learning targets , badge skills, performance objectives, suggestions of evidence of achievement (examples), alignment with Bloom’s Taxonomy, standards and frameworks alignments, learning resources. The story versions of the digital badge details included an overview of what the learner must do to earn the badge, an estimate of the time necessary to complete the badge requirements, and a description of the skills to be developed through the process.

Instructional resources and supports. Program materials were developed to explain digital badge instructional processes, procedures, and goals. These included an implementation manual and other documentation, teacher resources, and supports. Training, documentation, and curated resources, in the form of dynamically generated lists or visual aids, were created and shared.

Learning management system . The Makewaves (www.makewav.es) social learning system was selected as the digital badging platform for the study. As a secure learning system (LMS) and digital badging platform, it was suitable for minor participants. A project web site was created on the Makewaves platform (www.Makwav.es/badgebox) to share study information, to organize participants into groups, and to award student and teacher digital badges.

3.6.2 Implementation : Data Collection and Analysis

The study was implemented over 3–6 weeks during which the students worked on teacher-guided or approved projects to earn digital badges. Teacher A used the badges to supplement existing coursework and worked primarily with the Data Hacker badge series. Teacher B applied the InfoMaker digital badge series for a project in social studies; students were instructed to research a problem in a country or region, then provide a solution. Students completed surveys, pre and post intervention. A semi-structured, post project focus group was conducted with the teachers.

Descriptive and inferential statistical analyses, including factorial analysis and regressions were conducted. In addition, qualitative analysis was conducted on the transcribed interviews which were analyzed for emerging themes using nVivo . Open text questions from the Student Post Survey were coded online using the text analysis tool. Artifact analysis was conducted on the student work in the 7th grade social studies class. The results are abbreviated for publication.

3.7 Findings

This study investigated the overarching research question:

  • How does the use of a digital badge intervention for STEM learning impact student: motivation, task value, learning goal orientation, self-efficacy, learning behaviors and strategies, including self-regulation and persistence-at-task?

Due to space limitations for this publication, the data below represents a snapshot of the main findings. Specifically, measures which together, comprise the construct of “motivation” are reported, and qualitative findings which describe teacher viewpoints on the digital badges as pedagogical tools.

3.7.1 Quantitative Data

Paired sample T-test analyses were conducted on pre and post measures of student attitudes and beliefs of self-efficacy, self-regulation, task value, and goal orientation. Sub scales from the SALES instrument were used, which together measure student motivation, in this case on STEM content and using digital badges for learning STEM content.

There was no statistically significant difference in measures of student goals, except for a comparison of performance goals. “In my class or program, it is important to get good grades” (Q. 32, pre-program) and “In the digital badge program, it is important to earn badges” (Q.18, post program). For analysis, the file was initially split by grade, then by gender. There was a significant grade level difference of 0.027 in the 12th grade, significant at the p ≤ 0.5 level, with the comparison of means, indicating that students responded that grades were more important. Of interest, the comparison for 7th grade resulted in an identical mean value of 16.778 (with SD of 4.0520 and 4.8210 for the pre and post measures respectively). There was a significant gender difference for boys at 0.005, which is significant at the p ≤ 0.01 level; boys responded that the digital badges were not important as grades. The girls’ response was similar, except girls valued the badges more, hence the lack of statistical significance between the means.

There was a difference in pre and post mean values for the question “What I learn is interesting” of 0.13, which is significant at the p ≤ .01 level.

3.7.1.1 Self-Efficacy and Self-Regulation Pre and Post Measures

There were several items with statistically significant differences between the pre and the post measure for self-efficacy, three of which (see Table 27.1) were significant at the p ≤ .01 level. “I can understand the content taught” has a significant pre and post program difference, significant at the p ≤ .05 level.

Table 27.1 Paired samples test: measures of self efficacy
Full size table

Several pre and post program measures of self-regulation were statistically significant to the p ≤ .01 level (see Table 27.2). These measures indicated students’ willingness to persist at task and to “concentrate” or to pay attention, which was significant to the p ≤ .05 level.

Table 27.2 Paired samples test: measures of self-regulation
Full size table
3.7.1.2 Factorial Analysis

The results of the factorial analysis suggest that the post-treatment measures of self-efficacy and self-regulation differ in composition from the pre-treatment measures. The pre-treatment factors loaded on to either self-efficacy or self-regulation, only, which is anticipated because of the use of the adapted SALES survey questions which are designed to measure these attributes.

Items loading (values of 0.7 or higher) onto factor Self-Regulation A Post reflect students’ self-regulatory and persistence in learning behaviors as self-reported in the digital badge program. In addition, students’ belief about the importance of understanding the work (self-efficacy) and earning badges in the program loaded onto this factor:

  • persistence in working when tasks are uninteresting

  • persistence in working hard when I do not like what I am doing

  • concentrating to not miss important points

  • persistence when there are better things to do

  • importance that I understood my work and the

  • importance of earning badges in the badge program.

Items loading (values of 0.8 or higher) onto factor Self-Regulation B Post describe students’ self-regulatory learning behaviors and self-efficacy regarding in the digital program as well as measures of self-efficacy about their performance:

  • I am good at these subjects

  • I can understand the content taught

  • I will receive good grades

  • finishing work and assignments on time

  • persistence even when the work is difficult

  • concentrating in class or in the program and

  • persistence in working until the tasks are completed.

Qualitative data were used for confirmatory analysis and to generate emergent theory about the use of digital badges in similar contexts. Furthermore, qualitative data provided additional insights into instructional and assessment practices, and it also described the processes of implementation.

3.7.1.3 Instructional Processes

The preparation necessary for the Digital Badge program was “minimal” (Teachers A and B). It consisted of reviewing the materials, including the badge criteria (2–3 h each), and preparing student materials: “So it didn’t take that much planning time. And again, it’s planning that you already would have done for your classes anyway.”

Teacher A had made a shift in goals for the school year and she considered the digital badge program aligned with these objectives: “I think with my class this year—and this is not just digital badges—I shifted the focus from content to skills. So I’ve tried to build skills-based assessments throughout the year, and this just kind of played into that.” Teacher B concurred, “This is probably the way I would prefer to teach, because it’s all of them doing it on their own, and figuring out that they can, and that’s cool” (Teacher A).

The digital badge program was viewed as a strategy or pedagogy: “It’s just one more strategy to get that one little cohort [hard to reach] of students on board with something.” (Teacher A).

About the authentic applications or context: “Usually they totally shut down on that stuff. But to tell them ‘I want you to work through it so you can earn this [digital badge]’ … then they are a little more persistent with that” (Teacher A). Teacher B used a different teaching strategy and an authentic context: “The biggest thing was that in InfoMaker, they had to come up with resources that they needed, and all the materials to fix their problem. And I made them be extremely specific with that. I made them come up with basically everything that they could ever possibly imagine needing: how much each thing costs. And when they really had to think about that—that was pretty tough. That was probably one of the hardest things for them, is to *really* explain what’s needed to fix their problem.”

3.7.1.4 Assessment Practices

Teacher B implemented the digital badges as part of an extensive array of formative feedback strategies (see Feedback processes, below). Of particular note, Teacher B inserted another “step” in which students self-evaluated against the digital badge criteria, before submitting work to the teacher, fostering critical metacognitive skills. Having clear badge criteria, that is, clear learning targets, enhanced the feedback process by giving students and the teacher specific goals against which to gauge performance.

Products of learning . Teacher B remarked on the quality of the 7th grade Social Studies projects which were created for the digital badges program (Fig. 27.1):

Fig. 27.1
figure 1

Teacher feedback

Full size image

There were definitely different products, and some of them were actually phenomenal. That was maybe five of them, were … incredible. And holy smokes, I can’t believe they put that much work into it. And then 10 of them were pretty dang good…

3.7.1.5 Teachers : General Observations and Comments

In comparison to the regular class work, the digital badge program provided opportunities for authentic applications: “Yeah, I have the same problem in math, too. I need you to learn the procedure, but the whole point is to apply it to real world context. … the digital badges were nice, because we are taking a real world problem, or real world data, using those procedures we use in class normally,. But there’s a whole other point to it now” (Teacher A).

About watching students as they worked through problems, Teacher B commented: “Which was kinda nice for them—and me—because I got to watch them be proud, and then they really got to be frustrated before they figured something out, and then they were really proud of it as well…”

Teacher B elaborated on the learning processes and using the digital badges to encourage students to persist at task: “… it was fun.” Teacher A also commented: “But students get the mentality that this is the kind of student that I am. And this is the kind of work that I do, and that’s that. But if you have something that they are interested in, like badging, or making it more like a game, or levelling up, then they might motivate them more to try the harder stuff”

Of particular interest, Teacher A compared the grading experience of students with job performance metrics:

I mean, it’s the same with the students. We take you and all of these wonderful things about you, and all of these interests you have, and then we bubble down a GPA letter, or number or something. As a teacher, you kind of feel the same way. You put in all this work, and now you’re just checking boxes. So is there another way to kind of supplement that, to show what you are good at?

3.7.1.6 Students

In the beginning, students were apparently confused about how to proceed, and they struggled with the format of the digital badges: “Some of them thought it was really strange, that they got to do whatever they wanted. And that…there wasn’t an obvious answer, and that they… really had to think about it” (Teacher B).

With Algebra, it’s more like we do a unit, and you take a test kind of thing. So with this being more open and independent, there was some anxiety -- that we talked about. When you kind of give them this freedom, they don’t know what to do with it. (Teacher A)

Teacher B explained:

Our students here are, I think most students…Just the way the curriculum is made in textbooks, and all that stuff. They are used to Question, Scan, Answer, Copy, Paste, and then you know, write it down. So with this… so that they were pretty confused when there wasn’t an obvious answer to things. There’s not so much creative learning that this provides.

Initially, the students were concerned about the expectations and work for the digital badges: “So there was more anxiety, and a lot more questions in the beginning. ‘How do I do this?’ and ‘How am I going to get the grade?’ and ‘How…’ this and that, but once you kind of get past that stage, I think they kind of appreciated more” (Teacher A). In Teacher B’s class, the two-step system was used, where students were asked to go through their own checklist of badge criteria before work was submitted to the teacher. “Yeah, but sometimes they were pretty frustrated, because they were positive they were done, and they weren’t.”

As the program progressed, the students enjoyed working with the badges: “I think with my students, they really got on board with the idea that this is supposed to measure things that you are good at… that you are not getting measured at school.” Teacher A described how students felt about traditional grading systems: “I guess at the high school level that there are a lot of students that feel either disenfranchised or misrepresented by their grades, or kind of the whole system, traditionally, how their academics are.” The capabilities of digital badges transcend grades. Teacher A continued: “So to tell them so to look at some of the students who are not doing well academically, but are really great with other skills, other tasks, and letting them know this is the whole point, this is for you to bring that in, a lot of them got on board with that and thought that was nice. They like the connection, the gaming, like just having fun, and earning something… that to them is outside what you would normally do in class.”

4 Conclusions

Learners languish as skills gaps widen. A paradigm shift in our educational outcomes and processes is clearly necessary. Although many questions remain about the use of digital badges to scaffold, evidence, and communicate learning, crucial conversations about learning have reached a tipping point. Globally, there is interest in acknowledging and leveraging skill sets earned in out-of-school contexts for economic growth and equity.

Digital badges with robust learning trajectories can empower and motivate learners. They have potential to foster skills and habits of mind for engaged STEM learning. Digital badges can evidence the creativity, higher-order thinking and problem solving skills necessary for STEM disciplines and careers. Youth can learn the skills and language of communities of practices through authentic learning experiences, and ultimately, through the process of acculturation, develop positive STEM identities.

The findings of the study are that standards-aligned digital badge innovation was effective in increasing student motivation in this student population. There were several statistically significant increases in measures of student self-efficacy and self-regulation. Students and teachers reported a willingness to persist at tasks to earn the digital badges as well as increased product quality and complexity, particularly in the 7th grade social studies class. Measures of student perceived competence in task completion increased in the post measure.

The majority of student participants enjoyed using the digital badge program to learn. This was particularly evident in the qualitative data, students’ written responses, and as reported by the teachers. For example, students reported that the digital badges for learning were cool, a fun way to learn , that they would like to use them again. In addition, students were interested in earning additional badges if the program were longer. The majority of students reported understanding the badge requirements usually or all the time, and also wanted the opportunity to earn more badges and if the digital badge program were longer. There were no significant differences in task value (interest), with the exception of a difference in pre and post measures of student interest, significant at the p ≤ .05 level (p = 0.13). Interest is an essential component of student engagement, necessary for academic achievement.

Learning behaviors were also impacted by the use of the digital badges. Students referred repeatedly to the badge criteria to gauge the completeness of their work. As reported by their teacher, in the social studies class students notably used the badge criteria to check their performance. Such learning behaviors, scaffolded by digital badges, promoted increased levels of self-regulation in learning, enhanced metacognitive skills and perceived competence.

The majority of students agreed that the way the badges were structured helped them learn the subject well, and 94 % of students were interested in using digital badges for learning again. Students at every level (7th, 10th and 12th grades) indicated that they could incorporate learning from other contexts into their assignments using the digital badges some of the time or more. Furthermore, a minimum of 60 % of students at every grade level were interested in where to earn digital badges for out-of-school learning.

The younger students in particular, indicated that they understood the content more using the digital badges. The 7th grade students and their Teacher (B) worked collaboratively through formative feedback using the digital badge criteria as learning (and assessment) targets. Students thought the organization of the badge criteria was helpful in the learning process.

4.1 Motivation

As a complex construct, motivation is inferred by the presence of other attributes, such as self-efficacy, choice, persistence-at-task, and interest. For this population, many of these indicators had measurable, statistically significant differences. It is important to note that the pre and post measures were comparable, but did not measure the same constructs (self-efficacy in STEM subjects versus the digital badges based around STEM content).

When students were asked what they would say about working with digital badges, the responses were positive. They spoke about how the badges were motivating, fun, make things easy, and that they were a good way to learn. Of particular interest, when students were asked what they would change about the experience of learning with digital badges, many talked about changes they would make in their own attitudes or approaches, versus the badging processes or design.

Teachers also agreed that the digital badges were motivating for students, particularly students who weren’t regularly successful with traditional assessments.

Students were able to include learning from other contexts, and liked this aspect of the digital badges.

4.2 Student-Level Factors

Despite a low income context, the students are very much interconnected via ICT. Their favorite online activities are using digital games and media, communications and social media which reflect use as consumers versus producers of digital artifacts.

Digital badges are designed to reside online, to be shared with select audiences. The student population for this study did not actively share their badges. This may be due to cultural biases against bragging, or concern for envy and a cultural/religious propensity for modesty. Students may have equated the digital badges earned in class as analogous with grades or other accomplishments, which they tended not to share.

During the digital badge program, students reported being able to integrate learning from other contexts into their assignments a substantial amount of the time. Students agreed that they would like skills and knowledge from out of school learning to count. They wanted to know where they could earn more digital badges for learning.

4.3 Learning Environment and Implementation Factors

As an instructional tool, the digital badges supported existing curriculum; Teachers A and B reported a shift in learning goals and outcomes toward learning skills or competencies, and the badge learning trajectories were aligned with this goal. The digital badges aligned with an instructional goal for the school year to emphasize transferable skills or competencies (Teacher A). This idea of flexibility of content and context for learning skills was demonstrated by Teacher B who successfully integrated InfoMaker , a badge series aligned with Next Generation Science Standards, into a social studies class. Use of the digital badges required minimum preparation that you would do anyway (Teacher A).

The digital badges functioned as both formative and summative feedback strategies, and the students persisted with their tasks to earn the badges. Due to limited technology resources, students in the 10th and 12th grades were more likely to view actual printed badges on the windows of their classrooms (Teacher A).

These findings are of interest to educators and policy makers. Although it is not possible to generalize the findings of this research, due to the small, homogeneous student sample, the results of the study are promising. A great deal of research is needed, however, to understand how digital badges may function in different learning contexts and for whom (different student groups). Suggestions for future research include additional study on the design principles for standards-based digital badges, validity and transferability of skills, instructional practices and innovations using digital badges as well as research to understand the theoretical basis and best practices in using digital badges for motivating students.

5 Digital Badge Samples

Will be available at www.badgebox.net/research (to be constructed)