Introduction

With the advent of interactive and networked tools for game-making and “game-modding” (commercial games that are customizable), the capability now exists for designers of all ages to actively create and distribute interactive web games online. Educational research scholars have linked a range of positive learning outcomes to learner participation in game-making activity when it involves collaborative workshop-based computer programming and design across time. These outcomes include meaning-making, appropriation of, commitment to, and sustained engagement in a given creative project, as well as computational and systems-oriented thinking, learners’ development of a deeper understanding about the subject of the game, and a range of affective and motivational changes related to self regulation and self efficacy (Harel and Papert 1991).

This development study explores student self-reports of engagement and meaning-making, as well as dislikes and challenges, as they participate in a program of game design learning being offered to middle school, high school, and community college students in schools throughout West Virginia. This program is part of the Globaloria pilot project of the World Wide Workshop Foundation, a 501c-3 non-profit organization in New York City (NYC), in which students and educators participate together in workshop-based program under what the organization calls a “co-learning model.” Findings stem from qualitative student feedback on their experience in Globaloria, collected in a mid-program student feedback survey administered during the project’s Pilot Year 2 (2008/2009).

In this development study we describe the intervention and its components, present student experiential feedback, and draw linkages among themes emergent from student feedback and recent debates in the educational research literature on discovery-, inquiry-, and problem-based learning. We have implemented the study in order to identify suggested design improvements and more targeted and refined research questions that might be addressed in the larger context of this 5 year pilot initiative, the ongoing investigation of which can contribute towards furthering instructional theory development and practice strategies.

Program overview

The Globaloria-West Virginia (WV) Project is a five-year grant-supported pilot initiative being conducted by the World Wide Workshop, a 501c-3 non-profit organization in New York City. In 2007, this organization partnered with the WV Governor’s Office of Technology to establish and roll out a state-wide network and curricular program of collaborative game design and social media learning among teachers and students, bearing a mission to help mitigate technology gaps in public education, especially in West Virginia’s poorest rural locations, and to serve as a model for 21st century learning integrating technology in the classroom. The program employs a “co-learning model” in which students and educators learn together. Some of the learning interactions that occur in the program include (a) self-led learning in which students and educators learn individually through their own independent game design creative process; (b) peer-to-peer learning, in which students learn from other students, and educators learn from other educators by interacting both in person and online; (c) expert-guided learning in which professional game design experts help scaffold learning and help solve problems on demand in the Globaloria network through live in-person trainings and Skype and e-conference sessions; and (d) co-learning, in which students and educators learn together (online and offline). In this program, educators are co-learners, instead of didactic instructors.

The organization provides a guided, inquiry-based in-school curricular program for computer-supported collaborative game design, construction, and sharing to participating schools. Schools offer the game design classes to students as an elective for credit and a grade. Students participate daily. School partner participants receive the following affordances and learning supports from the World Wide Workshop Foundation: on-location professional development trainings for educators in West Virginia that occur twice a year; Flash software licenses for each student enrolled in game design classes; an open source game design course syllabus and curriculum; a wiki-based e-learning environment called MyGLife.org that includes pre-formatted wiki tools for students’ online collaboration, code sharing and game publishing; a suite of targeted free game design tutorials and resources embedded as links in the online syllabus to support game design learning in context; weekly virtual office hours via Skype and web conferencing tools with a Flash game design expert, and ongoing in-person student and educator training on-location throughout the school year and summer.

Each syllabus topic is presented as a link and webpage on each school’s wiki, providing access to a set of activities, tutorials, and other learning supports, all posted online. Through the guidance of their educator and Globaloria experts, full-year students typically proceed through the Game Design syllabus topics in Semester One, and Game Development topics in Semester Two. As they proceed in learning Actionscript and developing game assets, students upload and share their files and post assignments on their profile and projects pages. Teachers develop a schedule and assign deadlines for all of these assignments, based on the local needs and timeframe. A screenshot below presents the wiki homepage for a typical class. The syllabus is accessible through links in the left-hand navigation (Fig. 1).

Fig. 1
figure 1

School wiki homepage example

Full size image

While in Pilot Year 2 creation of social issue games was encouraged, most educators allowed students to choose their own game topic, supporting student project-based work centered on subjects of their own individual interests (Edelson and Joseph 2004; Hidi and Renninger 2006). To date, students in the program have created games that reflect topics in the following genres: (a) games about core educational subjects (e.g., games about math, science or civics), (b) social issue games that provide a social justice message (e.g., games with themes relating to poverty, nutrition, environmental issues such as global warming, and local issues central to Appalachia), or (c) games that could be classified as purely entertainment (for example, a fantasy game about ninja pandas or pop-culture oriented themes).

Theory supporting development of Globaloria

The founders of the Globaloria program have applied constructionist, situated learning and social learning systems principles to the program’s design and development. Constructionism is a philosophy and framework for learning and educative action developed by Seymour Papert, Idit Harel and colleagues at the MIT Media Lab in the 1970s and 1980s. Constructionist learning interventions typically involve students in project-based design work over time in a collaborative workshop-based setting, and engage students in technology programming activity to design and build a computational artifact representing an original idea, for instance a mathematical concept in a simulation, a digital game, or a robot’s programmed movement and action. Interventions that are considered to be “Constructionist” tend to include the following attributes in common (Harel and Papert 1991):

  • Workshop-based learning in a setting in which the learning process is made explicit; students can share openly, talk openly, and interact freely, supported by an expert mentor who guides students’ learning;

  • Students’ use of programming languages and computational design tools to create complex representational digital artifacts such as games bearing content themes and subjects that were developed with an audience of game players in mind (or learners, if the game created is educational);

  • Affordance to students of significant time daily, across several months, to pursue the completion of a final design artifact;

  • Frequent student reflection upon and social expression about their work in progress, knowledge-sharing, collaboration, and peer teamwork in a community of practice;

  • Sharing and presentation of final work in the team and group context.

The early Constructionist research (Harel and Papert 1991; Harel 1991) introduced young public elementary students to a year-long daily workshop-oriented learning intervention involving students’ creation of computational fraction games using the Logo programming language. Harel and Papert’s (1991) research was conducted in tandem with the program implementation and found that students evidenced a range of outcomes. These included development of technical programming and design skills, project-based learning skills such as prioritization and time management, teamwork expertise, meta-cognitive advances as evidenced through conversation and artifact analysis, content knowledge gains about the mathematical subject matter of students’ created games, and observed affective changes related to motivation for learning, self regulation and self efficacy (Harel and Papert 1991; Harel 1991). Follow up research has supported, refined and advanced the early findings and research methods utilized (e.g., Kafai 1995; Bruckman and Resnick 1995; Kafai and Resnick 1996; Urrea 2001; Kafai and Ching 2004; Kafai 2006; Kafai et al. 2007; Klopfer 2008; Reynolds 2008).

Scholars in the learning sciences have continued to develop and advance the design-based research method, which bears attributes aligning with the early Constructionist research methods applied; arguably, some of the origins for design-based research can be placed in the earlier work at MIT relating to children designers. And, scholars in the area of computer science education or “computing education” continue to draw upon the early Constructionist research in their exploration of best instructional theories and strategies in teaching students to effectively learn programming and computer science principles. For instance, Guzdial and colleagues report that introducing students to programming through creation of “computational media” products in constructionist interventions is an especially motivating way to teach computational thinking to undergraduate novices and can provide new pathways for under-represented students and groups to develop technical expertise and career interests, leading to greater diversity in the computer science disciplines (Guzdial and Soloway 2003; Rich et al. 2004; Forte and Guzdial 2005). Design-based research being conducted today, and the growing literature addressing computational thinking in the field of computing education are both relevant linkages for the current Globaloria work.

Six contemporary learning abilities

In Globaloria, students participate in a set of practices that are designed to cultivate six “contemporary learning abilities” (“Six CLAs”). Specifically, the CLA practices involve novice’s use of existing commercial and open source social media technologies, as well as their own creation of new computational artifacts through introductory programming activity. It is the integration of the participatory, social, and communicative practices central to social media, coupled with the computational project-based work practices central to game design, engaged with together over time, that we expect to be particularly meaningful and beneficial for students.

The 6-CLAs are presented below (Table 1). These practices were chosen as an integrated set of dimensions that we propose simulate productive contemporary digital knowledge-centered work. Therefore, students’ engagement in these practices in an integrated way can lead to a useful and important new set of expertise that may contribute to their successful participation in today’s increasingly technology-infused professional work cultures (Reynolds and Harel Caperton 2009). We also expect the Globaloria experience may bring about some of the same findings elicited in the earlier Constructionist work, such as advancing learning of the knowledge domain of the digital design project (e.g., the game’s subject), and influencing some students in affective and motivational domains, possibly enhancing their experience of learning in the school setting, through the creative, non-traditional social learning environment that is cultivated in pioneering West Virginia participating schools.

Table 1 Six contemporary learning abilities
Full size table

The Six CLAs serve as the learning objectives, outcome goals, and drivers for the continued program design and curriculum decisions made in iteratively developing the program. Approaches to teaching digital literacy in the school context have focused on the importance of imparting specific technology skills, and have been driven by association standards, for instance the NETS technology literacy standards promoted by the International Society for Technology in Education (ISTE 2007a, b), and the InfoPower information literacy standards promoted by the American Association of School Librarians and American Library Association (AASL 2007). While earlier versions of the technology and information literacy standards tended to focus on more “Web 1.0” forms of information-seeking activity, including searching, locating, evaluating and using informational resources online, the most recent updates to both sets of standards incorporate creative technology uses, and dispositions for productivity with technology tools.

For instance, the ISTE NETS standards for students include learning objectives categories such as Standard 1, “Creativity and Innovation,” which calls for students to be able to “demonstrate creative thinking, construct knowledge, and develop innovative products and processes using technology” (ISTE NETS Standards for Students 2007a, b). The AASL standards exhort students to gain not just technology skills, but dispositions to use those skills, and AASL Standard 4 is entirely focused on students’ pursuit of technology and information uses for personal and aesthetic growth. These standards offer considerable synergy with the Constructionist approaches we have adopted in Globaloria, and achieving these objectives could be seen to require Constructionist interventions. Further, the importance of involving learners in programs of creative, project-based digital work is gaining more and more credence as digital literacy, participatory culture and digital divide concerns enter the national educational agenda (e.g., Jenkins 2009; Hobbs 2010; Knight Commission on Information Needs of Communities in a Democracy 2009; Mossberger et al. 2007; National Education Technology Plan 2010). Unfortunately, while the updated standards and national priorities for instance those mapped out in the National Education Technology Plan of 2010 reflect significant advances in policy guidelines addressing technology integration for learning in schools, the reality is that actual implementation of substantive technology-based interventions in public schools nation-wide is still relatively rare.

Discovery-based learning

In order to support the 6-CLA program objectives, Globaloria incorporates a mix of both closely guided instruction and inquiry-based learning, as well as discovery-based learning strategies in its co-learning model, in which students and their educators work together using the syllabus and design resources provided. In Globaloria’s co-learning model, students have occasion to engage in self-directed inquiry online to seek out learning supports using resources provided by the World Wide Workshop Foundation. This is especially the case at schools in which the educator is a novice him/herself, and is therefore unable to provide expert scaffolding and support for a given game design task, due to the educator’s own ongoing learning curve. While a range of resources are provided in the online syllabus, including links to specific video-based and written tutorials, the use of such resources requires a measure of reflection and self-initiative by student participants as they experience an immediate need in the game design process. In these moments, students engage in inquiry to seek out resources in the Globaloria tutorial database that meet the need. The rationale for the co-learning model is the larger societal context of technological advancement in which we are educating today’s youth and training educators, and the immediate need to train teachers and students on effective technology uses, to bring about a computationally-literate public now—to stem digital divide gaps at both level 1 (access) and level 2 (sophistication of use).

In self-determination theory, scholars including Deci and Ryan (2008) have discovered that three primary constructs underlie intrinsically motivated human behavior, and are innate needs: the need for competence (to be effective), for autonomy (to have choice and control over one’s life), and for social relatedness (to feel connected to others, loved, and cared for) (Deci and Ryan 2008; Deci and Ryan 2000; Deci and Ryan 1985). These scholars propose that individuals will generally pursue goals that allow for these needs to be met, and when these needs are satisfied, the activity contributes to intrinsically motivated action and psychological well-being. When operating optimally, the Globaloria program as an inquiry- and project-based learning experience for students has all of the trappings of providing autonomy-supportiveness, fostering positive and productive social interactions and feelings of relatedness, and supporting children’s need to feel competent (i.e., confident).

However, some researchers have indicated that instructional strategies involving “discovery based learning” features are not optimal because they are not supportive enough of the learner, and frustration can result from the heavy working memory (cognitive) load required in such self-directed learning practices (Kirschner et al. 2006). These authors state, “Although unguided or minimally guided instructional approaches are very popular and intuitively appealing, these approaches ignore both the structures that constitute human cognitive architecture and evidence from empirical studies over the past half-century that consistently indicate that minimally guided instruction is less effective and less efficient than instructional approaches that place a strong emphasis on guidance of the student learning process” (p. 75). These findings appear to conflict with Deci and Ryan’s (2008) conception of the importance of autonomy.

Further, while cognitive load may be one concern, other recent research findings reveal specific learning benefits inherent to unstructured problem-solving contexts. For instance, Kapur and Kinzer (2009) report evidence of a “productive failure” phenomenon in teams of students who are placed in an “ill-structured” (IS) learning problem scenario. These scholars find that while students in the IS context initially perform more poorly when their solutions to an initial problem set are content analyzed and compared to students in a well structured (WS) context, when the groups are re-tested after using two types of later problem set post-tests, the IS students performed better than WS team students. The authors highlight discernability (Marton 2007; Schwartz and Bransford 1998) as a causal contributor, stating, “the ability to perceive and structure a complex, ill-structured problem is a critical dimension that seems to differentiate experts from novices, and a substantial amount of research speaks to this.”

It appears that Kapur and Kinzer’s (2009) characterization of problem-based learning in ill-structured contexts overlaps with Kirschner et al.’s (2006) definition of discovery-based learning. To the extent that Globaloria’s 6-CLA dimensions may initially feel to students like an ill-structured learning context, the productive failure findings may apply. However, to the extent that Globaloria’s program implementation results in minimally-guided activity, it may be that cognitive load and frustration results. Overall, appears that these two literatures offer yet further contradictory claims, in addition to those involving autonomy.

In the Globaloria project, in some events and contexts, educators and virtual game design experts offer close scaffolding; at other times, students engage in self-directed autonomous seeking and use of online tutorials to support their game design learning. This is necessitated in the context of the very grounded realities of educator and student co-learning about technology. Here we consider students’ overall impressions of their experiences in the program. We draw upon and interpret student feedback in relation to the literatures on self-determination theory, cognitive load and productive failure.

In particular, we explore the ways in which student self-reports on their experience provide evidence of engagement and meaning making as they participate. Engagement and meaning-making are primary constructs integral to social learning systems, and are objectives built into the Globaloria innovation’s design (Wenger 2003). Wenger (2003) defines engagement as “doing things together, talking, producing artifacts” (p. 78). In engaging together, members “identify gaps in their knowledge and work together to address them” (p. 82). In addition to engagement, two important facets of a social learning system are realistic imaginative activity, as well as alignment (the extent to which activity can be effective beyond the local engagement). Wenger (2003) suggests that every social learning system involves all three to some degree or another. Globaloria is a social learning system in which students and educators share a semester or year-long constructionist game design experience, sharing language, tools, artifacts and methods. Globaloria also builds in realistic imaginative activity in that students practice the professional role of game designer, and create games that are published online and playable by others—which reflects Wenger’s third attribute of alignment (effectiveness beyond the local engagement). Wenger (1998) suggests that meaning-making activities bring about learning and change.

Research Question 1 for this study is stated: In what ways is the Globaloria program engaging for participating students?

For this first question, we address evidence of meaning-making, given that the Globaloria program and learning environment (in class and online on the MyGLife.org wiki) provide the opportunity for students, educators and schools to actively participate in a “social learning system” (Wenger 2003) centered on a workshop model that is quite different from the traditional school format. Exploration of student engagement and meaning making can provide initial supporting evidence for the instantiation of this social learning system. The findings can also lend insight into debates around discovery-, inquiry- and problem-based learning interventions (e.g., Kirschner et al. 2006); if meaning-making and engagement occurred, such a finding could present evidence that the learning as it occurs in the context of this project’s eclectic intervention may be motivating.

Because this is a formative implementation study, we are also interested in critical feedback provided by students, in that it can signal the need for iterative adjustments and improvements. Therefore, research question 2 is stated: What insights emerge from students’ critical feedback, especially in relation to specific attributes of the learning intervention that can be changed? Such a finding may indicate the need for improved supports and more scaffolded guidance in specific areas of focus.

Method

In Pilot Year 2, the World Wide Workshop Foundation selected schools based on adherence to a set of criteria screening for hardware availability in the school, time availability, commitment of the educator, and the school’s ability to offer the class as a daily course for credit. Economically and technologically disadvantaged participant populations were given priority. Educators became aware of Globaloria through WV press releases, prior participation, and presentations made by the Program Manager at state-wide teacher conferences and events, then gained consensus with their school administration, and submitted an application. Once a school is confirmed, students self-select into Globaloria by enrolling in an elective course offered daily, for credit and a grade during regular school hours.

From Pilot Year 1 (PY1) to Pilot Year 2 (PY2), the number of middle school, high school and community college classes doubled, to involve 24 educators and 291 students in 11 counties throughout West Virginia in PY2 (93 middle school students, 161 high school students, 20 community college students, and 17 alternative education students). Thirteen PY2 partner schools implemented the curriculum as an in-school game design course elective offered to students daily for credit and a grade during the regular school day. In January of 2009, we asked all PY2 Globaloria students to respond to a 20 min online survey, consisting of thirteen open-ended survey questions, distributed via links on their Wiki homepages.

The voluntary survey had a total of 199 respondents. Using an IRB-standard consent form, we achieved parental or adult student consent for every participant in the Globaloria program. Survey completions span from January through May of 2009. The respondents include those who participated in the Fall semester only, the Spring semester only, as well as those participating in both Fall and Spring semesters. While the duration of program participation and time of survey completion varied in this dataset, for all students who responded, the survey was completed after at least 3 months of participation.

Because most students who completed the survey had not yet completed their final game design project, Pilot Year 2 mid-survey results reflect student feedback at a mid-point of their participation. Thus, we address the feedback as in-progress results mid-way through. In the “Discussion” section, we highlight necessary revisions and improvements to the research design as implemented, given limitations related to the in-progress findings as a data source.

Instrumentation and analysis

Mid-survey questions were refined and revised based on a similar survey conducted in Pilot Year 1. The questions were developed to elicit student feedback that would be informative to our program implementation in the context of an overall design-based research paradigm for the project. The four open-ended feedback survey questions we found most useful in contributing to the study research questions are as follows.

  1. 1.

    What are the three the most important things you have learned in this class so far?

  2. 2.

    How is this class different from your other classes and school work?

  3. 3.

    What are two things you dislike about it?

  4. 4.

    What was most difficult about the class?

We analyzed mid-program survey data using textual data analysis. Textual data analysis (TDA) involves the processing of a collection of textual data for interpretation and understanding. Open-ended responses were coded inductively for the purpose of construct development, seeking to identify second-order, theoretical constructs (Gephart and Wolfe 1989). First, we read responses to our four prioritized survey questions, and began to build a set of broad categories reflective of student responses to each question. We placed emergent categories in spreadsheet columns, adding to and revising as we reviewed question responses (in rows). When confident that categories reflected the range of responses present in our dataset, we reviewed responses again, and assigned each response to its given category (marking the column with a one). Students’ open responses often reflected more than one category, thus in some instances were coded in multiple categories. The unit of analysis was the entire response provided by the individual student, considered in full. In our results, we present descriptive data for each question and each category. When further reviewing and analyzing the responses and choosing examples, we were able to generally note the valence of responses. Not surprisingly, given the nature of the question text, the valence for responses to the first two survey questions tended to be positive. The valence for responses to the second two survey questions were generally negative.

Results

Evidence supporting the study’s Research Question 1 on student engagement and meaning-making emerges mostly from two feedback survey questions. One of these survey questions is “What are the three most important things you have learned in this class so far?” For this question, students’ responses contribute to our understanding of (a) what activities they deem to be important, and, (b) what they feel they have learned. Importance signifies that students have ascribed meaning to the given activity they mention. The activities/skills that students self-report they have learned signify that the responding students are developing confidence in their skills in the given area discussed. They may also signal actual knowledge gained, to the extent that self-reports correlate to actual knowledge. The categories of findings that emerged from our analysis of the range of student responses to this survey question are as follows (Table 2).

Table 2 Categorized responses to mid-survey question on “importance”
Full size table

The top five most commonly cited response categories are: creating or making games with Flash or other software; computer programming/coding; teamwork; proposing and project managing game design process; and how to use a wiki. These are the primary activities in Globaloria, and many students reflect agreement in their ascription of importance to their learning of these activities. Examples of student responses to the importance question include,

Three things i have learned in this class are how to propose an idea and follow through to create it, to work in flash and animate objects, and to work with others. (female H.S. student)

I have learned how to use flash, make a vector, and soon I will have learned how to make a game. (female H.S. student)

I learned a lot about ActionScript and coding in general. I have a very long way to go before I make a major game, but with determination I can do it. (male H.S. student)

Importance signifies value and indicates that students find these activities relevant and meaningful. This finding supports the work of Guzdial and colleagues who report that introducing students to programming in a context of creating “computational media” products using a Constructionist intervention is an especially motivating way to teach computational thinking (Guzdial and Soloway 2003; Rich et al. 2004; Forte and Guzdial 2005). Such interventions have been found to provide new pathways for disadvantaged students into computer science disciplines.

In addition to the “Importance” question, we also analyzed student responses to the survey question, “How is this class different from your other classes and school work?” Table 3 presents the categories of results that emerged.

Table 3 Categorized responses to mid-survey question on “difference”
Full size table

Fifty-one percent of students (101 out of 199) refer to the frequency of computing in Globaloria as a main difference over and above their other classes. Examples for this category are provided as follows.

This is The Only Class that I am using a computer. (male M.S. student, emphasis his)

This class is different from my other classes and school work because in this class, we don’t usually do worksheets, reports, or speeches. We mainly work on the computer, using programs to make a computer game. (female H.S. student)

Examples for the fun/not boring category include,

This class is different because it’s fun and interesting. It challenges you in a GOOD and POSITIVE way. (female H.S. student)

This class is different from my other classes by in this I have fun when I’m learning new skills on Flash. When in my other classes most of the work we do is boring, and I’m not really interested in it as much as I am with Flash. (female M.S. student)

A female high school student highlights the autonomy and choice inherent to the program in choosing the subject of one’s own game, responding,

This class is different from my other classes and school work because we get to learn exciting things. In most classes you do not get to learn how to make a video game. Plus what ever you decide to make your game on you get to learn about. I’m learning things about history for our game that you would not get to learn in school.

These findings support a National Education Association (NEA) educator survey study (2008) suggesting that while most educators used technology regularly at school for administrative tasks, substantially fewer used it for instruction-related tasks. Most believed that technology uses had improved students’ motivation for learning (National Education Association 2008). Supporting these findings, many student expressions regarding frequency of technology use are quite positive in tone. Further, thirty-two students (16%) state that Globaloria is different because it is fun and enjoyable. Students’ ascription of fun to the class could stem from several attributes, including the project activities themselves, as well as the workshop-style co-learning environment the educators are trained to foster, in which students work in teams and have the chance to communicate and share their work.

Several of the 199 students (fourteen; 7%) also report that the self-directed nature of some learning in Globaloria is different from other experiences they have had in school previously, and most of these 14 responses indicate a positive tone of enjoyment towards the self-directed learning they experience in Globaloria. Supporting evidence is provided as follows.

This class is different from the other classes at school because you don’t have to wait on the teacher to tell you everything, you can learn on your own, most of the time. (female M.S. student)

In this class, we get an assignment and we work with our teammate to figure out how to do the task. If we need help, we ask other students in the class, and if they do not know we ask the teacher. In doing this, we learn how to do the tasks by figuring them out on our own. When you try to figure out how to do something on the computer, you usually learn other things along the way. (female H.S. student)

Not all of the responses to this survey question that mention self-directed learning were positively valenced. One male H.S. student reports on the difficulty of self-driven learning in response to the difference question, stating,

This class has been harder than most classes I’ve taken. Not because the work is harder, but just getting things done in a self-taught environment is hard for me.

This response contrasts to the prior examples. On the whole, though, most of the responses to this question that mention the self-directed nature of the work are positive, creating the impression that for at least a set of students, some of the self-directed learning experiences they encounter in the program appear to contribute to students’ meaning-making, and their engagement and motivation. According to Deci and Ryan (2000), autonomy is one of the drivers of self-determined and intrinsically motivated behavior (along with perceived competence and social relatedness), and it could be that students’ feelings of autonomy are supported in Globaloria and contribute to this meaning-making.

Project-based work is a hallmark of constructionist interventions, and we also find evidence in student responses regarding enjoyment of project-based work. Twelve student responses (6%) attribute differences between Globaloria and other classes to the “hands-on” and design-based work, and many are again positive in tone. Examples of responses include the following:

The class is more hands-on than the core classes like History or Math. The work doesn’t seem like work to me; I enjoy doing my work, so it’s not work then. (male H.S. student)

This class is more hands on than most of my other classes, and focuses more on my future. I feel more responsible about my work and feel compelled to succeed in my future. (female H.S. student)

This class is different from my other classes because we learn how to create projects hands on using our unique perspective on life. (female M.S. student)

Further, 17 students mention the course focus on the design process in general as a differentiator from other schoolwork. Six students self report feeling more productive as they participate in Globaloria. Claims for productivity imply that those responding in this way find their work activity in Globaloria to be worthwhile, of value, and fulfilling. These responses mentioning hands-on, design-oriented learning and productivity provide additional support for RQ1, in that for several students, it appears the project-based nature of the work is particularly engaging.

Socio-constructivist perspectives underscore the design of the Globaloria intervention. Thirteen students out of 199 identify collaborative teamwork as a differentiator. Students state,

This class is not as pressuring. I mean, I’m here, I have my friends to help fix things, and I get to show off some of my artwork (Which is kinda hard on the flash…. > .<) Not only that but this class teaches team work alot better than the others, and as you know, team work is important in the outside world! (female H.S. student)

This class is different because we have fun for the main thing. Plus, we get to cooperate with each other and I think that this helps with alot of kids self esteem. We got introduced to a brand new thing instead of the same old things. (male M.S. student)

Here again, most of the responses discussing collaborative work are positive in valence.

Summary of responses, research question 1

In their responses to two mid-survey questions, many students present strong evidence of being engaged in the Globaloria program, and ascribing meaning to their experience. The top five categories of activities in which students ascribe importance and value are:

  • Flash and other software to create a game (42%)

  • Coding/programming (23%)

  • Teamwork/people skills/social skills (19%)

  • How to propose a project and then create it and complete it; time management; project management; paper prototyping (19%)

  • How to use the wiki (18%)

The top five categories of activities that students identified as differentiators were:

  • We use a computer more (51%)

  • Class is fun/not boring (16%)

  • Here we make/play games (10%)

  • Involves design/creation/Flash (9%)

  • Involves self-directed learning (7%)

Across both of these sets of categories for the importance and difference questions, the category that overlaps is the activity of learning Flash to create an interactive web game. It appears many students find this primary Globaloria activity meaningful.

Scholars such as Papert (1980, 1996) and Seely Brown (2005) emphasize the importance of digital literacy and collaboration in networked, evolving, technological environments. Seely Brown notes that “since nearly all of the significant problems of tomorrow are likely to be systemic problems—problems that can’t be addressed by any one specialty—our students will need to feel comfortable working in cross disciplinary teams that encompass multiple ways of knowing” (p. 2). He emphasizes the importance of “learning to be” through role-taking experiences, in contrast to “learning about,” through traditional top–down instruction. He further states that, “today’s students want to create and learn at the same time. They want to pull content into use immediately. They want it situated and actionable—all aspects of learning-to-be, which is also an identity-forming activity. This path bridges the gap between knowledge and knowing” (p. 6).

Globaloria offers students the opportunity to engage in such epistemic role-taking. Several students convey a positive affect towards autonomy afforded in the class, and express importance and value for the core Globaloria activities presented. These findings indicate the appeal of the workshop style of learning among some participants. Overall, the self-report findings for RQ1 indicate that for some students, Globaloria is offering a new mode of engagement in the school setting that facilitates student role-taking and “learning to be,” supporting Seely Brown’s (2005) claims that this kind of learning context resonates with today’s students.

Research question 2: student critiques of the program

Research Question 2 is, “What critical feedback do students provide about their experience in the program?” Here we explore students’ expressions of criticism and negative feedback, in that these may reveal potential areas for program improvement, a key goal in this pilot phase. RQ2 is addressed through analysis of feedback survey questions 3 and 4: “What are two things you dislike about Globaloria?” and “What do you find most difficult?”

The range of categories that emerged for this question are presented in Table 4. Forty-six students (24%) noted disliking that the experience overall is hard or difficult, and some noted disliking that learning Flash is hard. Examples include,

Table 4 Categorized responses to feedback survey question on “dislikes”
Full size table

It is hard. There is too much coding to put in the game. (male H.S. student)

Flash. I dislike it, it’s horridly difficult to learn. Also, I dislike the fact that some people pick it up faster than others. (male H.S. student)

A total of 39 students (20%) disliked not having enough time in the program to accomplish all of their goals. Examples of student responses include,

It goes by too fast. We’re always runnin around with our heads cut off. (female M.S. student)

Time, I wish it was longer because its hard to get alot done in only a 40 min period and if you dont have alot of knowledge of this subject its hard at first to get started (male H.S, student)

Globaloria requires students to learn a substantial amount of new material, in new ways, across either a single semester or a single year. Completing a fully functioning game file in Flash is challenging for some in this timeframe. At the same time, the call for more time from students may reflect an underlying engagement and motivation towards the activity. It is important to note here that students were still at the mid-program phase when answering this question.

Self-led learning A total of 17 students (9%) mentioned disliking some aspects of the tutorials or the need to find their own learning resources, stating,

Things are sometimes hard to figure out and that we sometimes don’t have help. (male M.S. student)

I did not like that most of the time the only help you got was from the other students in the class. I did not like that there was not enough tutorials to help with coding. (male H.S.student)

These results may support some of Kirschner et al.’s (2006) claims regarding discovery-based learning that is minimally guided. We discuss this result with other related results further below. Table 4 summarizes the “Dislikes” results.

Evidence from the “difficulty” question

Given that so many students answered the dislikes question with a response indicating that they found the program difficult, we decided to also analyze another survey question, which asked students specifically what aspects of the program they found most difficult. Table 5 below presents the categories that emerged for the survey question on difficulty. Making games is a primary objective for students in Globaloria. A total of 37% (74 students) state that coding or programming is difficult, and 24% (48 students) state that making games or designing/drawing in Flash is difficult. There is no doubt that programming is inherently difficult, especially for technology novices. This difficulty might be compounded by the fact that in many cases, in this early phase of the pilot, WV educators are also still novices. A total of 10% (19 students) cite time management as difficult, supporting similar findings for the “Dislikes” question. An example from these categories of the “Difficulty” question includes,

Table 5 Categorized responses to feedback survey question on difficulty
Full size table

Like all the animation that you had to put in ever-thing. Because I’m not so good at that stuff as the people who ever thought of this. But for the must part I like this class better than any-other. (female M.S. student)

A total of 8% (15 students) present a response indicating that the multiplicity of the activities in which they participate in Globaloria and the concentration required to engage in them all in an integrated way was difficult. For example, students state:

The most difficult thing about this class would be how we have to learn extremely different things in a short amount of time. (female M.S. student)

Trying to soak up all the information to do even basic stuff. (male H.S. student)

In Globaloria, students’ own creative ideas for a game drive the game design and development process. Coming up with a game design idea requires one to first conceptualize what a game is. In the “Getting Started” module of the syllabus, students are presented with an overview of games, and game examples to help prime their game design ideas. A total of 6% (11 students) mention developing a game concept as difficult.

Self-led learning. A total of 6% (11 students) mention aspects of self-driven learning as difficult. Examples of student statements in this category include,

The most difficult thing we have to do is to put actionscript into our game. I have watched many tutorials on actionscript and I still do not understand it quite yet.

–Female high school student

The most difficult about taking so slow to learn and watching the videos a million times, was being patient about learning. (female M.S. student)

The most difficult about this class is the fact that you have to learn it for yourself. You can’t sat down and (be) told what to do. (female M.S. student)

This finding supports results reported above for the “Dislikes” question in which 17 students (9%) mentioned aspects of the tutorials or the need to find their own learning resources.

Summary of responses, RQ2

Students referenced a range of dislikes and difficulties. One of the most oft-cited dislikes was that Globaloria is difficult. When analyzing the survey question specifically asking about student difficulties, in some cases student responses on difficulty paralleled the categories of dislike that emerged in the data. The top five categories of activities that students disliked were:

  • It is hard/frustrating/confusing

  • Time management

  • Self-driven learning/searching for supports

  • Hardware/Software/Installation issues

  • Blogging

The top six categoriesFootnote 1 of activity students found difficult were:

  • Coding/Programming

  • “Making Games;” Design/drawing in “Flash”

  • Time Management

  • Multiplicity/Depth of Comprehension

  • Game Concept

  • Self-driven learning/searching for supports

Difficulty on its own is not inherently a negative quality to attribute to an activity. The following categories were present in the top 5 for dislikes and difficulty.

  • Time management

  • Self-driven learning/searching for supports

These categories may indicate areas where students both disliked and found activities difficult. The dataset was not analyzed in such a way to determine relationships among categories, however given the commonality of student responses in these categories, we should consider these two areas as potential targets for improvement and enhanced support in the program, as they may cause particular frustration for students and could hinder their motivation and learning. In contrast, while many found the primary Globaloria activities Flash design and programming (the mainstay activities of the project) difficult, a relatively smaller number of students reported disliking these activities (13; 7%); and, these primary activities did not fall within the top five categories of dislikes.

Discussion

The RQ1 findings on engagement and meaning-making indicate that a combined total of 129 students out of 199 (65%) expressed views that game design, Flash programming and coding were the most important things that they learned, indicating value for the primary Globaloria activities. Further, a total of 16% of students self-report that the class is different from their other classes because it is fun and enjoyable. As for the RQ2 results, 24% say they dislike that the class is difficult (among other things), and when asked specifically what is difficult, a combined total of 61% of students indicate that the main course activities of game design and Flash programming are most difficult. Regarding the category of self-directed learning that emerged in three out of the four survey questions, while 7% (14) highlight autonomous use of online resources as a differentiator from their other classes and express largely a positive affect in their statements, 17 students (9%) state outright that they dislike some aspects of the tutorials or self-driven learning process, and 11 students (6%) state that aspects of self-led, minimally guided learning are difficult.

It is important to note that variation exists across locations as to student grade level, as well as level of educator expertise, and quality of educator scaffolding and support of students. Further, in the social learning system of Globaloria, the educator is not the only guide. A range of other materials and resources are provided. While in most cases, access to the wiki resources are provided front and center to students in the context of modular syllabus assignments they follow, on occasion, students may encounter a problem or hurdle that is not addressed by these resources. They have a range of options at that point. They can seek help themselves by searching online, ask peers or their educator, ask Globaloria game design experts via Skype or web conferencing during virtual office hours, or, skip that step in their learning in frustration. Our current research does not capture these variations that occur at the individual, group, and process/sequence levels.

Nonetheless, student responses at the varying locations did echo several common themes, and allow us to discern some patterns and make some inferences about the program and student experiences in Pilot Year 2. It appears from the contrasting results on engagement, meaning making, and critiques, that Globaloria presents a learning environment that many students find relevant, motivating, and in many cases fun, as well as quite difficult. Also, students are not without their dislikes on certain program attributes. While we did not analyze relationships among response categories, the findings above suggest that students may simultaneously hold contrasting and seemingly contradictory perceptions. Indeed, one student notes that she thinks Globaloria is both hard and fun, within the same sentence, stating,

Some of the programs we use are hard to get used to, but after you master it is pretty fun. Other than that i don’t really have any complaints. (female H.S. student)

“Hard fun” learning has long been observed in constructionist programs and environments (Papert 1998, 2002). Papert (2002) states that hard fun activities are those that are fun because of, not in spite of, their challenge. He suggests hard fun “…is expressed in many different ways, all of which boil down to the conclusion that everyone likes hard challenging things to do. But they have to be the right things matched to the individual and to the culture of the times. These rapidly changing times challenge educators to find areas of work that are hard in the right way: they must connect with the kids and also with the areas of knowledge, skills and… ethics that adults will need for the future world.” Hard fun learning requires student engagement in activity “matched to the individual and to the culture of the times” (Papert 2002). Student ascription of meaning and importance to game design appear to indicate that Globaloria learning activities are relevant and appealing to many.

At the same time, the two different valences (positive and negative) evidenced in student responses on self-driven learning aspects of the program in which students feel minimally guided appear to signal variation in student attitudes towards the autonomous work that is inherent to the co-learning model in which educators are still learning themselves. It is also important to highlight the preponderance of students who describe learning Flash as hard (61%), which may also indirectly relate to the minimally guided nature of some learning, in particular the use of online tutorials when a teacher lacks expertise to support the student’s learning.

These particular findings appear to support work cited by Kirschner et al. (2006) relating to cognitive load. Kirschner et al. (2006) suggest that frustration inherent to cognitive load is inversely related to expertise, and thus, novice learners are most susceptible. Thus, it may be that those students who respond positively on their relative enjoyment of self-driven learning practices are more advanced, whereas those who express direct aversion to it are less so.

Another area of scholarly literature that may help explain these results is Deci and Ryan’s (2000) research on self-determination theory, which holds that autonomy is a primary construct integral to self-determined behavior, and autonomy-supportive environments in which perceived competence and social relatedness are also supported are particularly conducive to supporting intrinsically motivated learning and engagement (which is particularly fulfilling for the individual). The variation in student affect towards self-led learning leads us to posit that those who have a more intrinsic and self-determined motivational orientation may enjoy and thrive in the context of the autonomy afforded in the self-led context, whereas those who have a more extrinsic motivational orientation may feel challenged, be more likely to express negatively valenced affect towards the context, and may perhaps need more structure, support and external incentives to be successful and learn. This hypothesis is an outcome of the inductive qualitative research presented herein, and will be explored in our ongoing research using motivational orientation survey instruments developed and validated by self-determination theory scholars.

Another literature that may offer insights to the Globaloria research presented here and moving forward is that relating to productive failure. Kapur and Kinzer (2009) have found that high school student engagement with ill-structured problem-solving tasks lead to improved performance in later repeated task completion over and above students who engage in a well-structured task, and this may be explained by the ill-structured task students’ cultivation of “discernability.” Kapur and Kinzer (2009, p. 39) state, “the ability to perceive and structure a complex, ill-structured problem is a critical dimension that seems to differentiate experts from novices, and a substantial amount of research speaks to this.” The proposition here appears to be that tasking students with ill-structured problems can bring about better problem-solving capacities through the experience with problem structuring such tasks provides.

Enhanced problem structuring and discernability are areas for continued exploration as a possible student outcome in Globaloria. Most of the productive failure research is conducted with interventions consisting of single problems at cross-sectional points (tasked in sequence), and doesn’t take into account cyclical learning events occurring in longitudinal project-based workshops across time. Research on the complex Globaloria social learning system presents both challenges and opportunities to researchers given the eclectic array of inquiry-, problem- and discovery-based learning experiences provided (Hmelo-Silver et al. 2007), some of which are closely scaffolded and some of which are minimally guided. More work is needed in the Globaloria research to link the level of scaffolding occurring in specific learning events at specific curriculum sequence points, to student motivation, affect, and learning outcomes.

Conclusion

Hmelo-Silver et al. (2007) pose a response to Kirschner et al. (2006). One of the key flaws in the original article that they highlight is that the authors conflate inquiry and problem-based learning as necessarily involving minimally guided instruction. However, on the whole, Hmelo-Silver et al. (2007) appear to agree with Kirschner et al’s (2006) main premise that minimally guided instruction is suboptimal. The criticism collectively posed by these authors towards minimally guided instruction stands in contrast to some tenets of socio-constructivist theory, as well as self-determination theory’s role of autonomy, and Wenger’s (1998) definition and goals of a social learning system, in which learners collaborate, problem-solve, and share knowledge together among peers in what is sometimes an ill-structured, open-ended situated learning context. The criticism also stands in contrast to the productive failure work, to the extent that ill-structured learning tasks are also minimally guided. More research is needed to reconcile these contrasts.

The reality is that most schools and teachers in the U.S. are under-prepared for supporting student learning with technology, and lack expertise required to do so effectively. Through Globaloria, the World Wide Workshop Foundation offers participating schools a flexible model for teachers’ and students’ development of digital literacy and contemporary learning abilities through game design and use of a wiki e-learning platform. In the early pilot phases of the program’s implementation, and as schools are newly added each year, some teachers are going to be as novice as the students, but through their ongoing experiences in the co-learning model, expertise in both populations will grow.

Overall, the evidence from this mid-survey conducted in the second year of the 5 year pilot has allowed for refinement of our research agenda and questions, through our interpretation of student feedback, our further explication of relevant variables for analysis, and our linkage of the results to related literatures. We cannot draw conclusions however from self-report data. Moving forward, we must more closely observe the ways in which specific learning events with varying types and amounts of scaffolding lead to varying outcomes. These outcomes include the attitudes students expressed in this survey towards different program attributes and towards their learning in general, but we also must include knowledge outcomes. Further, we will explore in what ways student individual differences such as motivational orientation may influence their experience and process in the program, and contribute to outcomes. Such exploration may help qualify some of the findings being surfaced in the productive failure literature in addition to the scholarly discussion and debate surrounding inquiry-, problem- and discovery-based learning and minimal guidance.

Our analysis of student feedback in Pilot Year 2 led us to improvements in the syllabus, course sequencing and game design resource supports. As of this publication, Globaloria is already in Pilot Year 4 (2010/2011), and we are actively engaged in case study and participant observational research at the individual and group/class levels of analysis longitudinally, as well as survey research in a growing dataset of 1,000 students, employing a non-experimental pre/post design and including motivational variables. We are continuing to explore how the program is becoming instantiated, and how learning and knowledge are created and shared among educators and students in the co-learning model, within and across varying networked locations, across multiple grade levels, and among participants with varying levels of expertise. The open source nature of the growing base of published artifacts and sample programming code serve as further scaffolds students can use in their learning and activity on the wiki. The program’s facilitation of cross-cultural teamwork among geographically distributed peers in both online and local environments is another important facet to consider. The findings presented in this study indicate that the Globaloria program may provide a particularly interesting and useful context in which to continue exploring several pertinent theoretical questions and debates occuring presently in the learning sciences, for instance those relating to inquiry-, problem-, and discovery-based learning, cognitive load, minimal guidance, productive failure, and self-determination theory. Our ongoing research will expand upon these initial linkages we have begun to map.