Keywords

1 Introduction

In principle, participatory design (PD) methods or similar interaction design methods take children’s ideas directly into design so as to meet children’s expectations for games [1]. Children should be critically contributing to the design with their ideas as experts of their experience. Designers should turn into reflective practitioners so that design becomes an act of “knowledge co-construction” in the sense of [2], for creating a shared experience. According to the adopted method and its philosophy, designers become full partners, peers, guides or facilitators of children’s expression, and bring in their professional expertise for the product under design.

However PD methods are also demanding on all participants; co-designing interactive products can require PD participants several resources [3, 4]. Games are the prototypical examples of interactive products that appeal to children and yet are demanding to design, in terms of time and commitment to learn: even the early design of a game can be complex to master and can require prolonged times in order to elaborate various elements of the game, ranging from its narrative to its organisation into levels.

Recently, the complexity of conducting an effective participatory game design (PGD) experience has led researchers to reflect on it. Assessing it becomes even more crucial when design moves into schools, which bring further requirements on design activities, e.g., [5]. This paper reports on two different PGD studies, one in 2014 and the other in 2015, in different primary schools, and assesses them critically. It does so by considering children’s game design (GD) products over time.

Most commonly in PD the outcome is the actual artefact or design delivered at the end of the experience; then an outcome embodies decisions and considerations and, as such, it brings researchers epistemology insights as design knowledge [6]. This paper partly embraces such a view: outcomes are children’s GD products, and epistemology is knowledge concerning such products. However, the paper also moves away from that view. Firstly, it considers products as the outcomes of collaboration and not of individual work, in line with what suggested in [5]. Secondly, the knowledge considered is not what specific game elements children could design at the end of their experience, as in [7]. Rather, this paper focuses on a complementary knowledge: what design issues recur over time in the GD process, by inspecting GD products by children. Therefore the paper is a reflection on PGD processes with children, and their unfolding over time, across two years of work.

Firstly the paper overviews related work for setting the context of the two PGD studies. Secondly, for space constraints, it only sketches their common PGD approach; for details, see [8]. Then the paper explains how two PGD processes for primary schools were organised, in 2014 and 2015. As GD products by children were evaluated in the same manner in both years, their evaluation approach and results are presented in a single section. Finally the paper discusses how children’s GD products evolved over time, and what categories of issues recurred in their products. By considering both studies in primary schools instead of one, across two years of work, we can compare differences and see what’s common across them in order to reason, on more general grounds, about outcomes of PGD with children, and about knowledge acquired through it concerning the PGD process in the conclusive part of this paper.

2 Related Work

A GD process is a complex interaction design process. At a fine-grained view, an early GD process is made of complex intertwined tasks, and specifically of: goal analysis; conceptualisation; prototyping. Game designers analyse the goal of the game, and create the game idea high-level conceptualisation, with the main rules for reaching the game goal. Designers conceptualise and prototype the core mechanics by refining the main rules and considering the progression across game levels. In case the game requires a storyline for stirring the game forward, within or across levels, designers conceptualise the storyline so as to make it consistent with the overall GD choices. If the game idea envisages an avatar, designers define that early, in conceptualisation documents and prototypes; in avatar-based gameplay modes, the player generally acts on the avatar, and mechanics rules are related to the avatar’s actions in the game; interface and interaction choices for the avatar have to be in line with the other GD choices. See [9].

GD for children has been differently approached [10]. Some designers prefer an individualistic approach to GD. Others, such as [11, 12], prefer a player/user centered design approach, involving players in the GD process and placing them at its centre. In recent years, PD has been receiving an increasing attention for involving children in the PGD process itself, as early designers.

PD forces designers to look at things from another point of view, which proves useful when participants are children. According to PD researchers, engaging children in the design process can lead to ideas that adults alone cannot envisage of [13]. Different PD methods have been devised for designing interactive products with children, and lately for designing games with children. A comprehensive overview can be found in [8].

Several PD methods assume that intergenerational small teams of children and adults work together for prolonged times, outside schools, e.g., [4, 14]. However the PD literature also counts PD studies with few design experts conducted within school hours and classrooms, e.g., [15, 16], in line with the manifesto in [17], which in 2014 foresaw that “elementary school children [will] learn about designing and co-designing through practical and fun hands-on experiences”. The studies reported in this paper, one in 2014 and one in 2015, follow the latter line of work. They required two researchers in the PGD processes within primary-schools: roles were well-specified in advance so as to make clear how and when adults would mediate children’s contribution, as recently recommended in [15].

Surprisingly, the assessment of children’s products in a PGD process, in relation to their evolution over time, is relatively under investigated in the PGD literature: despite the proliferation of PGD studies with children, at present, the “number of studies that provide a deeper understanding of the complex process of the design of games [with children] is limited” [18]. For instance, the research work of Moser counts different case studies of PGD with children, e.g., [19]. She conceived a GD framework with techniques for creating parts of games together with children, used in her case studies with children. However the PGD outcomes were not assessed in the case studies, as the focus was how to elicit children’s expectations for games and not the quality of the outcomes in GD tasks (person. comm.). More recently, Bonsignore, who coauthored several papers on co-design with children, reported co-design work concerning alternate reality games in [20]. The work of Bonsignore and colleauges inspect when the design process seems difficult for children, according to observational field-note data. This paper shares similar concerns but inspects GD products by children for drawing its conclusions, across two years of experience.

3 The Participatory Approach to the Design of Games with Children

Gamified Co-design with Cooperative Learning (GaCoCo) is the PGD approach that was used in the 2014 and 2015 studies reported in this paper. Conceived in [21], it was incrementally refined across studies for allowing primary-school children to design games with researchers and their teachers in GD sessions of c. 2 h, spread over different weeks, e.g., [7].

In GaCoCo, children work in groups of 3–5 members to conceptualise and prototype their game ideas. In line with [22], a teacher acts as intermediary between researchers and her or his class. At the start of a design session, teachers illustrate the work organisation and material to be used, according to the session protocol. Moreover, they are in classroom together with GaCoCo researchers during the entire design activity: teachers assist in the communication with children for the scaffolding of groups’ work, following a specific GaCoCo protocol for them. GaCoCo researchers have different types of expertise and roles. One is the GD expert, who delivers each group formative feedback through dialogue during a design session, and through written comments in between design sessions. The other, experienced of child development, is the education expert. This acts as observer during design sessions and maintains a constant dialogue with the other adults concerning children’s well being.

Last but not least, GaCoCo uses cooperative learning, an educational methodology based on constructivism, and gamification for engaging all children in the design work, as explained in [23].

4 Design of the 2014 and 2015 Studies

4.1 Study Design in 2014: From a Given Story to Group Games

4.1.1 Participants and Aim

The GD study of 2014 involved two classes from two different primary schools in North-Eastern Italy. Children were, in total, 35 (59% females), coming from a variety of socio-economic backgrounds. Classes were of different ages and sizes: at the start of the study, the younger class was of \(n = 15\) children, in grade 3, with mean age \(M = 8.85\) years, \(SD = 0.44\); the older class was of \(n = 20\) children, in grade 4, with mean age \(M = 9.72\) years, \(SD = 0.47\). All children participated on a voluntary basis, and their parents authorised their participation through a written consent form. The study involved 2 researchers and 2 teachers. Roles were in line with GaCoCo and, in line with it, teachers and researchers divided children in small groups of 3–5 members, heterogeneous in terms of learning and social skills, before the GD process started.

4.1.2 The GD Process

Each group of children of a class was asked to work on a game, composed of two levels. Technology-related choices were fixed: children were asked to design games for tablets. Games had to be designed as avatar-based, starting from a storyline. This was chosen by school teachers and designers together, and read at school as part of a traditional instructional activity.

The GD process was organised in line with GaCoCo. In particular, cooperative learning strategies for small heterogeneous groups were set in the GaCoCo protocol for children. Different cooperative learning roles, such as those of ambassador and secretary, were assigned by teachers to learners; roles rotated among group members so as to ensure that all children had a chance to train different skills. Rules for managing group work were explained to the class by their teacher.

The GD process in each school took a total of four design sessions, whereas the first session mainly served to create the identity of groups. A session lasted circa two hours and a half, and sessions were spread across different weeks. Each session was presented as a mission to children, with its own products as goal, generative toolkits and gamified probes, such as a progression map, for conveying a sense of progression, control and cooperation; these are explained in [24]. Missions followed a recurring pattern. At the start of a mission, the teacher recapped what children had produced at the end of the previous mission (if any) and outlined the goal products of the daily mission. Then each mission continued with its specific tasks. In particular, from the second mission onward, each group performed design tasks and released incremental GD products, that is, a GD document and prototype. Each mission and its products are detailed in the remainder.

First Mission: Group Identity. All children were trained by their teacher to cooperative learning rules and roles. The GD expert explained them how GD would work using metaphors. Finally, each group was assembled and worked on creating their identity, e.g., each group created their own badge, which served to track their progression on a progression map.

Second Mission: Group Game Idea and Avatar. Each group released the so-called high-level concept document of their game, containing their game idea. The document was structured as a form with scaffolding questions. Starting from the story read in class acting as storyline, each group was asked to create their game idea for continuing the story by filling in the document. Afterwards groups worked on prototyping the game avatars and their objects, using a specific template.

Third and Fourth Missions: Group Levels. Starting from the high-level concept document and avatar prototypes, each group conceptualised two game levels, working first in pairs and then sharing results in group, releasing the chore mechanics document for the levels. The document was again structured as a form with scaffolding questions. The document was used for prototyping levels, again first in pairs and then sharing results in group, using the avatars and objects prototyped in the second mission.

Fifth Mission: Group Passage Conditions. Groups of children firstly conceptualised the passage conditions between levels, filling in the so-called progression document. The document was again structured as a form with scaffolding questions. Secondly groups assembled their levels into a single game, using an ad hoc frame. See Fig. 1. They also presented their game to peers.

Fig. 1
figure 1

Game prototypes of 2014 (left) and 2015 (right)

Full size image

4.2 Study Design in 2015: From a Class Story to a Class Game

4.2.1 Participants and Aim

The GD study of 2015 involved two classes from another primary school in North-Eastern Italy. Children were, in total, 42 (45% females), coming from a variety of socio-economic backgrounds. Classes were in grade 4: one class was of \(n = 19\) children, and the other with \(n = 23\) children, with mean age \(M = 10.02\) years at the start of the activity, \(SD = 0.35\). All children participated on a voluntary basis, and their parents authorised their participation through a written consent form. The activity involved the same two researchers of 2014, and two new teachers. Before starting the GD process, classes were again divided into small heterogeneous groups of 3–5 members.

4.2.2 The GD Process

As in 2014, children were asked to design avatar-based games, starting from a storyline. However, the overall aim of GD in 2015 was a single game per class: differently than in 2014, in 2015 each group worked on a single level of the single game of their class. The storyline of the game was created by the class as specified below, and not assigned by adults as in 2014.

The GD process was again organised in line with GaCoCo. As in 2014, cooperative learning strategies for small heterogeneous groups were set in the GaCoCo protocol for children. Cooperative learning roles for children were explained and used like in 2014.

As in 2014, the GD process in each school took a total of five design sessions, presented as missions to children, and the first session mainly served to create the identity of groups. Also in 2015, each mission had its own products as goal, generative toolkits and gamified probes, e.g., a progression map, see [24]. Missions followed a recurring pattern also in 2015.

However, given its slightly diverse design aim, the 2015 GD process differently scheduled GD tasks and products in missions. Specifically, training to GD was spread across missions, and not limited to the first mission as in 2014. The first two missions required class work more than group work for designing. Missions in 2015 were concluded with a sixth mission involving other children in assembling the levels of each class in a single game (see [8]). The main differences in missions of 2015 with respect to 2014 were as follows.

First Mission: Group Identity and Class Stimulus Cards. As in 2014, children were trained by their teacher to cooperative learning rules and roles; then each group was assembled and worked on creating their identity, e.g., their badge for the progression map. Differently than in 2014, the entire class was then engaged in a brainstorming, run like in cooperative learning. The brainstorming aimed at eliciting alternative ideas, written on so-called stimulus cards, at the class level, for creating alternative storylines for the class game, concerning a history topic previously discussed at school—eating habits in ancient cultures. The two class teachers, the GD expert and children worked together. Teachers were responsible for guiding and moderating the class in the brainstorming process; the GD expert organised children’s ideas on the brainstorming panel, whereas the observer recorded them to produce so-called stimulus cards. Second Mission: Class Storyline. The cards of the first mission were used at the start of the conceptualisation stage of the second mission: each group selected cards for creating a group storyline. Each group released the group storyline document. The document was structured as a form with scaffolding questions. This was composed of four cards, corresponding to the main structures of narratives [25]. Afterwards, starting from the cards of the group storylines, the class worked under the direction of the GD expert, as in a focus group in order to create a single storyline document for the class game.

Third, Fourth and Fifth Missions: Group Level. Starting from the class storyline, each group ideated, conceptualised and prototyped a single game level, working like in the second, third and fourth missions of 2014. See Fig. 1.

5 Results of the 2014 of 2015 Studies

Data concerning the quality of GD products by groups of children were gathered at specific moments and analysed per group, as explained in details below.

5.1 Data Gathering

An expert review of an interactive product is an inspection method conducted by experts instead of users, usually in the early design. Expert reviews allow researchers to inspect issues for improving on design, e.g., see [26] for a general overview, and [27] for a working example. In 2014 and 2015, two GD experts evaluated the groups’ GD products released at the end of each mission at school; one was the GD expert present at school. They used heuristics of [28] for informing and uniformly structuring their evaluation, as well as their GD expertise.

The GD experts tracked all the encountered negative issues, in brief, issues, reporting them in a structured format, mission per mission. The evaluation results were provided as formative feedback to the groups of participating children in the follow-up mission, in written format.

After both studies at school, issues in GD products were categorised inductively with a thematic analysis. As suggested in Chap. 5 of [26], gathering issues in categories is useful when certain areas of products are “causing the most issues”; categories should be few, “typically three to eight”. The thematic analysis for discovering categories was run inductively and incrementally as follows, with peer reviews. Firstly, preliminary themes were created by the two GD experts. Then the emerged themes were independently assessed by two game developers. Finally, themes were jointly revised and refined into categories by maximising agreement so as to ensure that they could be applied consistently across GD products by groups of children [29]. Disagreements were resolved through discussions. The resulting categories of issues are as follows.

  1. 1.

    Elements. Elements are conceptualised in GD documents, such as secondary characters and objects, but not used in prototypes. Elements without specific functionalities, or inconsistent with other game elements, are also present.

  2. 2.

    Goals. The goals set in the high-level concept documents or in game levels are unclear, or inconsistently aligned in the final GD product.

  3. 3.

    Storyline. The interplay between the gameplay and the storyline is not maintained, e.g., no storyline elements are used in the gameplay.

  4. 4.

    Player. It is unclear what the player’s role is and how the player interacts with the game, e.g., if the player is the avatar or another character.

  5. 5.

    Incompleteness. Gameplay or mechanics information is missing, which was requested explicitly, e.g., in GD documents. Specifically, children do not specify how to tackle challenges for winning or losing, or how to pass between levels.

In 2014, all the above categories of issues were applicable to the products of all missions from the second onward, except for incompleteness, which was only applicable from the third mission onward.

In 2015, categories of issues were applicable to products in missions as follows: elements and storyline, from the third mission onward; goal, only in the third mission; incompleteness from the fourth mission onward.

5.2 Data Analyses

The two GD experts assessed the products by groups of children in 2014 and 2015 against the above categories of issues, whenever applicable in a mission.

Specifically, using categories, “quality of product” of a group in a mission was defined and operationalised as follows. If a category of issues was applicable to all the products of a mission, then: if a product presented an issue in that category then the product received a negative score of 0; else the product received a positive score of 1. Next, the quality of a (GD) product in a mission was computed as a proportion: it is the sum of scores across categories, divided by the total number of categories applicable in the mission.

The division by the number of categories of issues, applicable in a mission, is a normalisation that enables to compare the quality of products across missions in a uniform manner, on a scale from 0 (worst) to 1 (best).

Using STATA 12.1 for Windows, descriptive statistics and intercorrelations for the quality of products were calculated as follows, per study.

The 2014 study. Table 1 reports the quality of product per group and per mission, mean (\(M\)) and standard deviation (\(SD\)) across groups in 2014, which indicate an evolution in quality of products over time. A non-parametric Friedman’s test of differences among repeated measures was then conducted on the quality of products, and differences were found significant over time: \(\chi^{2} (3) = 15.038\), \(p = 0.002\).

Table 1 Quality of product in 2014 for each group product and mission, from the second onward; means (M) and standard deviations (SD) across group products
Full size table

The 2015 study. Table 2 reports the quality of products per group and per mission, mean (\(M\)) and standard deviation (\(SD\)) across groups in 2015, which indicate an evolution in quality of products over time. A non-parametric Friedman test of differences among repeated measures was also conducted on the quality of products, and again differences were found significant over time: \(\chi^{2} (2) = 12.929\), \(p = 0.002\).

Table 2 Quality of product in 2015 for each group and mission, from the third onward; mean (M) and standard deviation (SD) across groups
Full size table

Therefore, according to the conducted analyses, the effect of time on the considered quality of products was significant in both the 2014 and 2015 studies.

6 Interpretation of Results and Conclusions

This paper reported two PGD studies with primary school children, one in 2014 and one in 2015, both fragmented over different weeks of work and conducted with the GaCoCo approach. The paper operationalised and assessed the quality of products by groups of children released at the end of missions (design sessions), and their evolution over time, in both the 2014 study and the 2015 study.

According to the conducted data analyses, in both years, the considered quality of products tended to significantly increase over time. The decrease may be due to the continuous GD expert’s and peers’ feedback given to children. Therefore the formative evaluation of children’s products, mission per mission, seems promising for improving their quality, and should be maintained in future PGD experiences at school.

However, in the last mission, GD products still presented issues, e.g., they were still incomplete or had unclear functionalities of GD elements, as it is the case of powers attributed to game characters upon overcoming obstacles. Therefore children’s products were in general clear but still in need of design revisions before being used as specifications for their development.

A subset of categories of issues were recurring across years 2014 and 2015. Such a situation may be due to the cognitive skills of 8–10 year olds. On the other hand, the remaining issues in children’s products may well be due to the choice of generative design toolkits in both years. For instance, in both years, groups released GD paper-based documents and prototypes, which, alone, were not sufficient in for conveying interaction information. In future GD processes with children, their GD documents may be completed by GD experts, so as to fix remaining categories of issues concerning unclear functionalities and incompleteness of gameplay and mechanics elements. The GD expert, in classroom with children, seems a promising candidate for completing GD documents before passing them on to developers, in an act of collaboration across generations of participants.

Finally, we acknowledge that the work reported in this paper has its own limitations. Firstly, only two GD experts were involved in the assessment of the quality of products by groups of children, which may have created biases in their assessment. Secondly, this work was limited by the lack of adequate evaluation heuristics or guidelines for assessing early GD products, which may have affected the generality of our results. For instance, having heuristics would have allowed us to count issues for a category and a product, instead of indicating the presence of an issue for a category and a product as done in this paper.

However, this paper indicates the creation of guidelines or heuristics for assessing early GD products as a promising research direction [30]. Moreover it also purports the need of evaluation heuristics or guidelines specific for children’s GD products. The categories of issues presented in this paper may be used as starting point for their creation.