Keywords

1 Introduction

While developing the serious game “Composites Cup on Tortuga” the target group (students of the 8th grade and teachers of general public schools) was taken into account. The development was formatively evaluated. During the evaluation process, different qualitative and quantitative methods were used. The game testers received questionnaires to give feedback to the respective stage of the development of the game. A so-called “World Café” format [1] was carried out, where students could discuss their experience with the game. Expert interviews and feedback discussions were held with teachers and the so-called “Kraken” was used. This client-server-application can test to see if the information collected through common instruments really matches the actual usage of the game. In this paper, the experience with this evaluation instrument is illustrated in detail. The serious game was designed for the school student laboratory [2]. With the support of this game, students should playfully learn about fiber composite materials independently. Until now, the school lab visitors went through learning circles about fiber composites to receive an introduction to the topic, but the learning circle was rather high maintenance regarding the supervision of the student. Furthermore, students were only able to look at the individual materials (fiber and matrix) and then again at the finished fiber composites. In addition, changing characteristics, which caused the merging of these different materials, could not be visualized clearly. Such serious games can be assigned as homework in school, which was another aspect taken into consideration for the development of such a digital medium. Therefore, students would have more time on the practical work during their school laboratory unit. So far, class material that could replace this introductory tool was missing in German schools.

The focus on fiber composites was not only chosen because there is a school laboratory tool for students available at the University of Augsburg, but because lightweight material, which consist of fiber composites, in the course of the sustainable use of resources, will play a big role in the future. Nowadays, there is no aircraft, which does not contain fiber composite materials [3]. In the production of electronic automobiles, lightweight materials are also primarily utilized [4].

In the serious games composites cup on Tortuga, the focus lies on the most commonly fiber composites used for industry and every day applications (sport and leisure) with glass fiber reinforced plastic (GRP) and carbon reinforced/carbon dioxide reinforced plastic (CFK) [5]. The plastic is called matrix in a technical jargon [6]. In the game, the students should get to know the characteristics of fiber composites and apply the acquired knowledge in transfer tasks (for example the creation of a diving board).

2 Design Based Development of the Serious Game “Composites Cup on Tortuga”

The serious game “Composites Cup on Tortuga” was designed in an interdisciplinary cooperation of prospective informatics (students, who programmed the game), an educator, who created the content of the game and evaluated the development, as well as academic experts (teachers, composites experts and usability students).

Technically, the game was put together in the development environment Unity. In order for the game to be usable beyond the school laboratory, two versions, one for iPad application and one for desktop application, were created. The development process was evaluated with the design-based research approach taken into account. Design-based research normally goes through phases of problem definition, the development of a didactic design, the cyclical design implementation as well as the evaluation and reflection in tight cooperation of science and practice. Additionally, the goal is to address the development of practically relevant interventions and the advancement of scientific theories [7].

For the description of the serious game “Composites Cup on Tortuga”, the model of research and development cycles within the design-oriented research of Euler was used [8]. The sequence “design-test-analysis-redesign-test” was repeated multiple times (after every test with the target group).

Description of the Serious Game “Composite Cup on Tortuga”

The main information on the serious game is listed in the following Table 1. The description is based on the game description of the Serious Games Information Center (SG-IC) and oriented on the DIN SPEC 91380 [9].

Table 1. Description of the serious game according to SG-IC
Full size table

For the setting of the game, a pirate scenario was chosen, since the background story was compatible with this context. There are many possible applications to integrate fiber composite materials in such a setting. Furthermore, the game aims to address adolescence as of 13 years old, who were questioned about the setting.

The game is divided into four levels, which are called “Harpooning” (“Harpunieren”), “Prosthesis Run” (“Prothesenrennen”), “Plank Jumping” (“Plankenspringen”) and “Cannonball Trick Shooting” (“Kanonenkunstschießen”) in the game. Within these levels, there are consecutive tasks and quiz questions to deepen the newly acquired knowledge.

Description of the Learning Objective

The following learning contents are to be transported over the individual levels:

Harpooning: In the first level, the players have to solve four tasks about the topic fiber types and their characteristics. Following characteristics are addressed: load capacity, current conductivity, diameter, resilience and costs of the fibers.

Prosthesis Run: In the second level, the matrix is introduced and the tasks address the characteristics of fiber composites. It is visualized through a pirate with an artificial leg, which consists of fiber composite materials, who has to solve three challenges.

Plank Jumping: The third level is the hardest. The players have to apply the knowledge they previously gained in three tasks for the creation of a fiber composite. Mechanical characteristics of the materials as well as the fiber direction, the bending strength and the interaction of these characteristics have to be taken into account.

Cannonball trick shooting: The cannonball trick shooting is all about the thermal resistance of fiber composite materials. There are two tasks, in which the player has to choose the most heat-resistant matrix material.

The limitations of the fiber composite materials are addressed in all levels. Therefore, the costs of fiber composite materials out of CFK and GRP are pointed out and it is shown that not all fiber composites are suitable for all application areas. In a timely and content related restricted game like this one, this can only be addressed in a limited fashion.

The following example describes the design of a task. In the beginning of each task, the gamers receive an introductory text in the form of an inserted panel. See Fig. 1 for an example.

Fig. 1.
figure 1

Example of an introductory text before a task

Full size image

The aim of the level is visualized through a short dialogue with speech bubbles. These also refer to new user-interface-element (UI-elements in the following) and describe the user interface in the beginning of the game. The UI-elements are marked with a red circle or underlined with red. The player cannot continue unless all UI-elements have been clicked. Figure 2 shows the structure of the user interface using the example of the first task of the “Harpooning” levels.

Fig. 2.
figure 2

Structure of the user interface (Color figure online)

Full size image

Information texts support the solution of the individual tasks, which are accessible in every task. To change the characteristics of the tool, the player can choose either dropdown menus or sliders. Each change made influences the tools in the tool bar on the right hand side of the game. Only tool characteristics that are hard to visualize in the mini-game are shown in the tool bar, such as weight, costs and durability of the tools.

Parameter Search Space of the Levels

To find an ideal material combination is comprehended as an optimization of the parameter space. They can have the following dimensions: fiber material, diameter, matrix material, fiber direction and wall thickness.

The players receive direct feedback in the game, if they did not choose a fitting material combination, for example. See Fig. 3 on the following page.

Fig. 3.
figure 3

Direct feedback in the game

Full size image

When the players successfully solve a task, their performance is rated with one to three stars. They therefore receive another direct feedback from the game to see how well they managed to solve the task.

Quizlevel

The integrated quiz tasks in the individual levels were implemented to ensure the results [10] in the game. The students have to answer two to four questions with a maximum of four answer possibilities. If they chose the wrong answer, a short hint pops up to inform them as to why the answer was wrong. If too many questions are not answered correctly, the quiz has to be repeated.

3 Description of the Evaluation

The complete gradually evaluation of the game was conducted with N = 185 students (37% female, 58% male, 4% did not wish to enclose their gender). See Fig. 4 on the following page. The examinees were in different grades and from different types of schools. All participating individuals were able to choose whether they wanted to play the game as well as if they wanted to give feedback to the game. The traditional introduction via the learning circle was also a selectable option. The majority of the students happily participated in testing the game. The feedback was queried through different tools. These include questionnaires to give a feedback to the perspective phase of the game; a so-called “World Café” format was conducted, in which the students could discuss their experience in the game and expert interviews and feedback discussions were held with teachers. Furthermore, the so-called “Kraken” was utilized. All responses were anonymous. There was no assignment made, which test person played with which iPad.

Fig. 4.
figure 4

Distribution of the participants according to their grade

Full size image

The target group was deliberately chosen from different types of schools and different grades, as the student school laboratory can be booked just as individually. The focus target group however were eighth graders. The reason for this was that fiber composite materials are part of the curriculum “LehrplanPlus” (Curriculum Plus) of the Bavarian secondary schools as of the school year 20/21 [11] and is expected to increase user traffic at this age level. In the following, a survey instrument of the evaluation, the client-server-application “Kraken” is to be described.

Description of the Evaluation Tool “Kraken” (Game Event Logging)

In the last third of the evaluation, a module called “Kraken” was developed for the automated, anonymous player data collection. Its data supports the other feedback tools used in the process. These technical evaluation tools, which support the analysis and the understanding of the player behavior, provide an important contribution for the conception of serious games [12]. The acquired data can support the improvement and correction of possible design issues in a timely manner [13]. This way, the effectiveness of the serous games can be improved.

The data collection functions via a, for this purpose developed, client-server-application called “Kraken”. It consists of three parts:

  • Unity Plugin [14]: The Unity Plugin records, collects and sends events, if there is enough data, via Web-Request to the server. Since not all rooms in the school student laboratory have access to internet, the data is buffered in a file and sent, as soon as the server connection is restored.

  • “Kraken” Server: the server consists of a web interface and a MongoDB-database, which saves incoming events. Table 2 (on page 9) shows an extract of the saved events for one run-through of the first task in the cannonball trick shooting level.

    Table 2. Extract from the saved events of the cannonball level run.
    Full size table

  • “Kraken” Client: For all data, there is a python-library available, which can read the “Kraken” server and is available for the evaluation. Figure 5 shows a diagram of the system

    Fig. 5.
    figure 5

    Visual presentation of the “Kraken” system

    Full size image

The Event Format

The event format consists of seven parameters, which are explained in the following. With the GUID (globally unique identifier), the game structure can distinctively be identified. It is generated anew with each start of the game and restricts conclusions of personal data of the test person. The Session Number increases with each new start of the game and is continued from the previous memory state. Together with the parameter Time, which indicates the start of the game, the time used by the students for each program section can be measured. These two data sets do not provide any possibility to restore the exact time of events.

The Level Name indicates the currently played level, once the event is recorded. The event name has to be set to indicate the name of the event. It can optionally be expanded by a string parameter name and float parameter value. The “Kraken” Unity Plugin automatically fills in the parameters GUID, Session Number, Time and Level Name.

Table 2 additionally shows a short example for the events, which are recorded in each level. When choosing such events, the possibility of the failure to record in all levels was taken into consideration. It can be noticed in the Log that, after the introduction (IntroductionDelivered), experiments with different material types (ChangeDesignPart.Material/ChangeDesignPart.Matrixmaterial) were made.

Additionally to the Logging of the gameplay, the recording of the results of the quiz level are also part of the range of functions of the Loggers.

In the following, it is shown how the tool “Kraken” was implemented during the evaluation of the serious game “Composites Cup on Tortuga” with an example.

Practical Usage of the Evaluation Tool “Kraken”

Example: Jumping mechanism in the “Plank Jumping” levels

Design: In the initial implementation of the jumping mechanism in the “Plank Jumping” levels, the players had to tap for each individual jump. In doing so, the game character (male or female pirate) would exercise a downward aimed force on the plank as long as the click is held, in order to strain it. Letting go of the click gave the game character an upward impulse. Clicking or rather, tapping at the right time and for the right duration can make the plank swing sufficiently for the character to jump high enough. See Fig. 3 to see how the level is visualized. During the design of the individual tasks it was taken into consideration that it should not be demanded too little of the players, whereas they should not be overstrained either. To boost learning processes and motivate the players, a serious game should be a challenge, yet at the same time should be adapted to the capabilities of the learners and therefore not overstrain them. The optimal degree of severity can therefore be describes as pleasantly frustrating. Thus, it is based slightly above the learners’ competences, but is still manageable [15].

Result of the testing: The mechanism was strongly demotivating for most testers, since they not only had to choose the right material, but also needed a good jumping technique. It was frustrating for them, as they might have chosen the right material, yet the jump was not satisfying. Some players started doubting their choice of material and changed it again. This process became more frustrating the longer it went on.

Analysis: Problems with this jumping mechanism were identified during the observation of a player and the evaluation of the “Kraken” data. For these levels, the “Kraken” evaluation showed a significantly longer playing time. Thus, as can be seen in Fig. 5, more than half of the players of the old version needed a lot longer for the second and third “Plank Jumping” task.

Redesign: The first steps were to change the jumping characteristics of the materials. Therefore, the players were able to make the plank swing easier and it was not as hard to jump the needed height. However, some testers were still frustrated with the levels. Since optimizing one’s jumping technique made it much harder to find the optimal jumping board parameters and distracted the players from the main task, a much simpler jump mechanism was chosen, which greatly simplified the optimization of the jump technique without completely renouncing an additional small playful component. In the final version, the players merely have to choose the right starting point of the jumping process by one tap and the pirate then jumps on its own.

Result of the Redesign: The reduction of the level of severity in the “Plank Jumping” level had two outcomes. The time spent until success reduced significantly, as can be seen in Fig. 6. However, by reducing the level of severity, the replayability of the levels sank as well. The players did not have to test and try as much to be successful. Yet, the answers to the quiz level questions improved. This allows for the conclusion that the students had an easier time learning the intended teaching content with the easier jumping mechanism.

Fig. 6.
figure 6

“Kraken” evaluation of task 2 and 3 of “Plank Jumping” (“Plankenspringen”)

Full size image

4 Conclusion

During the usability engineering, information from the current application is levied by the Logging, to enable the examination of the interaction between system and user behavior. Time periods can be measured together with the interaction points in the software, to identify problem areas with a statistical evaluation. This is of particular advantage in games, because many interactions cannot be broken down to generally standardized methods, such as buttons. An example is the use of a plank with the game character, where there is a variety of possible interaction methods that should be compared in terms of their usability, as explained in this paper. Such analyses provide objective data about the behavior of the players in relation to the design elements and the learning through games. The visualization of the learning behavior enables a comparison between educational intents and the actual player behavior during the serious game [16].

In conclusion, the use of different qualitative and quantitative evaluation tools proves itself. The use of the technical evaluation instrument “Kraken” provided the significant impulses. On the one hand, this data collection tool could exclude the phenomenon of social desirability [17], because one person does not instruct the tested students and the evaluation runs in the background. The data was generated so the individual player is able to solve the task. However, one has to note that the development of the “Kraken” system, the effort to put on the server and to evaluate the event data, is related to a considerable amount of time. From the point of view of the development team, it only makes sense to use this technical evaluation tool if information should specifically be collected for a specific game sequence.