Keywords

1 Introduction

Video games for education are still hardly recognized and used as educational resources and tools for learning. On one side, not many teachers and educators are familiar with possible scenarios for using video games in their teaching strategies [1, 2]. On the other side, the player feedback and game assessment metrics, reported after playing educational video games, do not demonstrate clearly what are the students’ achievements in terms of learning outcomes. This way, despite the accumulating literature for the benefits of educational video games, teachers and lecturers still hesitate to incorporate video games in their teaching [3]. Therefore, selecting appropriate game metrics and game feedback reports should clearly demonstrate how playing educational video games can contribute to achieving specific learning outcomes and competence level, making parallels with other educational activities and tools for learning.

At the same time, contemporary, adaptive and context-oriented video games support teachers and lecturers to design sophisticated learning games, using them as a tool for personalized teaching and better align to the learners’ needs. The present research will focus on how the APOGEE platform (http://www.apogee.online/) aims to support teachers in developing and evaluating educational 3D video maze games enriched with mini-games of various types representing appropriate didactic tasks [4]. Based on the user-centered approach and focusing on teachers and learners, the paper aims to investigate the relevant metrics, game analytics, and feedback reports of adaptive video games for education.

The structure of the paper is as follows. The next section makes a short overview of the main concepts of user-centered approaches and summarizes the main factors and metrics for measuring the user experience in serious games and video games for education. Then, to set a general framework, the APOGEE game-generating platform is described, identifying its specifics, system functionality, and capacity for adaptation and contextualization. Next, the focus is put on the process of collecting and analyzing data about the user and user gaming experiences, including appropriate pre-game self-report, log files with game session data, and post-game questionnaires. Based on empirical investigation, the main feedback metrics of the APOGEE video game for education are defined, considering both the relevant data for the maze game and the embedded mini-games. In the end, a more specific analysis is made on the main features of the post-game questionnaires and the design of the feedback reports for teachers and learners.

2 User-Centered Design for Evaluating User Experience in Educational Video Games

A short overview of the literature shows considerable research on models and methods for evaluating user experience in serious games and, more especially, in educational video games. Many authors propose various scales and metrics, considering specific aspects of the learning and gaming experience of the players. However, in the APOGEE platform, the main designers of the adaptive video games for education are teachers and lecturers. Furthermore, many of these teachers and students, designing and using games for learning are not supposed to be gamers. Therefore, teachers (in the role of game designers) need to understand: (1) whether one game is good enough to be used as a learning tool; (2) what are the students’ preferences as learners and gamers, and (3) how effective for the learning is the designed video game. Furthermore, the games reports have to provide appropriate data and analysis for the teachers to adopt more personalized and student-oriented teaching strategies.

2.1 User-Centered Design and User Experience in Video Games for Learning

The main focus of the user-centered design (UCD) is, as part of the human-centered design (HCD), to estimate the needs of the end-users. Human-centered design is distinct from the traditional design practices because it prioritizes the people for whom the product, system, or service is intended, rather than the designer’s personal creative process or the material and technological substrates of the artefact [5]. In the HCD perspective, the learner is put in the center of the education system, while the teacher is the provider of an appropriate learning experience. UCD is thus a multistage problem-solving process (not restricted to interfaces or technologies) [6] in which the needs, wants, and limitations of the end-users of the product, service, or process are given extensive attention at each stage of the design process [7]. Furthermore, UCD highlights the role of testing and collecting data about user behaviour with real-world tests with actual users, using both qualitative and quantitative methods.

User experience (UX) is among the key concepts of the UCD and is a branch of Human-Computer Interaction research. Summarizing the literature [1] concludes that UX consists of all (qualitative) experience a user is making while interacting with a product; it includes affections, emotions, beliefs, and expectations that occur before, during, and after use of the product; it is directly and closely related to attaining the user goals. Further, [8] states that UX incorporates factors like effectiveness and efficiency, along with criteria like attractiveness, aesthetics, or joy-of-use, and UX focuses on the interaction between the users and the products or systems, and this interaction is depending on the surrounding elements.

Based on multiple studies, the UX in educational video games can cover the following factors:

  • Gaming experience, learning experience, adaptivity, usability [9, 10];

  • The quality of the product (game system experience), the quality of human-product interaction (individual player experience), and the quality of this interaction in a social, spatial, temporal, or another context [8];

  • Reliability, validity, sensitivity, robustness, non-intrusiveness, and convenience [11];

  • Physical elements (technology, demographic, architecture), social elements (political, cultural, economics), time elements (use the game for a certain time period), involvement (appeal, engagement, adaptation, retention); emotional and cognitive elements (inner emotional level); game content (goals of the game and learnability aspects [12]); fun and flow factors [1, 8];

  • Technology (Usability, Quality of the system), Player (Understandability, Motivation, Engagement, User experience), Community (Social Factors) [13];

  • Gaming experience (Fictional/Sensory Immersion/Affective Valence); Challenge (Challenge Absorption, Ownership); Playability (Variety, Clear Goals, Navigation, Help/Training); Usability (Control, Customisability, Consistency, Camera (Views) Game Interface) [14];

  • Learnability; Efficiency; Memorability; Errors; Satisfaction [15].

2.2 Selecting the UX Concepts for APOGEE Game Experience

Based on the review in the section above, we decided to summarize the main factors for UX in games in three general facets: playability, learnability, and usability.

Playability.

Playability consists of a group of factors such as Gaming experience, Challenge (the player’s perceptions of difficulty of the game), Immersion, Game flow, Affect (confidence, self-efficacy, and attitudes), and motivation to engage [12]. Fidelity in a serious game can be associated with the level of realism that the environment provides to the user. Presentation of a game’s graphics and audio affects player immersion and engagement.

Learnability.

The learning experience is often associated with educational game effectiveness, the setting of clear goals, available feedback to provide a learning opportunity. The game content should be aligned to the curriculum within which it is embedded and educational goals. Content appropriateness and integration refer to game activity, promoting reflection on the knowledge and skills that the game provides to players, as well as encouraging the integration of knowledge from different areas to support player decision making, and develop new knowledge from the game. Media matching within serious games is used to identify the most appropriate media form to use, such as animation, sound, picture, or text within a game.

Usability.

The UX characteristics of usability cover factors such as ease of use of the interface, user control within the gaming environment, the avoidance of errors, and satisfaction with the game’s interactive features [11]. Usability explains how the user can achieve specified goals with effectiveness, efficiency, and satisfaction in a specified context of use. The three components include (1) effectiveness: the accuracy and completeness with which players achieve specified goals - measured in completion rates and errors; (2) efficiency: the accuracy and completeness of goals achieved concerning player effort and resources such as time on task and number of trials; (3) satisfaction: the positive attitudes and experience from the use of the system.

In addition to factors of UX, that will be evaluated in the framework of the APOGEE game development, the following three questions (Table 1) will be considered in the game assessment: what should be assessed (what variables do we want to measure), why should be assessed (what is the purpose of collecting this information), how it will be assessed (the process of assessment).

Table 1. Factors for Evaluating APOGEE Video Games for Education
Full size table

3 APOGEE Software Platform for Educational Video Games

To set the general framework of the present study, we base the current research on the APOGEE software platform for the generation of personalized educational and adaptive video games. The platform is described in previous publications [16, 17], identifying its specifics, system functionality, and capacity for adaptation and contextualization. One of the main features of the APOGEE game platform is that it will enable non-professional IT experts such as teachers and lecturers to design and make their own educational video games. The game engine provides a pre-defined maze, but the game designers can modify and personalize the number of the maze halls and their elements, as well as the educational mini-games, located in each hall. Based on the APOGEE game adaptation process, the game flow and the difficulty of the mini-games can automatically adjust, based on the data from the user profiles and user in-game experience. An extensive learning analytics engine allows teachers to get more information about students’ learning and playing style and how this can contribute to learning adaptation inside and outside the game.

Figure 1 presents an overview of the APOGEE process of game creation and user data collection (top), game play and UX evaluation (middle), and game analytics and learning evaluation (bottom), which is developed based on [16]. The process is described step by step as each step provides a certain functionality and opportunity for the users of the APOGEE platform. After Game Design Evaluation, the process may optionally cycle on game design modification and further validation, generation, build and deploy of a new version of the same game.

Fig. 1.
figure 1

The APOGEE game development and evaluation life cycle

Full size image

Adopting a User-Centered design approach, the APOGEE game designers - teachers and lecturers, need to get better information about (1) the learning and playing experience of the students; (2) the general evaluation of their video game; (3) the learning outcomes with the game platform (that can be later compared to other learning activities and materials). In this perspective, teachers need to get relevant data and analysis reports, allowing them to fine-tune the mini-games and evaluate the efficiency and effectiveness of students’ experiences in the maze game.

To cover all phases of the UCD, the APOGEE platform proposes a three-level process of collecting and analyzing data about the user and user gaming experiences. The data collection mechanisms include appropriate pre-game self-report, log-file data, and post-game questionnaires. In plus, teachers can organize students’ observations, focus groups, and interviews, to get more insights and inspiration about how to design entertaining and efficient educational video games. The pre-tests are independent of the game experiences and thus learners can fill them only once when registering into the system. The log files collect data about the in-game experience. The post-game questionnaire can on one side evaluate the game learnability, playability, and usability, and on the other side, a specialized questionnaire is provided for assessing the usability and effectiveness of every mini-game.

The whole process of collecting data APOGEE platform includes 4 questionnaires as follows: Pre-game: a questionnaire for the personal characteristics of the player having 10 questions about age, gender, fun game and learning game experience, learning and playing goals, and initial knowledge in the learning domain; the VARK questionnaire on the player’ learning style with 16 questions and the ADOPTA questionnaire on the player’ playing style with 16 questions [18]; and a post-game experience questionnaire on learnability, playability, and usability of the game with 16 questions. All these questionnaires can further provide data to compose more relevant metrics and feedback reports based on descriptive characteristics (mean, deviation, and standard error), correlations, T-tests, effect size, ANOVA, and others between personal student characteristics, students’ learning, and playing styles, and the game outcomes (time, points, effectiveness and efficiency – for both the maze and its mini-games), learnability, playability, and game usability. The game outcomes can summarize the collected points, playing time, effectiveness, and efficiency for the maze game and any puzzle game. But more importantly, game designers – teachers, and lecturers will be able to better understand the impact and the feedback analysis of educational video games.

Figure 2 provides a screenshot of the APOGEE online analytic tool for presenting individual and per-game learning/playing metrics and analytics (correlations, T-tests, effect size, ANOVA, etc.). Based on the statistics about outcomes of the game and self-reports, the system helps teachers to infer (1) what are the students’ preferences as learners and gamers (2) whether the game is good enough to be used as a learning tool; and (3) how effective/efficient for the learning is the designed video game.

Fig. 2.
figure 2

View of the analytics page

Full size image

4 Discussion

Contemporary, adaptive, and context-oriented video games can facilitate teachers and lecturers not only to design sophisticated learning games but also to adopt personalized teaching strategies, better adapting to the learners’ needs. Educational video game analytics should collect data for generating meaningful feedback reports both providing relevant analysis for learners and teachers and considering the gaming and the learning experience. The useful game reports can convert the relevant data into a mix of indicators, covering learners’ knowledge and game experiences before, during, and after playing the game. The process can combine data about the learner’s profile (and his learning and gaming styles), along with other pre-test surveys, log system data, and post-test surveys for examining user experience and user interactions with the system. Furthermore, the game metrics have to be designed in a way to provide relevant data for both end user’s groups – game designers (teachers), and game users (students/learners).

Unfortunately, as discovered in [19] there is no specific recipe for fun. The usability and effectiveness of productivity tools can be measured in terms of production, throughput, efficacy, and efficiency. But other aspects such as learning impact, engagement, or fun are much more subjective and difficult to measure. This subjectivity and elusiveness impact formal usability testing protocols when applied to games. In summary, evaluating the usability of games presents unique challenges and requires metrics and methodologies that aim to contemplate their variability and subjectivity of interacting with games, as well as their uniqueness as exploratory experiences [19].

5 Conclusion

The design of serious games for education is a complex task in which designers need to create products that engage the audience and provide an engaging learning experience, weaving gameplay features with educational materials. Allowing teachers to adopt UCD and to explore how they can evaluate and fine-tune educational video games can generate further interest in their use. Furthermore, in-game metrics should not only focus on the main typical usability testing methods, focusing mainly on measurements such as productivity, efficacy, and efficiency as well as low variability, number of errors, and pauses. This way teachers and lecturers will learn that games are good to contemplate reflection, exploration, variety and trial, and error activities.