Keywords

1 Introduction

Artificial Intelligence (AI) is profoundly transforming our workforce around the globe. It is critical to prepare future generations with basic knowledge of AI, beginning with childhood learning. Given the limited research on this topic, currently there is little possibility of grounding the design of learning experiences in evidence-based accounts of how youth learn AI concepts, how understanding progresses across concepts, or what concepts are most appropriate for what age-levels. Given the packed course schedule of K-12 students, being able to connect AI learning to existing Science, Technology, Engineering and Mathematics (STEM) subjects becomes a more realistic approach to embed AI education in K-12 classrooms. Digital game-based learning (DGBL) is a technology-based approach that has shown promise in promoting student learning, including math and problem-solving skills [3]. There is currently very little research into educational games for youth AI education [1]. In this paper, we will discuss the design and evaluation of an educational game, called ARIN-561, for teaching high-school students about AI. We conducted a series of evaluation studies at high schools in the United States. Results indicate the potential of ARIN-561 to build AI knowledge, especially when students spend more time in the game.

2 ARIN-561

The educational game we have developed, ARIN-561, is designed to teach high-school students AI concepts, prompt them to apply their math knowledge, and develop their AI problem-solving skills. In the game, students play the role of a scientist who sets out on a scientific expedition, but unfortunately crash-lands on an alien planet. In order to safely return home, the scientist begins exploring the planet to gather resources needed to repair the broken ship while uncovering the mystery of the planet. The current implementation of the ARIN-561 game focuses on developing concepts around classical search algorithms (e.g., Breadth-First Search, Greedy Search, etc.). In-game challenges such as searching for missing spaceship parts or cracking passwords serve as natural opportunities for the introduction of search as a topic. The essential concepts, such as space and time complexity of search algorithms, lend opportunities to connect AI to math knowledge familiar to high-school students. Activities in the game aim to achieve three learning goals. The first goal is to develop understanding of how AI algorithms are used to solve problems in the real world. We take the approach of designing AI problem-solving in the game that mirrors real-world AI applications. The second goal is to learn how to weigh the strengths and weaknesses of AI algorithms in order to choose between them for problem-solving. In ARIN-561, each new AI algorithm is introduced as excelling at a task that previous algorithms are less suitable for. As the students progress through the game, further comparisons between the AI algorithms are prompted, pushing the students to take more agency in deciding which one is appropriate for the task at hand. The third goal is for the students to gain high-level understanding of how each AI algorithm works, which is achieved through the difficulty progression of the game-play. For each search algorithm, for example, the students are first provided with a tutorial task that teaches them how the algorithm works, and then walked through the task step by step, with less scaffolding as they progress. Subsequently, the students are presented with a transfer problem from a domain different from the tutorial’s that requires the students to apply what they have learned in the tutorial. Students are provided with less tutorial support during this task and need to apply internalized understanding of how the algorithms they have learned work. Embedded in all the tutorial and transfer modules are quizzes that help students pause and self-assess. The game pauses as the students answer the quiz question and continues when a correct answer is recorded. The quiz questions are part of the in-game dialogue, aligned with the narrative.

Fig. 1.
figure 1

Screenshots from ARIN-561. Top-left: the player character crash lands on a foreign planet. Top-right: the student is presented with a quiz question in a tutorial. Bottom left: the student is scaffolded through the greedy search algorithm in a tutorial. Bottom-right: the student solves an 8-puzzle as a transfer problem.

Full size image

3 Evaluation

To assess how ARIN-561 impacts AI learning for high-school students, we carried out a series of pilot studies in computer science classes at three high schools in the United States. The study is designed to fit in 3 to 4 class sessions (45–55 min long each). In the first session, students completed an online pre-survey, which consisted of items about demographic background, AI Use Type, Interest in AI, AI Knowledge, Math Self-efficacy [2], and Math Knowledge. All scales except the Math Self-efficacy were developed by the research team. The AI Use Type included items such as “When I think about how I’d like to interact with AI in the future, I expect that: I will use AI systems in my everyday life as a consumer, and I expect to USE AI systems as a part of my job.” The Interest in AI scale included questions such as “Outside of school I try to learn a lot about AI.” The assessment of AI knowledge and math knowledge focused on the content covered in ARIN-561. During the second and third sessions, students interacted with ARIN-561 online at their own pace. During the fourth session, students completed a post-survey, which included the same items on interest in AI and AI knowledge from the pre-survey. In addition to the surveys, game logs from ARIN-561 were collected, including the in-game click-stream data and responses to in-game quizzes (Fig. 1).

4 Results

A total of 125 students participated in the studies. The participants’ average age was 16.1 years old. A total of 73% of the students were male, 21% were female and 6% identified as other categories or preferred not to disclose. With restricted access to the school campus due to COVID-19, the data collection was carried out entirely by the participating teachers, without participation of the research team. As a result, 60 out of the 125 students did not complete the assessment of AI knowledge on either pre- or post-survey. Missing data were excluded from the corresponding analysis.

We hypothesized that interacting with ARIN-561 would help students gain knowledge in AI. Thus we conducted a paired-sample t-test to analyze the changes in AI Knowledge from pre- to post-survey. There were a total of 15 questions on AI knowledge (15 points total). Data from all students who completed both pre- and post- AI knowledge assessment (\(N=65\)) showed a positive, though not statistically significant increase of AI knowledge (\(M=0.427, SD=2.819, t(64)=1.221, p=.227\)). Given the varied completion rate of pre- and post-survey, we further examined the game logs from ARIN-561. In particular, students who completed less than half of the game modules (2 or fewer of 6 modules) were then excluded before we repeated the paired-sample t-test on the group of students who completed half or more of the game modules (\(N=47\)). Results indicated that, the group of students who completed at least half of the game demonstrated a statistically significant (\(M=1.0638, SD=2.637, t(46)=2.765, p=.008\)) positive change in AI knowledge, with a mean difference of 1.0638 and a medium effect size (\(d=0.403\)). Additionally, a one-way ANOVA comparing students who completed half or more modules (\(M=1.0638\)) and those who completed less than half game modules (\(M=-1.5469\)) revealed a statistically significant difference in the changes in AI knowledge between the two groups (\(F(1, 61)\,=\,[11.737],\;p\,=\,.001\)).

5 Discussion

This paper presents our approach to designing a game-based AI learning environment for high-school-aged youth and presents evidence for how the game may be contributing to AI learning among players. We observed statistically significant learning gains among students who completed at least half of the game. This suggests that the ARIN-561 educational game can support AI learning for high-school-aged youth, and in order to realize these potential gains, youth should engage in sufficient learning in the game. Given the stark difference between outcomes for those students who completed at least half of the game compared to those who did not, further analyses of game log data are needed to better understand how in-game behaviors may be contributing to learning gains, beyond the dosage effect reported here.