Keywords

1 Introduction

E-Learning is described as learning driven by digital electronic tools and media [11]. The fast and constant development of technology requires educational institutions to quickly adapt to these changes and include them in education [9].

Gamification and chatbots have emerged as promising tools for enhancing learners’ motivation and engagement in educational contexts [4, 10]. Gamification refers to the use of game design elements, such as points, badges, and leaderboards, in non-game contexts to make them more enjoyable and motivating [7]. These game elements are supposedly part of the reason why games are fun, thus including these in non-game-contexts could be seen as adding motivational elements to existing applications.

Chatbots, on the other hand, are computer programs that simulate human conversations and can interact with learners in natural language. They can answer basic questions [10], send reminders to students and notify them about new learning content [2] or assess the students based on learned content [16].

We claim that the combination of gamification and chatbots in educational settings, referred to as gamified chatbot-assisted learning activities, has the potential to provide personalized and interactive learning experiences that can increase learners’ motivation and improve their learning outcomes. Despite the growing interest in gamified chatbot-assisted learning activities, there is still a need for empirical evidence to underpin their effectiveness and to understand how they can be designed to optimize learners’ motivation and engagement. Therefore, this paper aims to investigate the potential of integrating gamification into chatbots to enhance user motivation and engagement in learning activities while also investigating how some demographic attributes might affect the results. To address this objective, we aimed to gamify a Quizbot by making use of standardized data and formulated the following research questions:

RQ1: How can gamification be integrated into educational chatbots to support Technology Enhanced Learning (TEL)? This involves investigating which technical methods and tools can be used to effectively integrate gamification elements into a chatbot system, how these integrations should be represented and how they can support specific learning outcomes and objectives.

RQ2: Could gamification positively affect the motivation to interact with chatbots? To this end, we investigate whether users feel interested and invested in communicating with a gamified chatbot over a short evaluation. We additionally investigate whether motivation is affected by previous game experience (RQ2.1) and player types (RQ2.2).

RQ3: How high is the acceptance in regards to a gamified chatbot? Here, we explore user acceptance of a gamified chatbot with integrated elements. We additionally investigate whether acceptance is affected by previous game experience (RQ3.1) and player types (RQ3.2).

2 Background and Related Work

The background and related work section presents theoretical concepts used during the implementation and an overview of the existing literature on gamification and chatbots in various contexts, highlighting the different gamification elements used to motivate users to engage with the application.

Aspects of Motivation. Motivation, which is defined as “[...] being moved to do something [...]” [18], is seen as directly linked to learning success [3]. The lack thereof is seen as one of the main reasons for students not engaging regularly with learning content. For that purpose, motivational models have been designed to explain which aspects of a person or a system affect motivation, which can then be used to design learning content or learning systems to aim at fulfilling the different aspects [6, 14]. Ryan and Deci’s Self-Determination Theory (SDT) posits that the three innate psychological needs Autonomy, Competence and Relatedness need to be fulfilled to foster motivation [6]. Similarly, Keller’s ARCS motivational model describes the four components Attention, Relevance, Confidence and Satisfaction as the human conditions needed to be met to foster motivation [14]. Research has shown that gamification has the potential to foster the user’s motivation, which implies that gamification elements fulfill the different components of the motivational models [21].

Gamified Chatbots. CiboPoliBot is a gamified TelegramFootnote 1 chatbot that aims to teach healthy lifestyles to children aged 8–14 [8]. The design process involved using the HEXAD gamification framework to select gamification elements that fit the player types based on learning goals, tasks, and target groups [15]. Then, Fadhil and Villafiorita integrated leaderboards and points into the bot [8]. The bot operates as an interface for a game, where users collect virtual foods, give or throw food away, or eat the food based on the result of a dice roll. Points are rewarded based on the collection of choices, and players are placed on a leaderboard at the end. However, the bot’s effectiveness has not been evaluated, it is currently impossible to conclude its motivational outcomes. Additionally, there is a lack of examples of how points and leaderboards would be implemented in a chat environment.

Escapeling is a gamified Telegram bot for English learning in secondary schools where English is a second language for students [12]. It promotes collaborative learning through three types of English tasks, where learners discuss and solve problems together. The bot engages users with a narrative, portraying them as trapped on an alien ship and requiring English problem-solving skills to escape. In addition to the narrative, the chatbot includes badges, streaks, unlockable content, and displayable achievements. User achievements are displayed in private chats, and new achievements are announced in group chats to activate the innate social comparison mechanism and further motivate users. Two evaluations of the app’s usability, user satisfaction, and success were conducted, indicating high performance and satisfaction scores. However, the evaluations did not emphasize the motivational aspects of the gamification elements, and no visualizations of the gamification elements were provided.

The popular live streaming platform Twitch.tv has gamified its user interface and chat environment for streamers and viewers [13, 20]. In addition to Twitch.tv’s built-in gamification, streamers were previously able to import third-party chatbots such as StreamlabsFootnote 2, DeepBotFootnote 3 and PhantomBotFootnote 4, which offer similar gamification features. Viewers receive points for commenting, following, subscribing and donating, which can be used to unlock special emotes or take part in mini-games that reward more points if won. These bots feature a leveling system, leaderboards, and customizable elements like badges, progress bars, and a streak system, using commands instead of Natural Language Processing (NLP). Despite Twitch.tv’s popularity, there has been no research on the effects of gamification elements or bots on user interaction or viewership.

Discord, a communication platform that offers private and public chat channels, has gamified chatbots like TatsuFootnote 5 and ArcaneFootnote 6 that can be added to any server, allowing users to earn currency points by sending text messages that can be used to purchase roles, badges, or virtual items. Users earn experience points by completing daily quests and leveling up to unlock special server features in both Tatsu and Arcane. Tatsu offers visualizations of progress, a user profile card, and a leaderboard upon request, while Arcane includes an experience point system, leaderboards, and moderator actions.

Table 1 contains a comparison of the previously discussed gamified bots and our proposed system. We noticed that existing gamified chatbots propose gamification elements with interesting designs. However, many of them have not been tested in an educational setting, lack empirical evaluations to support their effectiveness, are not open source, or use non-standardized data. These limitations restrict their potential for gamifying arbitrary chatbot data and reusability. Our solution aims to address and overcome these issues.

Table 1. Comparison of gamified chatbots and frameworks.

3 Concept and Realization

In this chapter, we discuss the implementation of our framework to enable gamification in chatbots, covering the technologies used, the integration of game-like elements and the creation of Quiz-GBot, a gamified Quiz-Bot.

Firstly, we decided to gamify standardized interactions stored in a Learning Record Store (LRS)Footnote 7 database in xAPIFootnote 8 format. In contrast to the previously mentioned gamified chatbots implemented for specific use cases, our approach allows the gamification of arbitrary interaction data using rules based on the content of the data. Regarding the gamification elements, we adopted a design-based research approach to identify and implement suitable gamification elements in the Quiz-GBot prototype. Initially, we pinpointed areas where gamification could enhance user engagement. Using these insights, we brainstormed, designed, and iteratively tested various gamification features, refining them based on user feedback. This systematic, user-centric, and iterative process ensured the gamification elements were not just added for novelty but genuinely enhanced the Quiz-GBot user experience. Some elements, such as achievements and level notifications, were displayed as text, while others like the profile card and the achievement list were integrated as images. We stick to these two basic formats, instead of opting for platform-dependent features, as it ensures cross-platform compatibility and accessibility. The use of text and images still manages to accommodate users with diverse devices and screen sizes while still offering a visually engaging and immersive user experience while maintaining compatibility across all platforms. While platforms like Telegram, DiscordFootnote 9, and SlackFootnote 10 have more polished user interfaces (buttons, checkboxes, select menus, ...), and hence could potentially offer a better user experience, they were not chosen because they are not open-source. This could lead to privacy concerns when using these platforms for learning activities. Thus, our design decisions were driven by a balance between user experience, privacy considerations, and the importance of broad accessibility. To further reason our choices for the implemented gamification elements, we created a mapping (see Table 2) of gamification elements to motivational aspects.

Table 2. Mapping of Gamification Elements to Motivational Aspects

This mapping showcases which gamification element we expect will affect the different motivational aspects to then foster motivation. Note that, it is not guaranteed that these aspects will be affected, as individual users might perceive them differently and the implementation and chosen semantics also play a role.

Points, Levels, Badges and Achievements. Our Gamification Framework enables earning points through actions or achievements, unlocking other gamification elements. When defining levels, the game creator must specify attributes such as the level number, name, necessary points to reach the level, and the notification message. Rules specify the available system actions, which assist users in progressing in the gamified environment and may also indicate the number of points users earn upon action completion. We utilize xAPI statements stored in an LRS to track users’ activities effectively. The xAPI statements offer granular details on each user’s interactions, making them invaluable for monitoring their journey and accomplishments within the game environment. We maintain a structured database record that maps each activity to its corresponding action, including the points it yields. This database not only aids in accurately awarding points but also provides a clear benchmark for the number of points necessary to transition to the subsequent level. A user will automatically advance to the next level once they have accumulated the necessary points. An example is shown in Fig. 1a. For badges, the game creator simply uploads a picture that will be resized by our Gamification Framework, and provides a badge name and ID. For achievements, in addition to the basic attributes such as achievement name, ID, and notification, the game creator must describe the achievement and the rewards that a user obtains upon unlocking it. These rewards can include a point value and/or a badge that will be awarded to the user. Figure 1b shows how a user asks for his badges.

Fig. 1.
figure 1

Screenshots of the gamified Quiz-GBot showing a user: (a) reaching a new level and unlocking achievements, and (b) requesting his earned badge.

Quests and Streaks. The quest element in our Gamification Framework consists of an unlockable achievement and the required actions to complete the quest and unlock the achievement. The game creator can choose which achievement to unlock from the list of existing achievements and which action to perform, along with the number of occurrences of the chosen action to unlock the achievement. Similar to quests, streaks allow the game creator to choose which action causes the streak level to increase. Moreover, achievements that can be unlocked have to be specified for streaks, along with the streak level at which they are unlocked. Streaks also require a duration value to specify how much time should pass between two consecutive action triggers until a streak level is lost. If a streak is lost, the level falls back to level zero.

Player Profile and Leaderboards. Our Gamification Framework provides a player profile card to reflect the current progress of users, similar to the Tatsu Discord bot. To avoid overwhelming users with information, the displayed information on the player profile card was kept minimalistic. The information displayed includes the collected points, current level, required points to reach the next level, number of unlocked and missing achievements, and number of unlocked and missing badges. To add a light form of personalization, the player’s nickname and a badge earned can be added to the profile card. The profile card was implemented as a PNG file that is sent to the user, as this option results in a visually appealing result that is not exclusive to any chat platform. The final design of the template contains an extra placeholder square, which can be used to personalize the profile card by inserting badges at the reserved emplacement and can be personalized by the user. A personalized and completed profile card example is shown in Fig. 2a. Users can also request a leaderboard that displays users’ scores based on a selected activity/metric and helps them compare their progress to other users. The leaderboards can be based on metrics that are present in every application gamified by our Gamification Framework, these metrics being collected points, unlocked achievements and collected badges. Additionally, they can also be based on possible actions in the gamified application, thus being application-specific. On the chat platform, the leaderboard is displayed as a plain text message using minimal formatting containing the three columns rank, number of occurrences/score and the member ID, which is a pseudonym to keep the users anonymous. Figure 2b shows an example leaderboard that displays scores based on points.

Fig. 2.
figure 2

Screenshots of the gamified Quiz-GBot showing a user requesting: (a) his player profile, and (b) the leaderboard.

The entire framework and the Quiz-GBot implementation are open-source and available on GitHubFootnote 11. The gamification component for Quiz-GBot was developed using Java and serves as a wrapper service for our Gamification Framework employed in this project. Its primary purpose is to connect chatbots with gamification elements, enabling seamless integration and communication between the two by gamifying interaction data stored in a LRS database.

4 Evaluation

The evaluation section primarily investigates the effects of a gamified chatbot on motivation, acceptance, and usability. By examining these factors, we aim to determine the feasibility and effectiveness of gamifying chatbots in educational contexts. We additionally compare the results based on the users’ previous game experience and player types, which can provide further insights into the findings.

4.1 Design, Procedure and Tasks

The evaluation consisted of two surveys and an actual testing session. An initial survey captured demographics, while a second focused on feedback regarding the tested project. The participants were invited to individual online meetings, which were held over Zoom. The recruitment process involved posting the study invitation in forums and sending individual invitations to acquaintances, which ultimately resulted in \(54\) participants participating in the evaluation. The evaluation commenced with an introduction to gamification and the study’s objective. The participants were provided with a link to the non-gamified version of the Quiz-Bot chatbot. They were prompted to greet the bot and initiate a quiz, which was offered on a variety of topics, including science, history, and pop culture. While participants engaged with various topics, analyzing the correlation between their selections and their educational background or interests was not within the scope of our research. Upon completing the quiz, the participants proceeded to the gamified version of Quiz-Bot, Quiz-GBot, where they first had to request gamification elements using methods such as selecting options, asking with text, or using a command. The participants were prompted to request the profile card and unlock any achievement. After completing a quiz, participants received notifications of unlocking a new level and achievement and should check their progress on the leaderboard and their profile card. The study concluded with the second survey, which sought their feedback on the experience.

Fig. 3.
figure 3

Game experience and identified HEXAD player types among the participants.

4.2 Results and Comparison

The first survey aimed to gather participants’ demographic information and their previous gaming experience. Out of the \(54\) participants, \(39\) were male and \(15\) were female, with \(44\) participants falling into the age range of 19–25 and ten participants being between the ages of 26–35. Figure 3a shows a breakdown of participants’ previous gaming experience, which was categorized into three groups: those with considerable gaming experience (\(39\) participants), those who were somewhat acquainted with games (\(12\) participants), and those who claimed to have almost no gaming experience (\(3\) participants). Furthermore, the Player Type questionnaire from the HEXAD gamification framework was also included in the first survey to categorize participants into different player types [15]. The results of the categorization can be seen in Fig. 3b. Note that the numbers don’t add up to \(54\), as some people can be categorized as multiple player types. Following the evaluation session, we administered a survey based on existing motivational models, Technology Acceptance Model (TAM), and questions directly comparing the gamified and non-gamified versions of the bots [5, 17].

Table 3. Participants’ responses on a five-point Likert scale (ranging from 1 = strongly disagree to 5 = strongly agree) to questions related to motivational models (\(n=54\)).

Questions Based on Motivational Models. The questions were adapted versions of those used in previous studies, such as the evaluation of gamified E-Learning systems by Shi and Cristea [19] which is based on Self-Determination Theory by Ryan and Deci, and the ARCS model of motivation by Alcasoda and Balaoro [1]. The questions were designed to answer the research questions related to the effects of gamification on motivation. The questions focused on aspects of the motivational models, such as “Autonomy (A1)”, “Attention (A2)”, “Competence (C1)”, “Relevance (R1)”, “Relatedness (R2)” and “Satisfaction (S1)”. Table 3 shows the mean and standard deviation values of participants’ responses to the motivational models’ questions.

Questions Based on Technology Acceptance Model. The second part of the survey aimed to answer the research questions regarding technology acceptance. We used questions that were inspired by a study evaluating an e-learning system for university students [17] with questions based on the TAM proposed by Davis et al. [5]. We adapted these questions to our project while ensuring that they addressed the different constructs that make up the TAM. The results are presented in Table 4, which shows the mean and standard deviation of users’ responses to questions regarding the perceived ease of use, usefulness, attitude, behavioral intention, and self-efficacy of the gamification elements in the chatbot-based learning system.

Table 4. Users’ perceptions of gamified chatbot-based learning activities on a seven-point Likert scale ranging from 1 = strongly disagree to 7 = strongly agree (\(n=54\)).

Questions Comparing Gamified and Non-gamified Chatbot. This last block of questions focuses on comparing both the non-gamified and gamified versions of the chatbot. Out of the 54 participants, 45 preferred the gamified chatbot, 8 saw both gamified and non-gamified versions as equally good and 1 participant preferred the non-gamified version. The participants also could explain their chatbot choice in an open text field. Some found the gamified version more interesting but more time-consuming, which could be a drawback during exam preparation. Others mentioned that the gamified bot felt “unserious” or distracting at times. Yet, regarding the positive aspects, participants mentioned that the gamified version felt “Fun” and more interesting than the non-gamified version which was considered boring by some. Achievements would give purpose to the interactions and let them feel in control of what they want to achieve. The visualizations of progress and immediate feedback on unlocking achievements or reaching a new level were also praised. Some mentioned that the gamified version compensated for the lack of incentives of the non-gamified version. The final two statements directly compared both chatbots in terms of fun and immersion. The high score for the statement “Interacting with the gamified bot was more fun than with the simple bot” (AVG = 4.60; SD = 0.49) confirms that gamification indeed made the bot more to interact with as opposed to the non-gamified version. While fun is not necessarily an indication of a better learning experience, the users’ enjoyment increased, which could potentially increase the motivation to interact with the bot. In terms of immersion, the statement “I felt more absorbed while interacting with the gamified bot than with the simple bot” (AVG = 4.30; SD = 0.66) also ended up with a high average, indicating that generally, participants were more focused on the interaction with the gamified bot than with the simple version. This hints towards the notion that gamification makes the participants concentrate more on the interaction.

Comparison Based on Game Experience. We investigated the impact of previous game experiences on the user’s perception of a gamified chatbot. The results showed that participants who never played games rated the gamified bot with a lower score than the other two groups, suggesting that those who do not play video games may not enjoy a gamified chatbot as much. However, the low number of participants in this group limits the certainty of this conclusion. Interestingly, participants in this group did not feel rewarded for their efforts when receiving achievements and expressed opposition towards the leaderboard. Nevertheless, they still agreed that the gamified bot was more fun and immersive than the simple bot, indicating that gamification can still improve the user experience for non-gamers. When comparing the two other groups, the “Rarely” group answered similarly to the “Regularly” group. Although the difference was often not significant, it still suggests that the amount of game experience does not necessarily correlate with a better perception of gamification. However, the “Regularly” group had a much higher participant number, which may have affected the results. Furthermore, the vagueness of the categorization of “Rarely” and “Regularly” may have led some participants who would rather fit in the more experienced category to classify themselves as more inexperienced participants. Overall, it can be concluded that people with some form of gaming experience are more likely to enjoy gamified elements in a chatbot, which may increase their engagement with the application. Our findings suggest that the gamified bot was well-received among all three groups, indicating that gamification can be a valuable tool to enhance the user experience of chatbots, regardless of the user’s level of game experience. These findings are supported by the results of the bot preference, which showed that the majority of participants in both the “Rarely” and “Regularly” groups preferred the gamified bot over the simple bot, further emphasizing the positive impact of gamification.

Comparison Based on Player Type. At first glance, there do not seem to be any notable differences in the participants’ answers based on their player type. Regarding how the different results match the expected behavior of the HEXAD gamification framework player types, we found some interesting results. The “Socializers”, defined by their need to interact with others, answered most negatively when asked whether they felt annoyed during the bot conversation, as these types of players prefer interaction with real people instead of a computer program. “Achievers”, who focus on mastery and learning to improve themselves, generally felt that learning to use the gamified elements was not difficult and claimed they became skillful at using them. Regarding unlocking achievements, the “Player” type seemed to enjoy them the most, which is fitting as this group is defined as focusing on rewards the most. “Philanthropists” scored the achievements the lowest out of all groups, which again fits as this group is described as not caring about rewards. “Free Spirits” seemed to be the most skeptical about leaderboards, which potentially means that leaderboards should be removed for this group.

In its current state, the gamified bot mainly consists of reward-based elements such as achievements and badges. This fact might be tied to the result that the “Player” users answered the most positively when questioned about their intention of use outside of the evaluation. However, the leaderboard was received negatively by some players but also positively by some. To address this, adding a new function that hides a player from the leaderboard might be reasonable. Overall, all player types seemed to enjoy the gamified elements during the short evaluation, which was focused on gathering a general first impression of the gamified chatbot. However, a more in-depth study will be needed to focus on individual elements and how they affect users, while also collecting user data to see which elements were used more.

4.3 Discussion

The present study aimed to investigate the impact of gamification on the motivation and acceptance of chatbots. Firstly, we aimed to answer RQ1 with our implementation of a gamified chatbot. The positive results suggest that we managed to create a somehow successful prototype. Yet, there still is room for improvement, as suggested by participants’ comments on different aspects such as better visualizations, new features and more. To answer RQ2, we analyzed the results of the survey questions related to the motivational aspects, behavioral intention and bot preference. Overall, the results suggest that gamification might increase the motivation and willingness to interact with a chatbot. Although some outliers showed less interest, they did not view gamification negatively but rather believed it would not increase their motivation. We further investigated subquestions of RQ2, namely RQ2.1 and RQ2.2. The results of RQ2.1 suggest that participants who previously played video games enjoyed the gamified bot more than those who never played. This implies a correlation between the previous game experience and motivation to interact with the gamified bot. In contrast, we did not find any significant correlation between player types and motivation in RQ2.2. Moving on to RQ3, our results suggest that the acceptance of the gamified bot is positive. As seen in the discussion of the results, the different constructs of the used TAM ended up with averages in the “I agree” category. The positive outcome of questions about usability and perceived usability further confirms our belief that the gamified elements made a good impression on the participants. Regarding subquestions of RQ3, we investigated the correlation between acceptance and previous game experience (RQ3.1). The results suggest that people who never played games before tended to be less in favor of the bot than those who did. In terms of the correlation between player types and acceptance (RQ3.2), we did not find any noteworthy results. The results suggest that each player type would enjoy and accept the bot similarly. Overall, our findings suggest that gamification can increase the motivation and acceptance of chatbots. Previous game experience seems to be a significant factor affecting motivation and acceptance of gamified chatbots. The results highlight the importance of the preferences of participants when designing gamified chatbots.

Threats to Validity. Threats to the validity of our evaluation results should be acknowledged to ensure that the conclusions made from this study are not misinterpreted or overstated. One potential limitation of our study is the different sampling sizes of the analyzed categories. In particular, the groups of participants who mentioned having no game experience and the “Disruptor” player type had smaller sample sizes, which may have resulted in less reliable data. Moreover, given that motivation and acceptance are complex constructs, it is plausible that our study, conducted over a relatively short period, may have not captured all the relevant aspects of these constructs. Thus, to address these limitations, we plan to conduct a longer study that will collect usage data and survey responses over a more extended period, which we hope will provide a more comprehensive picture of the effects of gamification on chatbot interaction, enabling us to draw more robust conclusions.

5 Conclusion, Implications and Future Work

In conclusion, this paper has presented a novel approach to increase learners’ motivation to interact with educational chatbots by integrating gamification elements. We identified and explored three distinct research inquiries on combining gamification and chatbot technology: how gamification and chatbots may be effectively combined to promote user engagement, the extent to which gamification serves to enhance motivation to interact with chatbots and the users’ acceptance of a gamified chatbot and its elements. We proposed a framework to gamify chatbots, independent of the chat platform, using standardized interaction data stored in a LRS and created Quiz-GBot, a gamified Quiz-Bot. The use of the standardized data format allows for the easy gamification of any educational chatbot, opening possibilities to include gamified chatbots in different TEL scenarios. The results of the evaluation of Quiz-GBot, which we based on different motivational models, showed that the gamification of educational chatbots could lead to increased user motivation and acceptance. Participants expressed enjoyment and satisfaction with the different gamification elements and reported an increased motivation to interact with the chatbot. However, our findings also suggest that while gamification can be an effective method for increasing user motivation, it may not be suitable for all users depending on prior gaming experience and other attributes. Yet, we saw that when categorizing users using the HEXAD framework, it made correct predictions about the users’ preferences. This leads to the notion that the HEXAD framework can be used as a base for implementing personalization into gamified bots. To improve the gamified bot, we plan to address minor technical issues, such as improving message formatting, visualizations, interactivity and the integration of models like ChatGPT for enhanced conversational realism. Additionally, in future studies, we want to evaluate the longer-term effects of gamification on user motivation and interaction rates in a real educational context.

Overall, this paper contributes to the growing body of research on gamification in education and lays the foundation for future studies in this field.