Keywords

1 Introduction

Personalised learning environments rely on repositories of digital learning materials, and on meta-data that provide semantic information about the digital content [10]. The semantic information is fundamental to the ability of AI agents to make ‘intelligent’ decisions, such as recommending content to learners, assisting teachers in search & discovery of learning resources, and for re-using materials between contexts [2,3,4,5, 15]. Recommendations about the learning resources is an important component of the semantic information, since teachers searching for learning materials in blended learning environments value the feedback and review of peers who have previously used these resources [7]. However, a major challenge in mining recommendations from teachers is their low motivation to contribute the time and effort needed to produce such feedback [12].

One possible solution to this challenge is the use of Gamification: a term describing the use of game elements (such as points, prizes, progression through levels, time pressure, competition, cognitive challenges, and more) to improve user experience and user engagement in non-game services and applications [9]. The underlying idea of gamification is that by making a task entertaining, it is possible to engage humans to do tasks that do not provide any other tangible reward [8, 16]. Gamification is being used in various domains and types of systems, including social networks, e-commerce, search engines, healthcare systems, and more [6, 8, 13, 19].

One of the most prominent fields in which gamification is used, is that of educational technology [8]. Attempts at applying gamification elements and methods in educational contexts have shown promising results [1, 14, 17, 18]. However, to the best of our knowledge, the potential of gamification to incentivise teachers in teacher-sourcing tasks was not evaluated before.

In this paper, we report on the results of a pilot research aimed at studying the impact of gamification on teachers’ motivation to contribute feedback on the resources that they have used (typically for in-class activities or as homework), and fuel a social-based recommendation system within an OER repository in Physics. Specifically, we seek to answer the following research questions (RQs):

  • RQ1: What gamification mechanisms do teachers believe will encourage them to provide feedback on the learning resources that they have used?

  • RQ2 Does implementing these elements actually enhance teachers’ willingness to provide feedback?

This paper makes the following contributions:

  1. 1.

    It is the first real-life design and implementation of a cutting-edge Gamification Taxonomy [21].

  2. 2.

    It presents, for the first time, a participatory design approach for introducing Gamification into a large OER system, used throughout a whole country.

  3. 3.

    The paper provides results on the implementation of the Gamification elements via a long-term pilot study within a real-life OER system for teachers.

  4. 4.

    Results show statistically significant increase in feedback from the teachers, to an unprecedented 4X increase, even when taking into account the ‘Covid-pandemic effect’.

2 The Learning Environment – PeTeL

PeTeL (Personalised Teaching and Learning) is a shared repository of open educational resources (OER), and a Learning Management System (LMS) that also includes social network features and learning analytics tools. It is developed at the Department of Science Teaching at the Weizmann Institute of Science, with the goal of assisting STEM teachers in providing personalised instruction in blended-learning environments.

PeTeL is divided into separate modules for each subject matter: Biology, Chemistry and Physics. It is implemented on top of a Moodle LMS. To assist teachers in searching and discovering learning materials that best suit their students’ needs, PeTeL provides common search filters such as subject matter, level of difficulty, duration, technical requirements (e.g. projector or mobile devices), nature of the activity (e.g. diagnostic questionnaire, interactive task, home assignment, etc.), and in addition, social-based search and discovery features. For example, teachers can follow other teachers within a social network-style collaborative environment (referred to as the ‘peer network’), receive recommendations from them, copy their teaching sequences, and more. Teachers can also search and rank materials based on reviews provided by their peers.

After using an activity in their class, the teachers are presented with a ‘pop-up’ window, requesting them to provide feedback concerning the resource they used. The teachers can either fill the pop-up survey, postpone filling out the form to a later date, or cancel it. This feedback mechanism was initially activated in PeTeL during the 2019–2020 school year. However, teachers’ cooperation was relatively low, and their response rate to the feedback requests during this first year was below 3%. Since the reviews were identified by the teachers as very influential on their decision on which activities to use, and also provide the basis for an automatic ranking algorithm that is currently under design, we marked the issue of increasing the response rate as a major challenge that should be addressed, and decided to examine gamification as a conceptual framework for addressing this challenge.

3 Gamification Taxonomy

Concerning the gamification elements, our conceptual framework relied on the new, cutting-edge Taxonomy of Gamification Elements for Educational Environments (TGEEE) [20, 21]. The taxonomy was built based on large-scale data collection on gamification preferences of educational users, and proposes 21 gamification elements suited for educational contexts. These elements are grouped into five major dimensions: Performance, Social, Ecological, Personal, and Fictional.

The Performance dimension includes elements that are related to the environment’s response to student interactions, such as badges and points. The Social dimension refers to elements that deal with interactions between the students in the environment, e.g. cooperation and competition. The Personal dimension is related to the learner using the environment, usually related to meaning and purpose, for example, by setting objectives. The Ecological dimension refers to properties/characteristics provided by the environment, such as economy and chance. Finally, the Fictional dimension deals with the context of the environment, affecting both users (Narrative) and the environment (Storytelling). A graphical representation of the elements, and their grouping into dimensions, is depicted in Fig. 1.

It is important to state that, according to the authors, an environment does not necessarily need to contain all the elements from all dimensions. The selection of elements should be aligned with the objectives of the environment and the users who will interact with it [22, 23]. This justified our first experiment, the participatory design with teachers, described next.

Fig. 1.
figure 1

The TGEEE Gamification Taxonomy from [21]

Full size image

4 First Experiment: Teacher Preferences

This experiment was designed to answer the first research question: What gamification mechanisms do teachers believe will encourage them to provide feedback on the learning resources that they have used?

4.1 Procedure and Materials

The first experiment, a participatory design experiment, was conducted with seventeen Physics teachers, ten women and seven men, who participated in a one-day yearly training conference on PeTeL during July 2020. This was part of a session on the search and discovery mechanisms that PeTeL offers. A previous iteration of this event, allowing for interviews with teachers, marked the social recommendation as especially valued by teachers. We discussed the low response rate on the pop-up surveys, the potential use of gamification as means to increase it, and presented the taxonomy to the teachers.

Then, the teachers were presented with five mock-ups of different gamification elements, each implementing a certain dimension of the Taxonomy (see below), and were requested to rate how much they believed that the concept underlying this element (e.g., social reward) could enhance teachers’ motivation to provide feedback (on a 1–5 Likert scale). In addition to the Likert questionnaire, the teachers were requested to expand their answers as much as they wished, via open-ended questions. Then, a group discussion was held. We note that the mock-ups were visually integrated into the front-end of PeTeL, to provide an authentic user experience.

4.2 The Five Elements Presented to Teachers

Badges: the first element was giving teachers virtual badges (gold, silver or bronze) according to the amount of reviews they gave. We based this element on two different concepts from the taxonomy: first, the “acknowledgement” concept from the “performance” dimension in the taxonomy, which refers to elements in the environment that praise the user’s actions. The second was the “reputation” concept from the “social” dimension in the taxonomy, meaning that teachers may value the possibility of being recognised by their peers as contributing members to the entire teacher community.

Leader-Board: the second element was a leader board, presenting the number of points each teacher accumulated by filling in reviews. This element was also based on two different concepts from the taxonomy: the first was the “points” concept taken from the “performance” dimension in the taxonomy, meaning that the notion of receiving credit for their performance could raise teachers’ motivation. The second was the “competition” concept from the “social” dimension in the taxonomy, indicating that the presentation of a teacher’s ranking in comparison to other teachers can encourage them to participate.

Progress-Bar: the third element was a progress bar, showing the accumulation of required feedbacks on each learning resource. This element was based on the three following concepts: “cooperation” taken from the “social” dimension in the taxonomy, the “progression” concept taken from the “performance” dimension in the taxonomy, and the “objectives” concept taken from the “personal” dimension. The “cooperation” element builds upon the notion that the teachers’ feeling that they are working together towards a common goal, could motivate them. The “progression” concept claims that allowing teachers to view their progression within the environment will foster their willingness to contribute information. Finally, the “objectives” concept states that giving teachers a clear goal will raise their motivation.

Virtual Applause: the fourth element was virtual applause, meaning that each time teachers filled out a feedback form, the learning environment would present them with an animation of fireworks, confetti, and the sound of an audience applauding. This element is based on the “sensation” concept taken from the “personal” dimension in the taxonomy. This means that using the teachers’ senses in the manner of visual or audio stimulation, can affect their motivation.

PeTeL Dollars: the fifth element was PeTeL Dollars, meaning that the teacher would receive virtual currency for giving feedbacks. At the end of the school year, if the teacher has reached a certain amount of virtual dollars, he/she can replace them for a real-life reward such as lab equipment or a field trip with the students. This element is based on the “Economy” concept from the “ecological” dimension in the taxonomy, meaning monetising teachers’ actions in the environment. We note that this element does not fully coincide with the aforementioned definition of gamification (“do not provide any tangible reward..”, see Sect. 1). However, previous attempts at implementing gamification in different contexts used such mechanisms (e.g., the Spanish league for cardiologistsFootnote 1). Therefore, we decided to include this extended definition in our first experiment.

An example of an item from the questionnaire, presenting a ‘virtual applause’ gamification element, is presented in Fig. 2.

Fig. 2.
figure 2

Example of an item from the gamification questionnaire

Full size image

4.3 Analysis and Results

Following the above participatory design phase, teachers’ ratings and responses to the open-ended questions, as well as the transcription of the group discussion were analysed. As can be seen in Table 1, the two elements that received the highest average ratings are the PeTeL-Dollars (3.67) and the progress bar (3.24). The virtual applause received the lowest rating (1.47).

The rating results were triangulated with the open-ended responses and the group discussion. This analysis yielded the following conclusions:

First, teachers want to have clear goals, and to know their status with respect to them. This was contrasted with the previous design, which sent feedback request on each resource that was being used, without giving any indication of what is the expected level of contribution.

The second conclusion was that teachers wish to know that their contribution matters, that it is useful for other teachers, and that it helps to improve the environment. This incentive was recognised as much stronger than competition or sensation. This was contrasted with the previous design, in which their feedback was ‘buried somewhere’, and they had no idea whether it was actually being used for anything.

The third, and maybe most surprising finding, was that social recognition matters – we found that for many teachers it was important that their contribution would be seen by the community. This was contrasted with the previous design, in which the individual contribution was not acknowledged. We interpreted this through the prism of the “going green to be seen” [11] phenomenon found among environmentally-aware consumers, who wish to signal a statement about themselves as responsible members of the community (this was used for example to explain the phenomenal success of the Toyota Prius, with its distinctive design, over similarly fuel-efficient cars with conventional designFootnote 2).

5 Second Experiment: The Effect of Gamification-Driven Design

This experiment was designed to answer the second research question: Does gamification-driven design enhance teachers’ willingness to provide feedback?

5.1 Procedure and Methods

Following the results of the first experiment, two gamification elements were implemented and integrated into PeTeL, which are described below.

Table 1. Teachers’ rating of the gamification elements
Full size table

Progress Bar. This element addresses the first conclusion – that teachers wish to have a clear goal and know their status with respect to it. A goal of five reviews per year was set (the value was decided by the Physics development team), and a progress bar feature showing for each teacher her progress towards this goal was designed and integrated into PeTeL. It is illustrated in Fig. 3. We note that in the original design presented to the teachers, the progress bar showed the accumulation of information per each resource in PeTeL, while the actual progress bar that was implemented showed the number of reviews filled per teacher. This change was performed due to a concern that capturing progress by resource will be harder to translate into an evident, global contribution of the individual teacher action to the whole system (which includes many resources). However, the new design still maintained the “progression” and “have a clear goal” dimensions of the original design. In addition, the ‘social’ aspect of the ‘by resource’ progress bar is actually captured by the public bulletin board (see below), while the eventual ‘by teacher’ design addresses the need for having an individual goal and status with respect to it.

Fig. 3.
figure 3

Request for feedback with progress bar

Full size image

Bulletin Board. The second element was a ‘bulletin-board’, showing teachers’ feedback on the activities that they have used. It is shown in Fig. 4. The bulletin-board is presented to the teachers in the main page of the learning environment. Each time a teacher reviews an activity, the bulletin-board is updated for all the teachers, with the new review on top and highlighted.

Each input in the bulletin-board contains the name of the teacher who reviewed the learning resource, and the title of the resource that has been reviewed. When hovering with the mouse over the review, a mouseover text showing the details of the review pops-up. The items in the bulletin-board are ‘linkable’, so teachers can easily follow a review, in case that they wish to mark a certain resource for future use in their class.

Fig. 4.
figure 4

Recommendation “bulletin-board”

Full size image

Although the bulletin-board was not one of the elements presented to the teachers in the first experiment, it addresses two key issues raised by the teachers in the open-ended questions and during the group discussion: First, that they wish to know that their contribution matters. The fact that everyone can see and use their recommendation, and they know that, addresses this. The second is the social recognition, achieved by presenting the name of the teacher who provided the review. We also note that the “PeTeL-Dollars” element was not implemented even though it was ranked highest among the elements, as we decided to avoid tangible rewards and test a model that is sustainable, budget-wise.

5.2 Analysis and Results

We monitored teachers feedback during the first 2 months after these two gamification elements were activated, and compared them with the data we had from the previous school year (Sep. 2019 - July 2020). We note that in order to allow for direct comparison, the pop-up review itself was not modified. For our analysis, we additionally took into account the fact that more people turned to online work during the Covid pandemic - what we call the ‘Covid-pandemic effect’.

We accounted for the effect via two metrics: i) The total amount of reviews, normalised by the amount of active teachers; and ii) response rate – the percentage of review requests that are answered.

Total Amount of Reviews. First, we compared the average amount of reviews received each month. Considering the ‘Covid-pandemic effect’, we did not consider only raw numbers, but normalised them by the amount of active teachers. Active teachers are teachers who used at least one learning resource in their class. Comparing the number of active teachers this year and in the previous one yielded that the number of active teachers was very similar (actually somewhat smaller this year, probably due to the shorter time period): 177 active teachers in the previous year (out of which 33 teachers filled reviews = 18.64% of the active teachers), 169 active teachers this year (out of which 34 filled reviews = 20.11% of active teachers). During the previous year, 62 reviews were provided by teachers over a period of 10 months, averaging at 6.2 reviews per month. During the 2 months since the implementation of the gamification elements, we received 56 reviews, an average of 28 reviews per month, more than X4 that of the previous year. Considering however that some of this increase may still be due to active teachers just spending more time online, we continued our analysis.

Response Rate. Next, we measured the difference in the response rate before-and-after the implementation of the gamification elements. The response rate is defined as the percentage of feedback requests that are answered by the teachers. Thus, the response rate accounts for other activities that might have increased in the system, such as learning resources usage. Last year, the teachers used a total amount of 2,372 learning resources, and filled 62 reviews, a response rate of 2.61%. During the 2 months since the implementation of the gamification elements into PeTeL, the teachers used a total amount of 840 learning resources, and filled 56 reviews: A response rate of 6.67%, more than X2.5 increase in comparison to the previous year. A proportion test confirmed that the gamification-driven design generated a significantly higher response rate than the previous design (\(6.7\%\) versus \(2.6\%\); \(z=5.4\), p-\(value<\)0.0001).

6 Conclusions

This paper describes a pilot research that aims at studying the potential of gamification-driven design as means to incentivise teachers to participate in crowdsourcing activities. Results show that teachers want to have clear goals, to know that their contribution matters, and to be recognised by peers as contributing members of the community. Following these findings, two gamification elements – a progress bar and a bulletin-board presenting teachers’ recommendations, were designed and integrated into the learning environment, and their impact on teachers’ motivation to provide reviews was measured. Analysing teachers behaviour two months after the new features were aired showed a substantial increase in the amount of reviews provided by the teachers and their response rate, suggesting that the use of gamification can indeed enhance teachers’ motivation to take part in crowdsourcing activities, and specifically, in recommending learning resources to other teachers.