Keywords

These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

1 Introduction

A broad consensus is associated with the idea of learning through digital serious games, and game-based learning has reached an high popularity [1, 2]. Some authors, however, are skeptical about that, questioning if students are really motivated to learn, or they just want to play [3, 4]. According to this sort of criticism, edutainment is at risk of being limited to packaging and special effects, thus diminishing, instead of reinforcing, cognitive and metacognitive strategies [5, 6].

Research on game-base learning and serious games grew rapidly in the last years, but there are still few studies strongly based on causal relationships [7]. Instructions and feedback are widely studied in the field of educational serious games. The multifaceted nature of serious games, and their characteristic to be both serious and playful, reflect in most of the analyses about instructions and feedback.

Erhel and Jamet distinguish learning instructions and entertainment instructions within a serious game: the first ones are explicitly addressed to educational goals, the second ones encourage to play rather than to learn. According to their work, based on empirical evidence, while learning instructions can be considered more useful for the educational goals of the serious game, they are less appreciated by the players, who eventually prefer entertainment instructions that explain how to play, rather than prescriptions on what to achieve in terms of learning. Learning instructions are demonstrated to result in deeper learning, while entertainment instructions are more linked to accidental learning. However, entertainment instructions can result more effective if associated with feedback, which compensate the educational content [8].

Feedback, as well, is central to most pedagogic theories and was analyzed under different points of view [9–11]. A critical aspect of feedback within a serious game is its integration with game mechanics. Feedback must not distract the player while maintaining its positive effects, just like it must not interrupt the flow experience as stated by Csikszentmihalyi [12]. Feedback is one of the major facilitators of flow, but it must be properly designed to be as unobtrusive as possible not to delay or interrupt a flow experience. At the same time feedback must be harmonic with the Vygotsky’s Zone of proximal development, otherwise excessive, or insufficient, challenge transmitted by feedback will induce anxiety, boredom, or apathy in the learner, especially when taking part to an educational serious game [13].

Dunwell et al. underlined the difficulty of inserting feedback within serious games. There is a strong relationship between the timing and content of feedback. Feedback must be provided autonomously and seamlessly alongside an engaging gameplay experience, and balancing these two factors is a substantial design challenge [14]. Bellotti et al. suggest to use feedback through the evolution of the serious game, observing the consequences of game actions, rather than being informed by an abstract and separated process [15].

The present paper wants to investigate the roles of instructions and feedback within a serious game, taking into consideration the duration of the game and the perceived level of difficulty as fundamental variables. The inspiration for the present paper originates from a case study on the impact of inadequate instructions and feedback in serious games. A previous study was, in fact, conducted on a set of 30 serious games for mobile devices, finding that both feedback and instructions were negatively linked to the willingness to play again [16]. This apparently anomalous finding can be interpreted as due to the perception of obstruction of feedback and instructions for the gameplay. A possible reason for this can be the extreme shortness and simplicity of the considered games. For this reason it seems of a certain interest to study a causal model formed by feedback and instructions together with the duration of the games and their perceived level of difficulty. Points of interest of the present study are those originating from its analysis of causal relationships in the field of serious games. In particular, this paper aims at showing which causal relationships are present among instructions, feedback, duration, and perceived level of difficulty for short and simple educational serious games.

2 The Serious Games

In 2012, at the end of a two-year European project, titled InTouch, a set of 30 serious games for mobile devices was developed. The set of 30 serious games was addressed to adult learners to be usable, to challenge players to confront them with work-related non-routine tasks.

Instructions within the games were extremely narrow. According to the classification given by Erhel and Jamet [8], learning instructions were given in the first frames of the games, together with the game scenario and the problem-based situation to be solved. For most games there were no entertainment instructions at all, since the game interactions were absolutely intuitive. Very short instructions were given, if necessary, indicating the number of correct answers/choices that were expected for each decision point. Feedback was given only in the last frame of the games, showing the final score and explaining why and how the given answers were, or were not, correct.

Figure 1 illustrates an example of instructions and feedback that were given within the games. The left part of the figure is a screenshot of a decision point within the games, where players were advised about the remaining time to answer and the number of correct answer for the proposed question. The right part of the figure is a screenshot of the last frame of a game, with the gained final score and an explanation of the outcome.

Fig. 1.
figure 1

Screenshots of a decision point and of a final frame of a serious game.

Full size image

3 Scope and Hypotheses

A self-developed questionnaire was proposed to a group of 54 players after the completion of the games asking them to express on a 10 point Likert scale their opinion about the adequacy of the duration of the serious games, the design (graphic, interface), the fun of the gameplay, the instructions, the adequacy of the level of difficulty, the interest for the goal, the learning/educative content, the feedback, the realism of the game narration, the willingness to play again. Table 1 reports the average scores that were obtained for the measured variables.

Table 1. Average scores on a 10-point Likert scale (n = 54)
Full size table

Scores were generally high enough for all variables, or at least above the pass mark of 6 points. Roughly, it can be said that games were appreciated under many aspects by the sample of players. The only negative exception is the willingness to play again. For this reason a multiple linear regression on the willingness to play again (outcome variable) was conducted and the results are showed in Table 2 [16].

Table 2. Multiple linear regression of all the measured variables on the outcome variable “willingness to play again”
Full size table

The regression coefficients for instructions and feedback are negative. At first sight it was quite surprising that instructions and feedback were perceived as contributing to discourage the willingness to play again. At the same time, the expressed adequacy of the duration and of the difficulty of the games resulted to be significant positive predictors of the willingness to play again. All these findings suggested to hypothesize that for short and simple games, like the ones that were considered, instructions and feedback could be perceived as obstructive for the gameplay. Players showed to desire to have fun while playing, and repeat their experience with the games, without stopping or interrupting the flow to receive instructions or feedback. This interpretation can be considered coherent with the fact that the games were not perceived as heavily educative, but rather a playful way to practice soft skills.

Starting from these results it was decided to deepen the analysis of the role of instructions and feedback within the serious games, investigating the influence of the perceived adequacy of the duration and of the difficulty. For the scope of the present study the four game variables of interest are thus the degrees of satisfaction expressed by the sample of users about: (a) the instructions, (b) the feedback, (c) the adequacy of the duration, and (d) the adequacy of the level of difficulty of the games.

The causal model to be tested reflects the order of appearance of the aforementioned variables within the serious games. Instructions are the first component that is given to the user at the very beginning of the game, while the perceived adequacy of the level of difficulty must be considered the final opinion of the user about the game. In between there are the opinions about the feedback and the adequacy of the duration of the games that can be considered as mediating variables. It is hypothesized that the quality of the instructions influences the opinion about the adequacy of the level of difficulty both directly and indirectly through the mediation of the feedback and of the adequacy of the duration of the games (Fig. 2). All causal relationships are hypothesized to be positive, since they represent the degree of satisfaction expressed by the users about the variables, as referred to the played games. It means that high values for the adequacy of the level of difficulty, for instance, are not necessarily equivalent to a very difficult game (in effect, the considered games were not difficult at all), but rather denote the appreciation for the challenge offered by the game that appears to be neither too difficult nor too easy.

Fig. 2.
figure 2

Graphical scheme and numbers of the paths for the analyzed causal model.

Full size image

4 Methods

This section contains an illustration of the methodology that was adopted in the present study: a description of the sample; the research procedure; the instruments and the statistical analyses that were adopted.

4.1 Participants

The target sample consisted of 54 workers of nine different Small and Medium-sized Enterprises (SMEs) operating in different business sectors (ICT, business support, education/training, etc.). The SMEs were selected on the basis of their willingness to participate in the study. Work positions were: 28 managers and 26 employees. 30 were males (56 %) and 24 were females (44 %). The mean age was 41.94 years (SD = 9.70).

4.2 Procedure

To test the developed kit of 30 mobile serious games the project partners held dedicated events (Learning Labs). During each Learning Lab a structured self-developed questionnaire was proposed to participants after the completion of the games.

Participation to Learning Labs and questionnaire compilation were obtained through an informed consent procedure asking for active consent from participants. Questionnaires took approximately 30 min to complete. Project staff members introduced the questionnaires, giving instructions about their compilation, explaining that they were voluntary and responses were anonymous and confidential. Project staff members were at the workers’ disposal during the questionnaires’ administration to answer questions and give explanations. All participants to different Learning Labs responded to the same questionnaire packet.

4.3 Measures

The self-developed questionnaire proposed to participants after the completion of the games was formed by the sections described below.

Demographics.

An Identifying Information Form was used to collect demographic information: age, gender, working role.

Game variables.

On a 10 point Likert scale it was asked to express one’s appreciation about: (a) the instructions (“How adequate was the quality of the instructions?”), (b) the feedback (“How adequate was the quality of the feedback?”), (c) the duration (“How adequate was the duration of the games?”), and (d) the level of difficulty of the games (“How adequate was the level of difficulty of the games?”).

4.4 Data Analysis

Preliminary Analysis.

As a preliminary analysis, skewness and kurtosis of game variables were checked. Overall, all variables showed to conform to the normal distribution.

Correlation.

As a first step the correlation matrix of all the variables was calculated.

Path Analysis.

The path model involving the aforementioned four variables was analyzed with LISREL software package, using maximum likelihood estimation procedures [17]. The R-square percentage of variance of the adequacy of the level of difficulty explained by the model was reported, to estimate the completeness of the considered set of predictors for the outcome variable.

5 Results

Table 3 reports correlation coefficients of (a) the instructions, (b) the feedback, (c) the adequacy of the duration, and (d) the adequacy of the level of difficulty of the games. Level of significance of the correlation coefficients (p-values) is indicated in the table footnote.

Table 3. Correlation matrix of the variables involved in the analyzed causal model
Full size table

Table 4 reports the results of the path analysis with the levels of significance of the causal paths (p-values) indicated in the table footnote. Paths numeration is the same as indicated in Fig. 2.

Table 4. Path analysis coefficients estimates of the causal model
Full size table

The R-squared value for the adequacy of the level of difficulty, considered as the final outcome variable, was found to be 0.85, which means that 85 % of the variance of the adequacy of the level of difficulty is explained by the considered model where the adequacy of the level of difficulty is predicted by the instructions, the feedback, and the adequacy of the duration.

The effect of the instructions (exogenous variable) on the adequacy of the level of difficulty (final outcome variable) is reported in Table 5, with the level of significance indicated in the table footnote. The direct effect is simply given by the coefficient of path 3. The indirect effect is composed by the sum of two parts: the one obtained by the meditation of the feedback (composition of paths 1 and 4) and the one obtained by the mediation of the adequacy of the duration (composition of paths 2 and 5).

Table 5. Effects of the quality of instructions on the adequacy of the level of difficulty
Full size table

The instructions have a significant total effect on the adequacy of the level of difficulty, obtained as the sum of a direct effect (path 3 = 0.32) and an indirect effect. The indirect effect is given by the sum of a significant mediation of the adequacy of the duration (path 2 path 5 = 0.36), and a non-significant mediation of the feedback (path 1.path 4 = 0.11).

6 Conclusion

As a first conclusion, it must be said that the use of multivariate analysis allowed to understand much more deeply the learning and game mechanics rather than the merely descriptive results obtained by the satisfaction questionnaire. This is a useful indication for the evaluation of serious games to go beyond the simple measurement of a satisfaction questionnaire. In the considered case, what seemed to be a good level of satisfaction, when analyzed with multivariate technique, unveiled unexpected and non-trivial relationships among variables. The use of multilinear regression and of the path analysis, in fact, shed light on the effective role of game components and gave access to interesting interpretations and research perspectives herein illustrated.

The explanatory power of the considered model is quite significant, since 85 % of the variance of the adequacy of the level of difficulty is explained. This means that considering the instructions, the feedback, and the adequacy of duration as predictors of the adequacy of the level of difficulty was a right and enough exhaustive choice.

Results seem to confirm the hypotheses that there is a strong casual interrelationship among the instructions, the feedback, the adequacy of the duration, and the adequacy of the level of difficulty within a serious game.

Instructions showed to be strictly connected to the adequacy of the duration of the game (path 2), while the adequacy of the duration is strongly connected to the adequacy of the level of difficulty (path 5). Thus, instructions are significantly linked to the adequacy of the level of difficulty both directly (path 3) and indirectly through the adequacy of the duration of the game. It can be said that the adequacy of the duration of the games has a significant role of mediation in the relationship between instructions and the adequacy of the level of difficulty. It can be hypothesized that for short and simple games, like the ones that were analyzed, there should not be complex instructions, otherwise there is the risk that players perceive them as obstructive for the gameplay and inadequate in connection to the level of difficulty of the game.

On the contrary, the relationships between the instructions and the feedback (path 1) and between the feedback and the adequacy of the level of difficulty (path 4), even if significant, are not so strong to result in a significant indirect effect of the instructions on the adequacy of the level of difficulty through the feedback. This result is coherent with the findings by Erhel and Jamet who showed that the feedback enhances serious games’ effectiveness, mostly if associated with entertainment instructions, while it acts separately from learning instructions like those that are present in the analyzed games [8].

It can be said that for the analyzed games, the instructions and the feedback contribute autonomously to the perception of the adequacy of the level of difficulty of the games. Attention must be paid both in the design of the instructions and of the feedback, so that both are coherent with the perception of the adequacy of the level of difficulty of the game, and none of them result to be obstructive for the gameplay.

The present study seems to confirm the suggestion to properly integrate feedback in the game, in order not to distract or bore the player [15]. This should be done especially for short and simple games, where players do not expect to find separate communication, but rather to test the consequences of their actions directly on the evolution of the game scenario.

The small sample size (n = 54) and the weak reliability of the instrument to measure the analyzed dimensions are the main limits of the present study. Instead of a generic self-developed satisfaction questionnaire, with one item for each variable, a validated instrument should be adopted, mapping multiple items to variables through factorization.

As a perspective for future work, what was found for short and simple serious games should be tested with other types of serious games (for instance more complex games with a longer duration) to check if the considered causal model is still valid. It should be tested if the role of the adequacy of the duration and of the level of difficulty is significant also when instructions and feedback, instead of being obstructive elements, positively contribute to enhance motivation and willingness to play again.