Keywords

1 Introduction

Digital game-based learning (DGBL) refers to learning partly or fully through computerized games. DGBL can be viewed as a subfield of multimedia learning, which is learning from materials combining words and pictures [25]. Digitalized multimedia learning materials are, for example, animations or computerized slides.

Recent major meta-analyses of studies comparing the effectiveness of the DGBL approach to “traditional” educational approaches demonstrated a small superiority of games [e.g., 12, 40]. In theory, the reasons for the games’ superiority are numerous, [e.g., 2, 15, 29, 34; see also 25, pp. 13–15]. In practice, they are generally unknown. This superiority can in fact be only seeming; it could have arisen because of a methodological artifact. For instance, both meta-analyses showed that the positive effects of games tend to attenuate in studies with random assignment. Elevated positive affective-motivational states of learners (motivational states throughout for brevity) are also one possible reason for the games’ alleged superiority. However, despite the hype about DGBL motivational benefits, the meta-analyses made it clear that only a fraction of the DGBL studies measured both motivational and learning outcome variables (see below for examples). Narrative DGBL reviews [e.g., 11, 22] also did not provide information on the strength of the relationship between learning outcomes and motivational variables. Still very little is known about this issue.

In the general multimedia learning field, motivational factors have only recently started to be studied [e.g., 18, 24, 37], and incorporated into multimedia learning theories [e.g., 27]. Syntheses of literature point to the fact that augmenting learning materials by using appealing bits of extraneous information, which elicit interest but are not needed for comprehending the key learning message (i.e., seductive or extraneous details), generally hampers learning [26, Chap. 4; 30]. Beyond that, little is known about the motivational–learning relationship in the context of learning from computerized multimedia (similar to the DGBL field).

From a methodological perspective, in comparative studies, the correlations between learning outcome variables and motivational variables depend on two things: on the experimental manipulation (which may have effects both on cognitive and motivational domains) and on underlying motivational–learning associations (which are detectable also in a correlational study). Information is insufficient about the possible strengths of either of those two influences.

From the perspective of educational psychology, this situation is not surprising. First, the key variables of interest are the learning outcomes (rather than motivational self-reports), because they are objective. Motivational variables are thus not always measured. Second, concerning comparative studies, it is now generally accepted that DGBL approaches (and additions of interesting embellishments to multimedia learning materials in general) can influence learning processes in two opposing ways, [e.g., 1, 10, 37]. They can enhance learning through positively activating and energizing learners: learners will invest more into learning. This idea is framed, for example, by the cognitive-affective theory of learning by media [27]. However, these approaches can also hamper learning by distracting attention away from learning (i.e., via the seductive details embellishing the instructional message, which, in the case of DGBL, is often the game’s entertainment feature). This idea is outlined, for instance, in the cognitive load theory [36]. Depending on how well the game/multimedia is designed and how well the learning and playing/interesting parts are integrated, the positives may (but also may not) outweigh the negatives.

Third, the level of distraction may play a moderating role (on motivational–learning link) also in single-group correlational studies, provided learners can relatively easily ignore the supposedly interesting part of the learning material. Consider a learning game, for example. When the gaming part is not well integrated with the learning part, the learners motivated by the gameplay will end up playing rather than learning. Whereas, the less motivated learners may ignore the gaming part and just learn (because they were told to do so), eventually outperforming the game-motivated learners in terms of learning outcomes.

Motivational–learning correlations can thus be confounded by different levels of distraction caused by different intervention designs. As a result, null-to-negative [e.g., 14, 21, see also 1], null [e.g., 38], null-to-positive [e.g., 16, 18], as well as positive [e.g., 32, see also 13, 17] associations have been reported in DGBL and multimedia learning literature.

In this paper, we re-analyze our 11 multimedia learning experiments dating from 2009–2017. The participants were primarily Czech and Slovak high school and university learners (total N = 1288). We focused on correlational outcomes in order to obtain information about the nature of motivational–learning association in the context of learning through digital games and related multimedia. These studies in particular used digital learning games (n = 689), complex interactive simulations (n = 140), brief animations (n = 278), and computerized slides (n = 181) as treatments. Despite the fact that all our studies were comparative, we focused primarily on the underlying motivational–learning associations rather than the effects of experimental manipulations. This is because the majority of our studies yielded null results (i.e., with two exceptions, there were no robust effects of experimental manipulations). The interventions were reasonably optimized by so-called multimedia learning principles [26] and, in the case of games, we paid special attention not to separate the gaming and learning parts. The studies were conducted by the same lab using the same/similar research methods. Therefore, at least a certain level of consistency across studies with respect to possible cognitive distraction (and methodological artifacts) can be guaranteed, and the general correlational trend can indicate how motivational variables are related to learning outcomes in DGBL-like contexts.

2 Method

2.1 Study Characteristics, Participants

The studies (Table 1) used experimental design with two or three groups. They either compared a DGBL approach to a different type of learning on the same topic (i.e., so-called media comparison studies [25]) or they compared interventions that differed in a single feature or a few features (i.e., so-called value-added studies [25]). Participants were typically above-average, high school or college learners from the capital city of the Czech Republic (see Table 1 for mean age).

Table 1. Pearson’s correlations between motivational and learning outcome variables.
Full size table

2.2 Interventions

The interventions ranged from using static computerized slides to a complex digital learning game (see Table 1).

In a set of three value-added studies [3, 4, 9], we compared different versions of a 2-h interactive simulation on the topic of beer brewing (Fig. 1). Participants studied how to brew beer using on-screen instructions, practiced key steps in the simulation, and eventually brewed beer in the simulation. We researched the added value of instructional texts in a conversational rather than a formal style (i.e., the so-called personalization principle: [26, Chap. 12]), the effects of an intrinsically motivating topic, and the added value of several gamification elements (e.g., points).

Fig. 1.
figure 1

Screenshots from the beer brewing simulation, animal training game, and Europe 2045.

Full size image

In another study [6], which consisted of four different experiments, we once again studied the personalization principle. We used about 7-min-long, self-paced animations. In two experiments, one with high school and the other with university students, participants studied how lightning forms. In the remaining two experiments, they studied how a biological wastewater treatment plant functions.

In a set of three media comparison studies [7, 8, 10], we compared a DGBL approach to a non-game instructional method, keeping the content and the length of exposure the same. In one case, we used the Europe 2045 computer game (Fig. 1). This is a complex, team role-playing game on the topic of European Union policies. The non-game control condition was organized around discussions that replicated in-game discussions but removed all game mechanics. In the second case, we used a 20-min simulation mini-game on how to train animals (Fig. 1). After an expository lecture on the animal training topic, participants either played the game or received a complementary lecture with videos on animal training. In the final study, after an expository lecture, students either played a mini-game individually at computers, or the teacher played it, while showing it to the whole class on a projector and prompting students on how to proceed with the game. One half of learners were exposed to the animal training mini-game, and the other half to a 20-min simulation mini-game on Mendelian genetics [28].

In the final study [35], we used 10-minute-long, self-paced instructional slides as a treatment. We researched the added value of augmenting schematic graphics by adding black-and-white or colorful anthropomorphic faces to non-human elements. This type of instructional redesign is called emotional design [37].

2.3 Variables

Learning Outcomes.

Our studies generally tested mental model acquisition. In the multimedia learning field (including DGBL), quality of acquired mental models is typically measured by so-called retention and transfer knowledge tests [26]. Retention tests assess “superficial” learning; i.e., whether the learner was able to memorize the material without necessarily understanding the core process/model in question (e.g., “Based on the animation you just saw, describe in detail how biological wastewater treatment works.”). Transfer tests assess “deep” learning; i.e., if learners truly understand the point and are thus able to “transfer” and use what they learnt into new contexts (e.g., “What would happen if a fungus first appeared in the wastewater treatment plant and then bacteria? Write down all consequences that come to mind based on the animation you saw today.”). We typically measured learning outcomes immediately after the treatment and three or four weeks later (i.e., delayed tests).

Motivational Variables.

There are indications that different positive motivational variables are differently related to learning outcomes [4]. Therefore, we report here correlations for individual variables rather than for a synthetic, composite variable: enjoyment (n = 1,288), flow (n = 892), and generalized positive affect (n = 892). We measured enjoyment typically on a 4-, 6- or 8-point Likert scale with 1–3 items (e.g., “I enjoyed today’s lesson.”, “This activity was appealing for me.”). These items correspond to items from intrinsic motivation inventories [e.g., 20]. We did not use the whole inventories for brevity. Generalized positive affect, called positive affect here for brevity, was measured by a PANAS (Positive and Negative Affect Schedule [39]). Participants had to rate their current feelings on a 5-point Likert scale. The list of feelings included 10 positive feelings (e.g., interested, excited, strong) and 10 negative feelings (e.g., distressed, upset, scared – these are not analyzed here). Flow was measured using ten, 7-point Likert items from the Flow Short Scale [31] (e.g., “I feel just the right amount of a challenge.”, “My thoughts run freely and smoothly.”, “I don’t notice time passing.”). Internal consistencies were generally good for all variables (Cronbach α generally > .8).

2.4 Procedures

Two studies [7, 8] took place in school settings. The rest was organized in a lab setting. In general, the studies followed the following procedure. After the introduction and filling in prior questionnaires, participants were randomly assigned to one of the conditions. Thereafter, they completed the intervention. In three cases [3, 4, 9], motivational variables were measured during the treatment and after it. The values were then averaged. In other cases, the variables were measured only once; typically, right after the treatment ended. Afterwards, participants filled in retention and transfer tests. With one exception [6], they attended a delayed testing session three/four weeks later and completed the tests once again. They were not informed in advance that the purpose of the delayed session was to assess their knowledge. Many of them probably guessed the purpose of the session; however, the majority of them reported back they did not study for the tests beforehand, even though they had a hunch they would be tested. Data collection was anonymized. The test performance had no consequences for students (e.g., did not impact their grades).

2.5 Data Treatment

Correlations between motivational variables and learning outcome variables are reported across the whole sample for each study, because subgroups are generally small (~30 per cell). Whole-sample correlations reflect both general motivational–learning associations and the effects of experimental interventions. However, only two studies [4, 10] found robust between-group differences: both in motivational and learning outcome variables. Special attention will be paid to these two studies. For one of them [10], the sample is actually so large (around 100 per cell) that correlations within subgroups can be considered for meaningful contrasting of whole-sample correlations to subgroup correlations. Other studies generally reported null results as concerns motivational variables as well as for learning outcomes (only 4 out of a total 52 between-group comparisons were significant at p < .05 level).

We summarized the correlation results across reported studies using the DerSimonian-Laird (DSL) random-effect meta-analytical approach. We used a metacor [23] and meta [33] packages for calculations. Because not all studies used strict retention and transfer tests (see Table 1), but all used immediate knowledge assessments and all but one used delayed knowledge assessments, we decided to obtain one correlation value per (1) each motivational variable and immediate learning outcome variable, and (2) each motivational variable and delayed learning outcome variable. Therefore, in studies featuring separate retention and transfer correlations, we transformed the values in Fisher’s z scale and we used their average (i.e., one average for immediate and another for delayed knowledge assessments). In addition to individual correlations, we report summary correlations, their 95% confidence intervals, corresponding p values, and dispersion measures (Q, I2, τ2).

3 Results

All correlations are reported in Table 1. As is apparent, there is a general trend for weaker and less stable associations (with learning outcomes) for enjoyment and positive affect compared to flow. At the whole sample level, correlations range from −.20 to .45 (median = .13) for enjoyment, from −.14 to .48 (median = .165) for positive affect, and from .06 to .45 (median = .30) for flow. This pattern does not change when correlations with immediate learning outcomes and delayed learning outcomes are considered separately, nor when the whole-sample correlations are summarized using the meta-analytic approach (see Fig. 2 and the last rows of Table 1). Even the weaker associations concerning enjoyment and positive affect are significant though (see Table 1). In correlations between positive affect and delayed learning outcomes we saw an evidence for substantial variability – it is likely the true effect sizes differ across the included studies.

Fig. 2.
figure 2

Correlations and summary effects between learning outcomes and motivation variables.

Full size image

Do these correlations reflect the effects of experimental manipulations or general motivational–learning associations? First, as already said, with the exception of two studies [4, 10], no robust between-group differences with respect to experimental comparisons emerged (i.e., generally, the studies reported null results; see Sect. 2.5). Second, when correlations at the subgroup level of the Europe 2045 study [10] (Table 1) are contrasted to whole-sample correlations, the general pattern of results remains the same; although that particular study was the only one in which flow correlations were weaker than positive affect correlations.Footnote 1 Thus, the strength of the underlying motivational–learning associations (i.e., without the effects of experimental manipulations) is indeed within the range reported in the last column of Table 1.

At the same time, in the two studies that reported between-group differences, we showed that experimental manipulations (i.e., a low vs. a high intrinsically motivating topic: [4]; a discussion vs. a game: [10]) influenced both motivational variables and learning outcomes. Flow partly mediated the effect of topic on learning outcomes in [4]. Positive affect, but not flow, partly mediated the effect of game play on learning outcomes in [10]. In neither case was enjoyment confirmed as a mediator. Therefore, there is also some evidence that motivational states can be experimentally elevated and, at the same time, they can be associated with enhanced learning.

It is also worth noting that one of the beer brewing studies [3] is the only one in which enjoyment correlations were consistently moderate to strong. We will return to this point in the discussion.

4 Discussion

This review has found that, in eleven experiments conducted by the same lab, with relatively homogeneous samples and with treatments “optimized” by multimedia learning principles, whole-sample associations between learning outcomes and three motivational variables (enjoyment, positive affect, flow) tended to be positive. Average correlations, across all experiments, were all positive and significant. Enjoyment and positive affect correlations were less stable and weaker compared to flow correlations. This pattern of results corroborates results from our study [4] that indicated directly that flow may be more strongly associated with learning outcomes compared to enjoyment and positive affect.

On the theoretical level, the following point is worth mentioning. Questions from the flow questionnaire are most closely linked (compared to positive affect and enjoyment questions) to focused attention on the learning content and increased cognitive activation, which have been posited to be the key causes of improved learning [e.g., 27]. Learners who are focused on learning (as indexed by flow) may still experience neutral rather than positive feelings (as indexed by enjoyment and positive affect). On the other hand, learners having positive feelings may not necessarily be cognitively focused on learning; especially in DGBL-like contexts, wherein enjoyment/positive affect may originate from playing rather than learning. In other words, flow-based questionnaires may better capture slight changes in cognitive activation and attention processes (which are very difficult to measure directly but are essential for successful learning).

We wish to emphasize that this pattern of results is provisional. Despite the fact that the studies were relatively similar, there were some confounding factors. This is an inevitable limitation. For example, in two studies above [3, 10], a learning–enjoyment relationship was still notable. What could be the reason for this? We believe that the culprit is the different (uncontrolled) heterogeneity of participants in terms of developed interest in the instructional domain, i.e., participants’ relatively stable predisposition to re-engage with particular types of content [19]. Specifically, we think more heterogeneous samples could have been recruited for these two studies, explaining generally stronger correlations. To corroborate this idea, in one unpublished study (N = 128; young adults; learning by means of a non-digital game; [5]), we intentionally recruited participants who were diverse in their developed domain interest: the enjoyment–learning outcome correlations were moderate-to-strong (r = .36 − .43), even somewhat stronger than flow (r = .25 − .35) or positive affect correlations (r = .19 − .35). This adds another dimension of complexity to thinking about the relationship between situational motivational factors and quality of knowledge acquisition. When a generally neutral developed domain interest can be expected for a particular sample and a learning situation, the motivational–learning correlations would then be influenced primarily by situational factors. When differing developed domain interests can be expected (e.g., an electromechanical topic), the correlations may also be influenced by this type of (stable) interest. The same can be said with respect to use of media through which the learning message is presented. For instance, computerized slides (i.e., the medium as such) would unlikely trigger hatred or affection in present-day college learners; unlike a new, complex educational game (some may love it, others will hate it).

That said, the methodological implication is that care should be taken when considering what motivational measures are used in future DGBL and multimedia learning studies. Using multiple measures at the same time is advisable, as is controlling for developed domain interest and attitude toward instructional media.

Given the arguments above in this section and in the introduction, can one expect that our results generalize to different contexts (e.g., different labs, different intervention types, different levels of distraction, different age groups, different measures)? This is not guaranteed. Still, it is worth noting that, in this report, flow–learning correlations were in the range previously reported in the context of interest–academic achievement associations [42] and the relation between interest and learning from texts [41]. It can thus be speculated that these correlations reflect general motivation–learning associations rather than something specific to game-based/multimedia learning. Future research should focus more on this issue.

At the same time, we also showed in two studies [4, 10] that motivational states induced by re-design of the intervention can be connected to better learning outcomes. Additional evidence supporting the idea that this is possible exists in DGBL [e.g., 13, 17] and multimedia learning [e.g., 37, 43] contexts. This means that motivational states elevated by specific DGBL/multimedia learning approaches can facilitate learning after all. However, whether they do so depends on the quality of the re-design in question.