Keywords

1 Introduction

Currently, instructors in United States Navy (USN) and United States Marine Corps (USMC) courses are faced with the challenge of modernizing instruction to provide Sailors and Marines individualized training at the point of need. It is becoming increasingly difficult to manage expanding training curricula and growing classroom sizes without the help of additional training days [1]. One potential solution to this issue could be the use of adaptive training (AT). AT has been defined as “training interventions whose content can be tailored to an individual learner’s aptitudes, learning preferences, or styles prior to training and that can be adjusted, either in real time or at the end of a training session, to reflect the learner’s on-task performance” [2]. Although AT systems have been shown to be effective at improving learning outcomes [3, 4], they typically are not implemented as required elements of curricula. Instead, these systems are provided to students as supplemental resources they can use for additional instruction on their own time. This places responsibility on the student to judge their own need for the AT system and use it sufficiently to address that need. Flashcards are a popular form of supplemental instruction used by students to study outside the classroom environment. To that end, we partnered with a USMC course to provide students with an adaptive flashcard system to identify factors that contribute to usage.

Flashcard-based adaptive training has been shown to be effective in many contexts, including geography [5], chemistry [6], species categorization [7], and word pairs [8]. In a pilot study, Whitmer and colleagues [9] developed the Flexible Adaptive Sequencing for Training (FAST) testbed to test whether using adaptive flashcards would be effective for learning outside the laboratory in a real-world classroom. The authors examined course outcomes by comparing the grade-point averages (GPA) and learning objective failures (i.e., failure to meet performance criteria for a particular course objective) among two cohorts of Marines enrolled in the USMC Automotive Maintenance Technician Basic Course (AMTBC). The AMTBC is a 52-day course divided into six sections that teach theories, troubleshooting, diagnosing, and maintenance of light, medium, and heavy automotive systems. Students begin the course by attending classroom-based lectures pertaining to automotive theory before progressing to hands-on sessions detailing the maintenance and repair of vehicles. Though students are provided lecture outlines and technical manuals relating to automotive maintenance, Whitmer and colleagues considered that students would benefit from a self-paced training system incorporating relevant course materials they could reference at the point of need. Additionally, use of the training system was completely voluntary and not required for the course. Results from their study found a 50% reduction in learning objective failures in the cohort that utilized FAST. Critically, the study also showed a high degree of variability of student engagement in terms of FAST usage.

The high variability in FAST usage observed by Whitmer and colleagues [9] is a potential barrier to the successful implementation of FAST in USMC and USN courses, and of AT systems in real-world learning contexts more broadly. Based on their findings, Whitmer and colleagues proposed future research aimed at linking usage rates among students to specific individual difference variables. These authors argued that understanding which traits are associated with system usage could help AT system designers develop systems aimed at increased engagement and, ultimately, improved learning outcomes. Previous AT research has emphasized the need to consider individual differences such as learner goals, motivation, and personality when designing adaptive approaches for instruction [10, 11]. Likewise, student engagement, or student involvement in educationally purposeful activities [12], has been linked with learners’ achievement goals as well as positive outcomes such as graduation [13, 14]. Therefore, we designed the current study to examine individual differences in USMC students as predictors for usage of a flashcard AT system during students’ course. The present study sought to explore the relationships between learning-related individual difference variables and FAST usage to investigate the reasons behind usage variability. Specifically, we chose to investigate associations between self-regulated learning (SRL), achievement goals, and intrinsic motivation (IM) and FAST usage, as these individual differences have been shown to predict both learning outcomes and study strategies [15,16,17,18,19,20,21,22,23,24].

1.1 Adaptive Training with Flashcards

AT is a training approach whereby training is tailored to individual trainees based on their current performance [2]. AT has been shown to be effective in a variety of contexts, including mathematics [25, 26], biology [27, 28] computer programming [29, 30], and medicine [31, 32]. The benefits of AT also extend to military domains involving multi-step functions such as problem solving and decision making [33,34,35,36,37,38]. An important development in AT research has been the incorporation of AT into flashcard-based learning procedures. These AT interventions typically use student performance to address when to retire flashcards and how to space them to take advantage of the spacing effect, which refers to the research finding that study sessions spaced over time facilitate long-term retention more effectively than sessions spaced close together [39].

Students have long used flashcards as a study method to test themselves on information and assess their current level of proficiency [40,41,42,43]. While using flashcards, students typically must make several decisions to assess their current progress. These decisions include determining when they have reached an acceptable level of proficiency for the domain, as well as understanding which flashcards require more study. Students often struggle, however, to make these decisions effectively [44]. This may result from students’ inability to judge their own learning [45], or from their failure to identify effective learning strategies [41]. To address these shortcomings, researchers have fused AT and flashcard-based training approaches by developing systems that adapt the spacing between flashcard presentations [5, 46, and 6], the retirement criteria for individual flashcards [47], or both [6, 27, and 48]. Researchers adapting the spacing between flashcards develop algorithms that select flashcards for presentation based on several criteria, including accuracy, reaction times, and number of trials for previous presentations of a flashcard. Researchers adapting flashcard training based on retirement implement performance criteria for each flashcard that, when achieved, drops the flashcard from training. Prior research has shown that increasing retirement criteria for flashcard leads to greater learning efficiency [49, 50], though these findings do need to be considered in terms of their diminishing returns on long-term retention [51]. During their examination of USMC AMTBC students, Whitmer and colleagues [9] adapted both spacing and retirement to gain insights on how these interventions pair together to influence USMC course outcomes. For the purposes of the current study, however, we chose to only adapt retirement given the mixed findings of adaptive spacing presented by Whitmer and colleagues [9]. In the following sections, we describe the individual differences that may be important in predicting FAST usage.

1.2 Individual Differences in Learning

Self-regulated Learning.

In considering individual differences that may predict FAST usage, self-regulated learning emerged as a potentially crucial predictor. SRL can be defined as the process by which learners monitor and direct their cognition, motivation, and behavior in pursuit of learning goals [52]. Self-regulating learners proactively monitor their progress toward their goals and adjust their methods of learning accordingly. Specifically, Zimmerman [53] outlined eight components of self-regulated learning: setting specific goals, using strategies to achieve those goals (e.g., self-directed practice), monitoring progress toward set goals, restructuring the physical and social context to be conducive to the achievement of goals (e.g., studying in a quiet environment), self-evaluating one’s own learning methods, attributing causation to results, and adapting future learning methods. SRL has been associated with higher course grades [15], standardized achievement tests scores, [18], and scores on classroom assignments [20]. Experimental studies have corroborated these findings; an SRL mathematics intervention in which students were taught SRL strategies (e.g., self-evaluation, goal setting and planning) resulted in higher scores on a mathematics achievement test as compared to a group of students who did not receive the intervention [17].

Central to the current study, flashcard use can be considered a form of self-regulated learning, as it involves self-evaluation and is usually directed by the students themselves rather than by instructors [54]. Indeed, students have reported using self-testing as a means to monitor their learning [55], suggesting the use of flashcards is a self-regulated process. Thus, although students may vary in how effectively they use flashcards, self-regulated learners should be more likely to use a flashcard-based study system at all than students who are less effective at regulating their learning. However, it is important to note that post-secondary students often are not effective self-regulating learners and frequently use inefficient study strategies [44, 56].

Achievement Goals.

Achievement Goal Theory has been a prominent area of research regarding individual differences in learning [57, 58]. Elliot and McGregor [59] defined achievement goals as the “purpose or cognitive-dynamic focus of competence-relevant behavior (p. 501)”. In other words, achievement goals refer to the motivation behind learners’ competence-related behavior, such as academic performance. Achievement goal theorists have established a 2 × 2 framework of achievement goals. The first dimension refers to the types of achievement goals one can have: either mastery or performance goals. Mastery goals are motivated by the development of competence (e.g., mastering a given task or concept), whereas performance goals are motivated by the demonstration of competence relative to others (e.g., performing well relative to classmates). The second dimension in the framework is the valence of achievement goals, which can be classified as either approach- or avoid-based goals. Approach goals are focused on achieving success, whereas avoid goals are focused on avoiding failure [60]. Together, the goal types and goal valences result in four achievement goal orientations: mastery-approach (MAP), mastery-avoid (MAV), performance-approach (PAP), and performance-avoid (PAV). At a high level, MAP-oriented individuals strive to achieve success relative to their own personal goals, whereas MAV-oriented individuals strive to avoid being less successful than their own personal standards [61]. On the other hand, PAP-oriented individuals strive to succeed in performing as well as or better than others, whereas PAV-oriented individuals strive to avoid not performing as well or better than others.

Although the distinction between approach and avoid goals may seem subtle, they have been shown to have differential effects on academic achievement. Specifically, approach goals are consistently better for achievement than avoid goals. For example, a meta-analysis by Baranik and colleagues [16] revealed that MAP and PAP were positively associated with academic performance (e.g., GPA, exam performance), whereas MAV and PAV were negatively correlated with performance. These results were corroborated by experimental studies, where encouraging approach-based goals was more beneficial for academic achievement than encouraging avoidance-based goals [24]. Additionally, experimentally inducing mastery-approach goals benefitted achievement more than inducing performance-approach goals, which corroborates the overall correlational finding that mastery goals are better for achievement than performance goals.

Beyond academic performance, achievement goals also predict studying behaviors. Elliot and colleagues [19] found mastery goals positively predicted the use of deep processing study strategies (e.g., thinking through topics and developing key points rather than simply re-reading course material), effort devoted to studying, and persistence in studying (e.g., allocating more time and effort to topics that were confusing rather than giving up). Though performance goals also positively predicted effort and persistence, they did not predict the use of deep processing study strategies; instead, they predicted the use of surface processing strategies (e.g., re-reading material repeatedly). Geller and colleagues [62] also observed that avoidance goals, regardless of goal type, were associated with increased use of cramming study strategies. More importantly for the present study, however, MAP goals were positively associated flashcard use and self-testing methods of studying, PAV goals were positively associated with use of self-testing strategies (but not flashcards specifically), and PAP and MAV goals were not associated with either method of study. Relatedly, Wallace and colleagues [63] found that mastery goals, regardless of valence, predicted increased use of retrieval practice for studying for an undergraduate course. Taken together, the results of these studies suggest that mastery-oriented students, and particularly mastery-approach oriented students, should be more likely to use the FAST testbed to study for the AMTBC.

Intrinsic Motivation.

Intrinsic motivation (IM) is a psychological construct with a long history of empirical research as it pertains to well-being, performance, and engagement in various contexts. Intrinsic, as opposed to extrinsic, motivation refers to the completion of tasks out of inherent interest or enjoyment rather than for separate outcomes like rewards or punishments [64, 65]. Rather than a trait, IM can be thought of as a state, such that it can be either facilitated or hindered by the environment. For example, Deci and colleagues [66] found that using language that promotes a sense of autonomy rather than a sense of being controlled (e.g., ‘you should’ vs. ‘you must’) during an experiment resulted in higher self-reported IM. Thus, IM is task-specific; for example, a student may not be intrinsically motivated to study for a course, but they may be intrinsically motivated to play a video game.

In addition to its positive effect on psychological states, IM for learning has been reliably linked to learning outcomes [22]. In both a meta-analysis and a series of longitudinal studies, IM was positively associated with GPA for both high school and college students [23]. Intrinsically motivated students also tend to review course content more frequently, take the initiative to complete supplementary academic tasks, and evaluate their learning progress [21]. Thus, the current study is interested in students’ IM for learning AMTBC material, as students high in IM factors may be more likely to use FAST to study for their course.

1.3 The Present Study

Given previously reported variability in FAST usage [9], the present study aimed to explore individual differences that predict usage of the AT system in a USMC course. Specifically, we were interested in the potential relationships between SRL, IM, achievement goals and FAST use, as these constructs have been shown to affect both academic achievement and use of learning strategies in laboratory and classroom environments. Furthermore, all three of these constructs have led to improved learning outcomes when manipulated experimentally, indicating that incorporating features into an AT system that promotes these constructs may result in both higher use and higher achievement. Thus, exploring whether SRL, IM, and achievement goals are related to FAST usage will provide evidence toward individual differences predicting use of AT instructional tools in a military setting, as well as provide insights to improve the ability of future AT systems to increase student learning outcomes. The data reported here are part of an ongoing experiment examining how to increase AT system usage rates by including performance-based feedback as part of the AT system.

2 Method

2.1 Participants

Participants were recruited from two separate AMTBC cohorts. The first cohort consisted of 36 students with a single student electing not to participate. The second cohort consisted of 46 students with a single student electing not to participate. Furthermore, three students’ data were missing from either being re-classed midway through the course or from swapping out their computer. Overall, the sample included 77 participants (one female, MAge = 20.67, SDAge = 2.30).

2.2 Testbed

The Flexible Adaptive Sequencing for Training (FAST) system is a testbed that allows experimenters to manipulate the types of content and how flashcards are presented to students by varying the spacing, retirement criteria, feedback, and other settings for testing purposes. For this study, the FAST testbed was provided to students enrolled in AMTBC to allow them to study course material. FAST was installed by instructors on the course-issued laptops provided to students at the start of the course. Once opened, FAST allowed students to select a content area to study. Study sessions were structured such that training would last until all cards in the deck were correctly answered three out of the last four times they were presented. Otherwise, training would end after 30 min, or when 300 trials were completed if mastery of the deck was not achieved. Students could also opt to end early by closing a training session, which would bring them back to the content selection screen. During training, FAST presented content randomly and logged performance data for each trial presented to students. This data was aggregated to gather usage statistics.

Overall, 18 flashcard decks were available for students to study. The flashcard decks were created using content provided by AMTBC instructors and covered eight automotive systems, including electrical, compressed air, hydraulic, power plant, hydraulic and air-over-hydraulic brakes, and light and heavy suspension. Within the flashcard decks, students received three types of flashcards: identification, function, and location. Identification cards presented an image of a component and asked students to identify its name. Function cards presented an image of a component and its name and asked students to identify its function. Both identification and function cards required students to select the correct answer from four options. Feedback was provided such that if the student answered correctly, their selection would turn green. If the student answered incorrectly, their selection would turn red and the correct answer would turn green. Alternatively, location cards presented students with an image of an area of the automotive system and asked students to click the location in the image where a specific component was located. If the student selected the correct region of the screen, FAST would display “Correct” at the bottom of the screen and circle the component in the image. If the student selected an incorrect component, “Incorrect” would be displayed at the bottom of the screen and the correct component would be circled. Figure 1 provides examples of the three types of cards.

Fig. 1.
figure 1

Examples of location, function, and identification flashcards in FAST. (Color figure oinline)

Full size image

2.3 Measures

Several surveys were administered to participants to measure potential individual difference predictors of FAST usage. The surveys included the Intrinsic Motivation Inventory [IMI; 67] to measure intrinsic motivation. The IMI includes 25 items and four scales: interest/enjoyment, perceived competence, effort/importance, and value/usefulness. Participants indicate how true a statement is by using a 7-point Likert scale with anchors that included: 1-not at all true, 4-somewhat true, and 7-very true. To measure self-regulated learning, the Self-Regulated Learning survey [SRL; 68] was administered. The SRL survey is a 30-item questionnaire with five scales: goal setting, help seeking, self-study strategies, managing physical environment, and effort regulation. Each item presents a statement (e.g., I contact someone to discuss my understanding) and participants rate their agreement on a five-point Likert-scale (1-strongly disagree, 2-disagree, 3-slightly agree, 4-agree, 5-strongly agree). Finally, the Achievement Goal Questionnaire – Revised (AGQ-R; [59] was used to identify the goal orientation of participants in the study. The AGQ-R is a 12-item questionnaire with four scales: mastery-approach, mastery-avoidance, performance-approach, and performance-avoidance. Each item presents a statement (e.g., my aim is to perform well relative to other students) to which participants indicate their level of agreement on a five-point Likert-scale (1-strongly disagree, 5- strongly agree). Additionally, a general demographics questionnaire was given to participants to record age, rank, gender, and other demographic information.

2.4 Procedure

The study took place at the Marine Corps Combat Service Support School (MCCSSS) schoolhouse where the AMTBC is taught. Researchers administered the study procedures on two separate days: once prior to students beginning their coursework, and once towards the end of the course approximately 10 weeks later. On the first day, students were briefed on the experiment and informed that the content within FAST was developed by their instructors and could be a useful tool for ensuring their success in the course. They were also told that they could use FAST as much or as little as they wanted throughout the course. Afterwards, all instructors and course staff left the classroom so researchers could obtain consent from the students. Students were also informed that their instructors would have no way of knowing whether they chose to participate or not. Those who did not wish to participate in the research study were informed they would still have access to FAST, but their data would not be collected. After the consent process, researchers administered the pre-test, general demographics questionnaire, AGQ-R, and SRL. Next, researchers led students through a tutorial of FAST that included how to find the program, login, and select content for training. Once students were comfortable with this process, they were instructed to spend the next 30 min training with flashcards from the Electrical Theory deck. After the 30 min had expired, researchers told the students that FAST would be available for them to use as a study tool during the rest of their course and they could use it as little or as much as they preferred. Ten weeks later, researchers returned to the schoolhouse to administer a post-test along with the IMI. Afterwards, researchers conducted informal interviews with participants before they were debriefed and thanked for their participation in the study.

3 Results

3.1 Statistical Analysis

To assess whether our selected individual difference variables (i.e., SRL, AGQ-R, IMI) predicted FAST usage, we conducted exploratory Pearson product-moment correlations using each of the individual difference variables and FAST usage indicators (i.e., number of times participant used FAST, total trials completed, average number of trials completed, number of unique days trained in FAST, number of cards mastered, and number of unique decks the participant trained on). Given that we were interested in individual differences that related to usage, we excluded participants who did not use FAST after the initial training session from our analyses. Thus, our final sample included 36 participants. Of the included participants, one did not complete the AGQ-R; sample sizes for AGQ-R and usage correlations are therefore reduced to 35 participants. In the subsequent sections, we report the correlations between each individual difference variable and FAST usage.

3.2 Overall Usage

Consistent with Whitmer and colleagues [9], usage rates varied, with most students not using FAST beyond the initial introduction (see Table 1). As expected, those who used FAST multiple times tended to complete more trials (see Table 2). Interestingly, students who used FAST multiple times completed fewer cards during subsequent uses relative to the average number of trials completed by all students during their initial FAST tutorial on Day 1 of the study. Similarly, the number of mastered cards was generally low, indicating that students ended training prior to mastering all the cards in the deck.

Table 1. Descriptive statistics of FAST usage. Statistics in parentheses represent students who interacted with FAST beyond the initial introduction.
Full size table
Table 2. Correlations between FAST Usage Metrics for students who used FAST more than once.
Full size table

3.3 Achievement Goals

Table 3 includes the correlation coefficients for the AGQ-R scales and FAST usage metrics. The MAP goal orientation was significantly and positively associated with the number of trials completed in FAST, whereas the MAV orientation was associated with a higher number of unique days trained in FAST. The PAV orientation was also positively correlated with the number of unique days trained, but the PAP orientation was not associated with any usage indicators.

Table 3. Correlations between Achievement Goals and FAST Usage.
Full size table

3.4 Self-regulated Learning

None of the scores on the SRL scales significantly correlated with FAST usage (see Table 4).

Table 4. Correlations between Self-Regulated Learning and FAST Usage.
Full size table

3.5 Intrinsic Motivation

As seen in Table 5, scores on all four IMI scales were associated with a higher number of completed FAST trials. Perceived competence, effort/importance, and value/usefulness scales were associated both with a higher number of unique days and unique decks trained in FAST. Additionally, scores on the Effort/Importance and Value/Usefulness scales of the IMI were significantly and positively correlated with total number of FAST uses. Lastly, effort/importance was associated with a higher number of mastered cards.

Table 5. Correlations between IMI scales and FAST Usage.
Full size table

4 Discussion

The present study explored how a flashcard-based AT system was used during a USMC course and whether there were associations between usage, SRL, IM, and achievement goals. To that end, the study presented correlations for each of these constructs in terms of several usage metrics in a real-world setting. Overall, usage rates varied considerably, with wide ranges of usage among each of the usage metrics (e.g., number of FAST uses, total number of trials completed, etc.). Additionally, only 36 of the 77 total participants used FAST after the initial training session. Based on interviews with students following the study, these results may be due to students forgetting FAST was available to them. Instructors did not require students to use FAST, and many students reported that they forgot FAST was installed on their laptops despite both the initial introduction by the instructor and researchers and the application being located directly on their desktop screen. Moreover, students who engaged with FAST after the initial training session mastered fewer cards relative to the average number of cards all students mastered during their Day 1 introduction to FAST. Interviews conducted with students after the study also revealed some students thought the content was not relevant for the course section with which they wanted supplemental training. The content provided within FAST was intended to cover function and identification of automotive components, which are primarily covered in the first two sections of the course. Later sections reinforce components’ function; however, these sections focus heavily on hands-on maintenance actions and troubleshooting, which FAST content did not cover. Students also commented they thought the content was too easy and only did a handful of flashcards before closing FAST or moving on to a different content area. This behavior is in line with research suggesting learners tend to exercise poor judgement regarding when to stop studying a given flashcard [39, 54], which may have contributed to their ending training prior to reaching the mastery criteria.

Regarding individual differences and FAST usage, results indicated that achievement goals and IM, but not SRL, were positively associated with aspects of FAST usage. Although previous research has shown that SRL relates to positive learning activities, the current study did not find a significant relationship between SRL and FAST usage. As the current study focused on FAST usage and not other forms of positive learning activities (e.g., referencing technical manuals, course outlines, and other course-related materials asking for feedback from instructors), we may not have captured the effects of SRL in our outcome measures. That is not to say students with higher levels of SRL did not implement more successful strategies; these strategies may have simply manifested in other behaviors that were not measured during this study. Future studies examining AT system usage should consider how learning strategies are implemented and assessed when gauging how and why students interact with their system.

Regarding achievement goals, associations between goal orientations differed across usage metrics. Both avoid-orientations of the AGQ-R (MAV and PAV) were associated with a higher number of unique days trained in FAST, but neither were associated with any of the other usage metrics. MAP, however, was associated with a higher number of total trials completed. MAP and PAV have been associated with more use of self-testing [19], and mastery goals, regardless of valence, have been associated with more use of retrieval practice [63]. Higher levels of avoidance led to an increase in number of system uses; however, there is no evidence these goal orientations led to more trials completed within the training system. As previous research has discussed the association between avoidance goals and a decreased likelihood of establishing structured study routines [19, 62], designers of future AT systems may consider including features that provide structure during use to facilitate approach goals, which have been linked to improved performance [24]. Examples of such features include intermittent knowledge checks during study and feedback detailing training progress.

Results also revealed several correlations between IMI scales and flashcard system usage. All scales of the IMI correlated positively with total number of trials completed by students. This finding is consistent with previous research indicating higher levels of IM lead to behaviors that enhance learners’ engagement in academic tasks on their own initiative [e.g., 21]. It is encouraging that students’ reported understanding of the importance and value of the course material correlated with usage of the course’s flashcard training system. Emphasizing these qualities of the course, as well as how the training system may facilitate students’ feelings of competence in course topics, may lead to increased student usage of adaptive training systems in future courses. Future AT systems should consider implementing features to highlight the importance of critical course information to facilitate intrinsic motivation and, ultimately, increased system usage. These features could take the form of summary notes at the conclusion of sections as well as information conveying practical applications of course content.

4.1 Limitations and Conclusion

Despite our efforts to extend the research conducted by Whitmer and colleagues [9], the current study was limited in some ways. For example, students were not required to use FAST during their course nor were instructors obligated to remind students the tool was available, as this could have potentially confounded the findings and introduced coercion. Though this may have reduced the overall usage of FAST and our sample size, it did allow us to identify relationships between individual differences and those who chose to use FAST on their own. Relatedly, we are unable to present individual difference findings that explain why students used FAST more than once throughout the course. As discussed previously, there are a multitude of factors that may have hindered students’ engagement with FAST, including forgetting the system was available. Future research should investigate students’ reasons for using and not using AT systems, as well as their preferences for system features. These limitations notwithstanding, the current study provides valuable insight into certain individual differences that relate to supplemental AT system usage in a USMC classroom. These results extend the research conducted by Whitmer and colleagues [9] by identifying individual differences that predict FAST usage. Overall, our study suggests that goal orientation and IM are related to FAST usage; however, no significant correlations were uncovered for our measure of SRL. Future research should consider these relationships when exploring ways to promote engagement with AT systems, as these systems have the capability to modernize learning approaches to USN and USMC courses.