Introduction

Learner control refers to the extent to which a learner can affect their learning experience by altering their learning environment, usually in the context of learning delivered via computer (Friend and Cole 1990). The study of learner control became more common with the increased popularity of personal computers for learning in the 1970s (Shyu and Brown 1992a, b), making learner control more accessible to designers. In the recent decades, the study of learner control has shifted almost entirely to the context of web-based instruction (WBI), in which instructional content is provided to learners via the Internet and through a web browser, enabling learning experiences not previously possible (Khan 1997). In organizational training, WBI has become a popular training medium, accounting for approximately 20 % of formal learning hours in organizations (American Society for Training and Development 2012).

In WBI, some element of learner control is almost always present, which is reflective of a general movement in the practice of both education and organizational learning toward self-directed learning (Valjataga and Laanpere 2010). This change in practice is in part triggered by an increasing body of scholarly evidence that the traditional model of instruction, in which an instructor in a classroom is viewed as an expert who must find approaches to deliver an objectively defined body of knowledge unto the eager-to-learn masses (i.e., an objectivist theory of learning), is not as effective at creating knowledge as approaches considering a learner to be a complex individual that generates meaning through his or her perceived experiences (i.e., a constructivist theory of learning). This shift is sometimes referred to as a movement from the study of teaching to that of learning, and it has become a common theoretical perspective in both educational and organizational domains (Kraiger 2008; Lin and Hsieh 2001).

Despite its theoretical basis in constructivism, empirical evidence supporting learner control is highly inconsistent. Research comparing learners exposed to WBI with learner control to those without is full of contradictions, with studies reporting negative, positive, and null effects on learning (Niemiec et al. 1996), reactions to training (DeRouin et al. 2004), and time to complete training (Kraiger 2008). Few, if any, learner control effects appear consistently, which presents a major challenge for a training designer trying to decide whether or not to implement learner control in a new WBI training program.

We contend that these conflicting empirical findings and resulting practical difficulties are primarily due to discrepancies between studies in the operationalization of learner control resulting from underdeveloped theory regarding the nature of learner control interventions. Learner control is typically treated as a unidimensional concept where increased control is assumed to be better in some way; however, learner control can be operationalized dramatically differently from study to study. For example, one researcher might operationalize learner control by allowing learners to complete modules in an order of their choosing, whereas another might do so by allowing learner to skip content about which they do not wish to learn, and another might allow both.

Critically, current learner control theory suggests only positive effects of increased control (Milheim and Martin 1991). However, the various approaches called learner control have dramatically different theoretical processes that have not been well-explored (Ross and Morrison 1989). Even when multiple types of control are given in a single training program, learner use of different types of control are weakly or negatively correlated, precluding the existence of any clear higher-order factors (Cavanaugh and Landers 2014).

Three major meta-analyses examining the comparative effectiveness of learner control among adults on knowledge and skill outcomes appear in the published literature. The first meta-analysis by Niemiec et al. (1996) examined the effect of learner control in computer-assisted instruction across 24 studies. This examination is thus broader than WBI, and includes computer-based approaches that do not use the web (e.g., CD-ROM based instruction). The researchers identified a moderate, negative effect of learner control across outcomes and types (d = − 0.41; k = 24), with a slightly positive effect for elementary school students (d = 0.07; k = 9), a moderately positive effect for high school students (d = 0.28; k = 4), and a moderately negative effect for college students (d = − 0.23; k = 12).

The second meta-analysis by Kraiger and Jerden (2007) dramatically expanded and improved upon Niemiec and colleagues’ learner control framework, with a critical change in focus to adult populations, as significant concerns about the capabilities of children to utilize learner control appropriately (Goetzfried and Hannafin 1985; Niemiec et al. 1996) limit the generalizability of the findings. Kraiger and Jerden thus conducted the first meta-analytic investigation of the impact of learner control on adults, finding a small effect of learner control across learning outcome categories (d = 0.19) in 30 studies of 2655 people. Examinations of the dimensionality of learner control they proposed were limited but provocative. After describing a four-dimensional framework of control consisting of pace, sequence, content, and advisement control, only two moderators were examined empirically: “control over pace/navigation” (d = 0.34; k = 19; N = 1661), which collapsed two proposed types of learner control into one analysis, and “control over content” (d = − 0.03; k = 7; N = 611). Thus, the dimensions of learner control meta-analytically examined did not correspond to the framework of learner control that they presented. Despite this limitation, Kraiger and Jerden’s meta-analysis did provide preliminary evidence that learner control can be made more or less effective based upon how it is implemented within adult populations and served as a catalyst for subsequent learner control research within the domain of adult learning.

The third and most recent meta-analysis by Carolan, Hutchins, Wickens and Cumming (2014) took a somewhat different approach from the previous two, framing learner control as one example of an “active learner method” alongside exploratory learning, proposing that both increase cognitive load. According to this perspective, any active learning technique should improve learning if the increased cognitive effort among learners is learning-directed but harm learning if the effort is contextually directed (i.e., toward the system they are using). In line with this broader treatment of learner control, Carolan and colleagues also utilized broader inclusionary criteria for their meta-analyses than either Niemiec et al. (1996) or Kraiger and Jerden (2007); for example, their analysis includes high school students (Tennyson and Buttrey 1980) despite stating their inclusionary criteria as “adult populations representative of the Army population” (Carolan et al., p. 1003). As a result, their meta-analysis included a variety of studies that may not generalize well to the decisions made by workplace training designers implementing WBI. In addition, although Carolan and colleagues report that their overall learner control effect estimate is calculated from 144 effects, only 47 sources contributing learner control effects are provided in their reference list, suggesting that individual studies containing multiple effects calculated from the same sample are likely treated as independent effects in their analysis despite current recommendations to the contrary (Schmidt and Hunter 2015). With these caveats in mind, their overall results were similar to those of Kraiger and Jerden (2007); specifically, the overall effect of learning was near zero (g = .02; k = 144), and some differences were found regarding type of learner control implemented (i.e., “LC of pace” was found to be beneficial, and “LC of feedback/practice” was found to be harmful). Unfortunately, the number of studies included in each of these moderator analyses was small, ranging from 8 to 22, and considering the effects of non-independence at small sample sizes, makes these estimates difficult to meaningfully interpret. It is also unclear how the four moderators tested were chosen (i.e., branching, pace, feedback/practice, and lesson truncation), and the theoretical suggestion regarding cognitive load was never empirically tested.

Given the challenges described above and these prior meta-analyses, the major purposes of this article are as follows. First, we develop an expanded framework of learner control features in WBI tied closely to the technology used to permit that control based upon a review of the extant learner control literature. In doing so, we will fill an important gap in our understanding of the structure of learner control as well as how specific learner control implementations affect learning. Second, responding to the call by Sitzmann et al. (2006) to explore which features of learner control actually facilitate learning in organizational learning, we link this framework to prior theoretical and empirical research suggesting the impact of each dimension. Finally, we empirically evaluate each dimension of objective learner control using psychometric meta-analysis (Schmidt and Hunter 2015) where a sufficient number of studies are available to evaluate them. In doing so, we also present a meta-analysis with the most comprehensive aggregation of studies to date in this domain.

Prior Objective Learner Control Frameworks in WBI

Early studies of learner control did not typically specify a framework of control features, and they often treated learner control as a single unidimensional construct. For example, Hannafin (1984) discussed learner control as an extension of the study of the individual difference locus of control, labeling the concept in WBI as “locus of instructional control” and defining it as either internal (i.e., controlled by the learner) or external (i.e., controlled by the program). Distinctions in the ways internal locus of instructional control might be realized were not discussed. Instead, it was defined as a single continuum, ranging from fully internal to fully external. Tennyson et al. (1985) examined learner control by assigning participants to either an adaptive time condition where the computer automatically controlled the material presentation speed or a learner-control condition where the learner could set their own pace, finding that learner-controlled pacing produced both poorer learning and retention of learned concepts. However, control over pacing was not fit into any larger framework or theory of control.

In an early paper attempting to remedy the lack of theory to explain effects found in the study of learner control, Milheim and Martin (1991) presented three theoretical frameworks, all of which are commonly used to support the use of learner control across types. From these frameworks, they argued that increasing control should generally benefit learners. First, they argued for Keller’s (1983) motivational theory of learning, which posits that interest, relevance, expectancy and satisfaction of the learner are the primary drivers of learning. By allowing users to choose the sequence of content experienced, learning outcomes should be higher. Second, they argued that attribution theory (Bar-Tal 1978) suggests learner control is effective because it improves learner perceptions of locus, stability, and controllability. Specifically, if learners attribute the cause of their learning to their own choices, their learning should increase. Third, they argued that information processing theory (Gagne 1985) suggests learner control is effective because it allows learners to mentally organize information for themselves in a meaningful way. They also argued for three types of learner control: the pace of exposure to content (pacing control), the sequence of content presented (sequence control), and the quantity of content (content control). Milheim and Martin furthermore described the use of learner control with advisement, during which learners are provided suggestions regarding how they might best use their control over content, sequence, or quantity as they progress through WBI. Following such suggestions is optional for the learner. However, Milheim and Martin considered neither the inclusion of advisement nor the learner’s ability to react to advisement to be distinct types of control. Neither the three theoretical frameworks nor the designs were tested empirically. Furthermore, the authors did not consider any possible contextual harm done by allowing control or particular types of control. Instead, they relied solely upon evidence broadly supporting the use of control across situations.

A more recent and comprehensive framework for understanding learner control features in WBI is that of Kraiger and Jerden (2007), who describe four types of learner control features consisting of pace, sequence, content, and advisory control, citing both of the studies described above as their foundation. In their framework, pace, sequence, and content control are defined as by Milheim and Martin (1991). Advisory control, labeled as such, does not appear in any previous framework or model; it was defined by Kraiger and Jerden as “program-generated advice that informs learners of their progress or suggests a course of action” (p. 67). Most likely, this redefines the concept of learner control with advisement from Milheim and Martin’s work, who did not consider it a distinct type of control, as a new dimension of a proposed learner control construct. Kraiger and Jerden further utilize the terms “learner control” and “program control” introduced by Ross and Rakow (1981) to refer to the extremes on each continuum representing the four distinct learner control dimensions. In other words, a program forbidding changes to content sequence by the learner utilizes program-controlled sequencing of content; a program allowing absolute control by the learner over sequence utilizes learner-controlled sequencing of content. Presumably, a continuum of middle-grounds exists between the two.

In the most recent model of objective learner control, Karim and Behrend (2014) built upon Kraiger and Jerden’s (2007) model by adding a new major category of control called scheduling control, which includes trainee choice about the location and time frame of training. This type of control is commonly considered central to organizational training but is rarely examined empirically (Orvis et al. 2009). Kraiger and Jerden’s dimensions were also included in Karim and Behrend’s model as a second broader dimension of a single multidimensional learner control construct, referring to those dimensions as instructional control. In their article, Karim and Behrend demonstrated that learners perceived scheduling control differently from instructional control, although they did not explore the impacts of specific dimensions empirically.

Proposed Objective Learner Control Framework

Given this prior work and a thorough review of the empirical learner control literature, we have developed the nine-dimensional framework, hierarchically organized within three broad types, that appears in Table 1 and Fig. 1. The purposes of this framework are (1) to guide training design decisions for practitioners during the implementation of training and learning programs, and (2) to provide researchers an organizational system by which to isolate theoretically meaningful learner control effects.

Table 1 Framework of objective learner control in WBI
Full size table
Fig. 1
figure 1

Hierarchical framework of objective learner control

Full size image

Unlike the researchers who proposed the model, the present framework is based upon (i.e., Karim and Behrend 2014), we have labeled this a “framework” of objective learner control instead of a “model” for two reasons. First, evidence to be presented in this meta-analysis suggests that the dimensions can be and are implemented by training designers independently. This suggests that there is no single latent “learner control design” construct. Second, even when the dimensions are implemented simultaneously, learners utilize them inconsistently, and this usage lacks a latent factor structure (Cavanaugh and Landers 2014), suggesting there is also no single latent “learner control usage” construct. Thus, we frame learner control as a family of training design decisions grouped together by instructional philosophy and designer intent rather than by any underlying higher-order causal processes.

One of the major criticisms of prior learner control research is that definitions of learner control have insufficient precision to actually describe what the learner can control. Because objective learner control has no latent structure, this is particularly problematic, leading at least one researcher to label the entire area of study “pseudoscience” (Reeves 1993). This ambiguity in the extant literature has occurred because learner control studies typically combine a variety of types of learner control into a single unidimensional construct ranging from none to high (e.g., Orvis et al. 2011). This approach likely mixes the effects of dissimilar features into a single composite effect, masking the more subtle orthogonal and interactional effects of each individual feature contained therein (Cavanaugh and Landers 2014). By meta-analytically investigating the components individually, the effect of each dimension can be examined in a way that is difficult to do within a single empirical study.

In comparison to Kraiger and Jerden’s (2007) framework, the present framework first splits content control into two elements and advisory control into two elements based on differing theoretical support for the two approaches. Second, it adds style control, time control, and location control from Karim and Behrend’s (2014) model. At the highest level, dimensions are organized into three broad types by intent: instructional control allows learners to change the content of training, style control allows learners to change the presentation of that content, and scheduling control allows learners to change when and where that presentation takes place. In the following sections, we will describe each of the nine dimensions, their implementations, and current theoretical and empirical support for their use. We also present hypotheses based upon current learner control theory suggesting increases in control are generally beneficial to learners (Milheim and Martin 1991), but we also note specific types of control where empirical evidence or competing theoretical perspectives suggest a particular advantage, or in one case, suggest the opposite effect.

Instructional Control

Instructional control is the most extensively researched of the three higher-order control types and consists of six specific dimensions: skip, supplement, sequence, pace, practice, and guidance control.

Skip Control

Skip control enables the learner to skip content that the training designer has included as a default part of training. Key to this definition is that a default path exists; that is, if the trainee proceeds through training content in a linear fashion from start to finish, they will by default experience all of that content, which is what most learners tend to do even when control is provided (DeRouin et al. 2004). Skip control is commonly implemented with sequence control because the most straightforward way to skip content is to navigate past it by clicking on a hyperlink to another section of course content. However, skip control might be implemented without sequence control. For example, at the end of a module, the trainee might be asked: “The first module reviews foundational material; do you wish to complete this review?”, but then be required to complete the following modules sequentially. Thus, with learner-controlled skipping but program-controlled sequence, all trainees experience the content in the same order but may not see identical content. Complete skip learner control indicates that the user would be able to skip all content if they chose to do so; complete skip program control indicates that the user would be required to view 100 % of content before completing training.

If learners have a finite amount of time to spend viewing course content (a common restriction in an employee training setting), the ability to skip sections of content that employees already know allows for greater time spent in areas that they do not. Time on task is one of the most powerful and consistent predictors of knowledge gained in training (Brown 2001; Ericsson et al. 1993; Simon and Chase 1973). Thus, if a trainee accurately assesses their knowledge of the overall training content domain, correctly identifying their strengths and weaknesses, skipping content in areas of strength to focus upon areas of weakness, overall learning should be greater for those given such control. In addition, when training targets a particular level of learner proficiency, learners may spend less overall time in training, decreasing time spent away from job tasks.

Supplement Control

Supplement control enables the learner to add content to WBI that the training designer has not included as a default part of training. Much like skip control, key to this definition is that a default path exists; that is, if the trainee proceeds through training content in a linear fashion from start to finish, they will by default experience a certain amount of content, but that additional content is available if they choose to view or experience it. Supplement control can be implemented in many different ways. For example, hyperlinks might appear in course content directing learners to a popup window with additional content, or sequence control might be incorporated in combination with supplement control such that learners experience an alternate, more content-filled path through training than those that do not choose that path. As with skip control, supplement control can be used with or without other types of control, including skip. For example, Cavanaugh and Landers (2014) implemented skip control by allowing learners to skip content along the default path through training but also implemented supplement control by providing links to additional information that learners were not required to follow. Either of these features could have been eliminated without affecting the other. Critically, they found behavioral indicators of skip control usage to be uncorrelated with that of supplement control usage.

Supplement control enables learners to view extra instructional content in areas of personal interest. This permits an individualized learning experience at the moment of maximum motivation to learn, similar in principle to “just-in-time” approaches to learning (Bolton 1999; Marquardt 1996). In traditional linear WBI, a learner only has access to the default content provided by the training designer, but with supplement control, learners are encouraged to view additional information if they believe it would be useful or interesting. This ability may be more valuable for learners inclined toward self-directed learning (Dynan et al. 2008; Schunk and Zimmerman 2008), because it provides a specific starting point for learning beyond the content contained in formal training.

Sequence Control

Sequence control enables the learner to order course content. It is typically operationalized as a list of hyperlinks to other sections of course content, such that users can jump to any content they wish at any time. This may be done with text or multimedia (e.g., Orvis, et al. 2011). Some finer distinctions of sequence control could be drawn, such as the use of sequence control in general (allowing complete freedom of movement), across modules (requiring learners to view content within a module in order but allowing them to complete the modules themselves in any order), or within modules (allowing learners to view content within a module in any order but requiring them to complete modules in order) but such variations are uncommon in the current research literature (with the notable exception of Gray 1987). Like other forms of control, sequence control can be used independently of other types of control. For example, sequence control but neither supplement nor skip control is present if learners are required to view all material at least once before completing training but can move throughout training at will using navigational hyperlinks.

The major theoretical support for sequence control is provided by Milheim and Martin’s (1991) description of information processing theories. When trying to create meaningful mental models, learners can use sequence control to find the most relevant information needed immediately, to the extent that the WBI software allows this and the learner is able to identify where it might be. In this way, learners are able to construct their own understanding more efficiently (Kraiger 2008).

Pace Control

Pace control is defined as the freedom of the learner to choose how much time to spend on each topic or page of content in training. Most commonly, pace is program-controlled when the instructional medium is experienced only once, for a limited amount of time. For example, pace is program-controlled when the complete instructional experience is a single instructional video that cannot be paused. Unlike the other types of instructional control, the presence of pace control might be considered the “default” in WBI; learners usually can spend as much time as they want looking at different portions of training content, even if they are restricted in other ways. Greater training designer effort is generally required to restrain pace control than to provide it. Thus, although pace control could theoretically be removed in WBI, we suspect it is quite uncommon in practice. For example, Alonso and Norman (1996) examined learner control in the context of a simulation watched by the learner; in this study, once the simulation was started, it ran continuously without interruption. In most modern contexts, however, WBI is likely to consist of text and streaming videos, both of which are difficult to restrict in terms of pacing.

Pace control should benefit learners due to learner variation in cognitive ability, an individual difference variable often defined as the ability to learn (Le Pine et al. 2000). Those lower in cognitive ability, with lesser ability to learn, are likely to require more time to reach the same level of learning as others with greater cognitive ability. To the extent that all learners can correctly identify the optimal amount of time to spend on each piece of content within a training program that they are learning, pace control should result in better learning outcomes. Unfortunately, little research is available on how people actually make such judgments.

Practice Control

Practice control allows learners to choose whether or not they complete assessments that could in principle be used to measure knowledge or skill gain, or exercises intended to elicit targeted knowledge or skill. Two harmful effects likely occur when learners skip practice by choosing not to complete assessments or exercises. First, recent evidence on the testing effect (Roediger and Karpicke 2006) suggests that the simple act of completing assessments (i.e., practicing remembering) improves learning, even if no feedback is provided on their performance. Second, a great deal of evidence demonstrates that feedback, when provided and interpreted correctly, improves both performance and learning outcomes (Kluger and DeNisi 1996; Bell and Kozlowski 2002; Azevedo and Bernard 1995). A learner choosing to avoid exercises and assessments reduces the overall impact of both the testing effect and the feedback effect (if feedback is provided).

Guidance Control

Guidance control provides learners a choice of follow-up action whenever feedback is presented. When combined with program control of practice (i.e., mandatory completion of any knowledge and skill assessments), guidance control may be referred to as adaptive guidance (Bell and Kozlowski 2002). For example, at the end of a training module, the learner completes a quiz; if that quiz is completed poorly, a suggestion is provided to the learner to review the previous module of content or to review additional optional materials. However, guidance control can be implemented with or without practice control; for example, a trainee might proceed through a training program only to experience a required quiz (program control of practice) at the end of a module. Upon failing to reach some predetermined “pass” level, the trainee could then be required to retake the module (program control of guidance). Learner control of guidance is only provided when the learner can choose what to do as a result of a score they have received. Although program control of guidance can be implemented with learner control of practice (i.e., granting learners the ability to take or skip assessments at particular stages of a linear training program), we suspect this combination is rare in the field. For guidance control to be effective, it likely must be paired with other types of control, such as sequence control in order to facilitate review of previously seen content or supplement control in order to enable review of optional remedial content.

Guidance control should benefit learners because for guidance control to even be possible, feedback has already been provided, which is critical to an employee’s ability to improve (Kluger and DeNisi 1996) and encourages learners to choose to engage in self-regulated learning (Butler and Winne 1995). By prompting the learner to experience remedial training content or return to training content in areas of weakness, designers can add instruction individualized to the learner, one of the primary advantages to WBI over traditional instruction (Azevedo and Jacobson 2008).

Style Control

Style control allows learners to alter aspects of the experience of training in ways that do not alter their exposure to course content (DeRouin et al. 2004). For example, Behrend and Thompson (2012) allowed trainees to change the physical appearance, personality, and feedback style of an animated agent delivering training content, finding effects on self-efficacy and training motivation. Style control can be provided over any such contextual element of training; for example, Landers and Callan (2014) allowed trainees to change the avatar that represented them in interactions with other learners. Because style control does not alter the content itself, the benefits of style control are primarily motivational as suggested by attribution theory; specifically, as the locus of controllability shifts to the learner, the learner will be more motivated to exert that control and experience positive affect. Similarly, self-determination theory suggests that increasing perceptions of autonomy will increase motivation (Ryan and Deci 2000).

Scheduling Control

In real organizations, the choice of scheduling is often restricted subtly and distally. Users are often provided a specific span of time (e.g., within the next week, by February) to complete a training program but are free to identify a specific time to complete the training on their own. When a learner has a highly restrictive time window in which to complete training, even if other learner control features are present, we contend that there is less likely to be sufficient time to engage in beneficial self-regulation, encouraging learners to take a performance orientation (i.e., approach with a goal of completing training) rather than a mastery orientation (i.e., approach with a goal of learning as much as possible). Because learner control features are optional for the learner, time and location constraints are additionally likely to have a diminishing effect on the use of those features and as a consequence, learning (Karim and Behrend 2014). We would expect restrictions on location (e.g., in the break room on the computer used for employee training) to introduce similar psychological consequences. Alternatively, scheduling control can be used to ensure employees have a distraction-free environment and dedicated work time in which to complete training, both of which should improve concentration.

Meta-Analytic Hypotheses and Research Questions

Given the framework described above and the availability of a meta-analytic database, we initially sought to replicate Kraiger and Jerden’s (2007) finding that the use of learner control results in superior reactions to training.

Hypothesis 1

Those completing training with learner control have more positive reactions to training than those completing training without learner control.

Next, we sought to test the effect of each dimension on all three outcomes of interest commonly studied: reactions, knowledge, and skills. However, we encountered roadblocks in the existing research literature that rendered several untestable. First, we discovered that restriction of pace control is uncommon in current applications generalizable to WBI. In few WBI studies that we could locate researchers did use complete program control of pace, even in experimental control conditions that researchers labeled as representing “no” learner control. More typically, text or video content was presented that learners could linger on or rush through at will. Even when time control was not provided (e.g., if trainees had one hour to complete a training program) and sequence control was not provided (i.e., trainees were forced to navigate through many webpages of content from start to finish), trainees were generally free to spend as much time on individual pages of content as they wished within that larger timespan in existing research. In short, exclusion of articles containing pace control in their control condition would have eliminated approximately 84 % of our database. Given this, we were unable to empirically examine the effects of pace control. This finding also allows us to note the troubling lack of consistency among researchers implementing and describing this dimension.

Second, we were unable to test style control because only one study that we could locate experimentally manipulated it (i.e., Gay 1986). This study also was not an exemplar for the effect of style control because it also included both pace and sequence control in its learner control condition, muddling the effect.

Third, we were unable to test either time or location control, because both of these types of control were perfectly confounded with control group design. Specifically, all studies that provided either type of scheduling control to those in the learner control group also provided it to their comparison group, even if no other features of learner control were present. For example, Orvis et al. (2011) randomly assigned two existing college courses to either “high” or “low” learner control conditions; however, both conditions consisted of existing online college courses. Thus, students in both of their quasi-experimental conditions had both time and location control, although low control students had no additional forms of control. Because eliminating studies with this design would have eliminated all field studies from our database, and because the presence or the absence of scheduling control was held constant across conditions in every learner control study we located, we instead chose not to include scheduling control in our database development and meta-analytic analyses, although we include them in descriptive analyses.

These limitations of the extant literature, combined with the theoretical background described above, resulted in five additional hypotheses regarding the framework to be tested, which follow. Importantly, each of these hypotheses assumes that trainees are motivated and capable of utilizing each type of learner control effectively.

Hypothesis 2

Those completing training with skip control have more positive knowledge and skill outcomes than those completing training without skip control.

Hypothesis 3

Those completing training with supplement control have more positive knowledge and skill outcomes than those completing training without supplement control.

Hypothesis 4

Those completing training with sequence control have more positive knowledge and skill outcomes than those completing training without sequence control.

Hypothesis 5

Those completing training with practice control have more negative knowledge and skill outcomes than those completing training without practice control.

Hypothesis 6

Those completing training with guidance control have more positive learning outcomes than those completing training without guidance control.

Beyond the testing of our framework, we also include three additional questions of common interest to learner control researchers. First, in the study of a technology, time is a more critical area of focus than in the study of stable psychological characteristics because technology evolves at a readily observable pace. While WBI today shares many characteristics in common with WBI of even five to ten years ago, the technology underlying its use has become more sophisticated and more user-friendly with our increased understanding of the psychology of human–computer interaction and application of this knowledge to web design (Card et al. 1983). In their meta-analysis of the learner control literature, Kraiger and Jerden (2007) noted the rapid development of technology within the field, finding a positive correlation between effect size and publication date (r = .41; k = 30; p < .05), crediting the effect to technological progress. We expect to replicate this finding, but with a more precise effect estimate given a much larger sample size.

Hypothesis 7

The effect of learner control is greater for more recently published studies.

Second, a common concern in learning research is that results from student samples may not generalize to organizational samples and vice versa. Given this, we seek to compare students experiencing learner control while participating for course credit, students taking courses in artificially constructed settings, and adult learners outside of college settings.

Research Question 1

What is the impact of sampling on the effectiveness of objective learner control?

Third, a common challenge in training research is rigorous experimental design. Training studies often incorporate quasi-experimentation on preexisting groups or permit trainees to self-select into conditions. This may present a substantial challenge in interpretation, because the use of preexisting groups without control variables can influence group post-training test results in unpredictable ways. Self-selection additionally introduces a systematic positive bias to the results of such studies if learners self-select into experimental treatments because they are more motivated to use those treatments than learners who do not self-select (Heinsman and Shadish 1996). Including studies comparing groups with pre-training differences in meta-analysis produces a biased meta-analytic estimate (Aguinis et al. 2010). To address this problem, we examine quasi-experimentation as a moderator, per the recommendations of Schmidt and Hunter (2015). If quasi-experiments of learner control produced substantially greater effect sizes than experiments, quasi-experiments would be excluded from future analyses. We thus compare groups determined experimentally, quasi-experimentally via self-selection into condition, and quasi-experimentally by some other means.

Research Question 2

What is the impact of quasi-experimental design and self-selection by condition on the apparent effectiveness of objective learner control?

Method

Literature Search

To create the meta-analytic database for use in this study, we first searched for the keyword “learner control” in four scholarly databases: ABI Inform, ERIC, PsycINFO, and The Social Sciences Index. The initial article search, conducted in early 2012, was limited to dates after 1989 to limit the initial inclusion of articles examining learner control but not in the context of WBI. This search resulted in 87 potential articles. An unpublished literature search was conducted, including review of Society for Industrial and Organizational Psychology conference programs, emailing a variety of scholars who have published on the topic of learner control, and posting to several academic listservs, including those of the Academy of Management (Human Resources, Organizational Communication and Information Systems, and Organizational Behavior) and the American Educational Research Association (Division B, Curriculum Studies; Division C, Learning and Instruction; Division D, Measurement and Research Methodology; Division H, Research, Evaluation, and Assessment in Schools; and Division I, Education in the Professions). This resulted in the identification of 5 additional papers. A snowballing technique was employed to identify additional published and unpublished sources using the review articles already identified; reference lists of meta-analyses of online training were examined for additional potential inclusions, which resulted in identification of 24 additional papers. Next, all twenty-eight papers cited for inclusion in Kraiger and Jerden’s (2007) original meta-analysis were collected. Thus, the initial article collection effort produced 144 papers. After this process, at the suggestion of our associate editor, all papers reviewed by Carolan et al. (2014) not previously identified were also collected.

Inclusionary Criteria

Inclusionary criteria were set as follows: (1) studies must contain an adult sample and (2) studies must conduct a between-subjects comparison such that an effect size can be calculated comparing the presence of learner control in WBI to the absence of learner control in WBI. WBI was interpreted to include any instructional content that could likely be viewed in a web browser even if no web browser was necessarily used; for example, Doolittle (2010) reported a study of Adobe Flash-based training, and several other authors explored learner control of video-based instruction. Review of the initial 144 papers produced a meta-analytic database containing 48 independent sources of data, each providing one or more effect size estimates. Within each study, independently collected samples were first split into independent meta-analytic cases. Second, if effect sizes were reported separately by outcome type (e.g., if both a knowledge and skill measure were reported), these effects were considered independent cases (Bijmolt and Pieters 2001; Tracz et al. 1992). Third, within outcome type (e.g., if multiple knowledge measures were reported), mean effect size estimates were calculated. When both post-training and retention tests were presented, only post-training results were included. The initial meta-analytic database included 50 independent samples, including approximately 30 % from unpublished sources. The addition of articles meta-analyzed by Carolan and colleagues that met our inclusionary criteria expanded the database by 12, resulting in a final database including 62 independent samples from 59 sources and 97 total target effects.

Coding Procedures

Either the first and second author of this article or the second author and a graduate research assistant independently coded all cases in the final meta-analytic database using a frame-of-reference training approach to ensure high agreement. After coding ten sources each, they met to discuss discrepancies between their codes on those ten sources, after which they independently coded the remaining articles. As estimates of interrater reliability, simple percentage agreement was calculated for each study variable, and these values ranged from 85 % to 100 % (mean = 94 %). Afterward, the authors met to discuss the source of all disagreements and resolved them through article review and discussion.

Learner Control Dimensions

Skip control was present when learners had a default navigational path through training from which they could remove or skip content if desired that they would have otherwise encountered. Supplement control was present when learners had a default navigational path through training to which they could add additional content if desired that they would otherwise not encounter. Sequence control was present when learners had a default navigational path through training which they could alter at will. Sequence within modules and sequence across entire training programs were not distinguished. Pace control was present when learners had discretion as to how long they lingered on individual components of training (e.g., individual webpages within a module). Practice control was present when learners could choose to complete or skip knowledge or skill assessments/exercises. Guidance control was present when such assessments were available (with or without practice control) but learners were not required to act upon their results. Style control was present when learners were able to influence any aesthetic characteristics of their learning environment. Time control was present when learners did not have specific time restrictions on completion of their training program. Location control was present when learners did not have a specific location to complete training. We also generally assumed that if a description of an experimental manipulation did not mention a particular type of control, that type of control was not present. Each dimension was coded as present (1) or not present (0).

Study Date

Study date was determined from the publication date of the article or book chapter, or by the defense date if a dissertation. Although this is an imperfect proxy for study date, only four studies reported the actual year in which data collection took place, making this the best indicator available.

Learner Type

Learner type was identified first by determining if the sample consisted of employees or students. Student samples were further subdivided into two categories: (1) participation for course purposes, where assessment scores were tied directly to course grades or (2) participation for lab purposes (which was typically for extra credit in research participation pools). Two variables were thus created. The first enabled comparisons of non-student samples (1) with student samples (0). The second enabled comparisons of students participating for their course grade (1) with students participating in a lab setting (0).

Experimental Design

Experimental design was determined based upon author descriptions of random assignment. Studies incorporating random assignment of individuals to conditions were considered experiments, whereas studies without such random assignment were considered quasi-experiments. Design was coded as either quasi-experimental (1) or experimental (0).

Outcome Type

To classify outcome types, any post-training surveys intended to capture subjective evaluations of the training itself were first identified as reactions measures (Kirkpatrick 1996). Next, Campbell and Kuncel’s (2001) taxonomy of training outcomes was adopted to distinguish knowledge from skills. In their taxonomy, knowledge is defined as knowledge of labels, facts, rules, procedures, plans, or goals. Skills are defined as either observable skill, where knowledge is applied to solve a specific structured problem; and problem-solving skills, where knowledge and observable skills are applied to solve an ambiguous problem. Both observable skills and problem-solving skills were considered “skills” for the purposes of this meta-analysis. Knowledge assessments generally took the form of multiple-choice tests whereas skill assessment typically required demonstration of the learned skill.Footnote 1 Outcomes were thus was coded as reaction, skill or knowledge. Because a single study might provide both a knowledge and outcome measure, three variables were created to enable inclusion of outcome type in Table 2. First, reaction measures were coded as either present (1) or not present (0). Second, a variable indicating the inclusion of a knowledge measure, a skill measure, or both was dummy coded such that the use of a skill measure alone was coded as either present (1) or not present (0), and the use of both a knowledge measure and a skill measure in the same study was coded as present (1) or not present (0). Thus, studies including only knowledge measures were coded as “not present” on both variables.

Table 2 Sample size and moderator presence correlation matrix and means
Full size table

Effect Size

Standardized differences scores (d) were determined for each independent sample. These were calculated based upon means and standard deviations, when available. If means and standard deviations were unavailable, any appropriate reported d-statistic was used. In cases where neither means and standard deviations nor d-statistics were available, t–statistics or other information was converted to a d-statistic using the formulas provided by Schmidt and Hunter (2015). If an effect size was not able to be calculated based upon information present in the paper, the author was contacted to request group means and standard deviations. Effect sizes were coded in such a way that positive effect sizes indicated a gain in scores associated with the use of learner control, whereas negative effect sizes indicated a decrease in scores associated with the use of learner control. Where reaction measures were assessed, the same pattern was followed. To ensure no violations of the assumption of independence, a mean d-statistic was calculated in situations where multiple d-statistics could have been calculated within the same outcome type. For example, if two d-statistics were provided on two different knowledge measures with the same sample, the mean of these two d–statistics was entered into our meta-analytic database. Finally, one learner control effect on a skill outcome was found to be so substantially outlying that it was highly influential in all analyses where it was included and was therefore removed from the database (i.e., Simon and Werner 1996; d = 11.33).

Outcome Reliability

Reliabilities were collected from each study for the outcome measure reported. These were, almost universally, internal consistency reliability estimates. These internal consistent reliabilities were used as the basis for an artifact distribution correction to be described in the next section.

Meta-Analytic Procedures

To produce meta-analytic estimates, sample-size-weighted psychometric random-effects meta-analysis was conducted (Schmidt and Hunter 2015). Sample-weighted mean d-statistics were calculated, after which corrections for attenuation due to sampling error and unreliability were applied in order to produce a true-score estimate of the effect (δ) and its standard deviation (SDδ). Because many studies did not report the reliability of the learning outcome measure or of any reactions measures, an artifact distribution was used for these corrections. 27 % of studies reported the reliability of their learning outcome measures, and 32 % of studies reported the reliability of their reactions measure. Because the use of an artifact distribution assumes the reliabilities of measures from studies which did not report reliability to be drawn from the same population of reliabilities, the magnitude of corrections should be thus interpreted with some caution. Following the recommendations of Schmidt and Hunter (2015), examination of hypotheses was conducted by examining effect size magnitude, credibility interval width, and percentage of variance explained by study artifacts.

Results

Before analyzing study hypotheses and research questions, we first sought to understand the particular combinations of learner control dimensions appearing in our database. As noted earlier, every study we identified provided some degree of pace control in the learner control condition and only a few restricted pace control in the control condition. The distribution of dimensions manipulated simultaneously appears in Fig. 2. This pattern confirmed our suspicion that learner control is often treated as a unidimensional construct by researchers; 97 % of studies implemented at least two types of control simultaneously within a single experimental or quasi-experimental manipulation. To better understand which dimensions are commonly confounded with others, we also created the stacked bar graph shown in Fig. 3. No study examined any type of control other than pace control without confounding it with at least one other type of control. The most common combination of learner control features within a single study was a simultaneous implementation of pace, skip, and sequence control, and 22 % of our database contained studies incorporating this combination.

Fig. 2
figure 2

Number of dimensions of learner control manipulated simultaneously

Full size image
Fig. 3
figure 3

Stacked bar graph demonstrating the number of studies confounding additional dimensions within the manipulation of each type

Full size image

A correlation matrix and means of all variables at the data source level appears in Table 2. Analysis of the means revealed that pace control is the most commonly implemented type of control (84 % of studies). Sequence control, supplement control, and skip control are the next most commonly implemented types (67, 37, and 33 % of studies, respectively). All other types of control are used in fewer than 20 % of studies. Location control is notably uncommon (10 %) considering its common use in organizational settings. Sample sizes also tend to be small in this domain (mean N = 94.42), although more recent studies tend to have larger samples (r = .32). Analysis of the matrix indicated few, but notable, correlations among learner control dimension implementation. Skip and supplement control, which were combined as content control in Kraiger and Jerden’s (2007) framework, are only correlated .16, providing discriminant validity evidence. Skip control varied with both time control (r = −.26) and sequence control (r = .36), indicating that studies enabling learners to skip content tend to also allow control over sequence and restrict scheduling of time. The largest correlation in the matrix is that between use of location control and student use of learner control in their enrolled courses (r = .52), which primarily reflects the freedom college students have in choosing where they complete online courses.

We next sought to understand overall effects by outcome type. The effect of learner control on reactions was minimal (see Table 3; δ = 0.01, k = 19; failing to support H1), the effect on knowledge was minimal (see Table 4; δ = − 0.03, k = 60), and the effect was skill was moderate and positive (see Table 5; δ = 0.18, k = 18). The credibility interval surrounding the estimate for skill outcomes was mostly above zero, suggesting that most true score effects on skill are positive. However, estimates of the percentage of sampling error explained by known study artifacts (33, 33, and 47 %, respectively) suggests that moderators play an important role in all three outcomes.

Table 3 Meta-analyses of effects of objective instructional learner control dimensions on reaction outcomes
Full size table
Table 4 Meta-analyses of effects of objective instructional learner control dimensions on knowledge outcomes
Full size table
Table 5 Meta-analyses of effects of objective instructional learner control dimensions on skill outcomes
Full size table

Because the use of dimensions was correlated, individual effect sizes may not be directly interpretable when considering moderator effects by type of control due to multicollinearity effects. For example, the observed effect of studies utilizing learner skip control also contains the effects of time control and sequence control, with which it is correlated (see Table 2). Thus, in order to better isolate the effect of each individual control type, we focused upon comparisons of all studies that did incorporate a type of control with all those that did not and used this comparison to evaluate study hypotheses. For example, to evaluate the effectiveness of skip control, we contrasted the δ for all studies that did not incorporate skip control with the δ for all studies that did. Ideally, structural equation modeling would be used to address this problem more directly, but the database was far too small to meet minimal standards enabling this type of analysis (generally at least 100; see Wang and Wang 2012).

To determine the standard by which we would categorize the practical significance of these effects, we consulted a correlational effect size benchmarking study conducted by Bosco et al. (2015) examining effects reported in Journal of Applied Psychology and Personnel Psychology between 1980 and 2010. In this study, Bosco and colleagues examined correlations between work-related behaviors and attitudes/evaluations, such as organizational policies, perceptions of supervisors, and compensation. In examining relationships with training performance, they found the 33rd percentile of such correlations to be .08, the 50th percentile to be .14, and the 67th percentile to be .22. By converting these values to their equivalent d-statistics (Rosenthal, 1994), we created our standards. Differences within pairs larger than 0.45 (67th percentile) are interpreted as large effects, larger than 0.28 (median) as moderate effects, larger than 0.16 (33rd percentile) as small effects, and less than 0.16 as minimal effects.

Provision of skip control had a small negative effect on reactions, a moderate positive effect on knowledge outcomes, and a moderate positive effect on skill outcomes, supporting H2. Supplement control had a small negative effect on reactions, a minimal effect on knowledge outcomes, and a minimal effect on skill outcomes, failing to support H3. Sequence control had a small positive effect on reactions, a minimal effect on knowledge outcomes, and a minimal effect on skill outcomes, failing to support H4.

Practice control had a large positive effect on reactions (although k = 2), a small positive effect on knowledge outcomes, and a small positive effect on skill outcomes (although k = 1), failing to support H5. During our coding effort, we noticed that practice control tended to be implemented in two different ways in comparison to its control group; the control group either experienced practice or it did not. Thus, we post hoc hypothesized that the option to remove practice in comparison to the control group would result in negative outcomes whereas the option to add practice in comparison to the control group would result in positive outcomes, and the k observed for knowledge outcomes enabled us to test this empirically. However, as seen in Table 4, the option to remove practice actually resulted in a greater positive impact than the option to add practice, contrary to our expectations and providing further evidence against H5.

Guidance control had a small positive effect on reactions, a minimal effect on knowledge outcomes, and a small negative effect on skill outcomes, initially failing to support H6. However, in our coding, we noticed two problems. First, guidance control was sometimes operationalized with a confounding methodology in its control group. Specifically, guidance control was implemented simultaneously with feedback such that the control group experienced neither feedback nor guidance control. Isolating these studies revealed a strong positive effect (δ = .51; k = 2), as would be expected from a feedback intervention (Kluger and DeNisi 1996). Second, the provision of feedback was sometimes confounded with the control group. Specifically, two studies asked learners if they would like to receive feedback before providing that feedback. Thus, learners who consistently said “no” to viewing feedback would never have witnessed the guidance prompts. Isolating these studies revealed a strong negative effect (δ = −.48; k = 2), potentially a replication of Carolan and colleague’s finding of a negative effect for “feedback/practice control,” but with a greater magnitude. By removing these four studies and two more for which we could not determine confoundedness, we have therefore isolated a better estimate of the true effect of guidance control (δ = .17; k = 9), revealing a small effect for knowledge. We had insufficient sample size to repeat this approach for reactions and skill. Thus we ultimately present mixed support for H6.

Because publication date was a continuous moderator, weighted least-squares regression was chosen to test it based on current best practices (Schmidt and Hunter 2015; Steel and Kammeyer-Mueller 2002). To conduct these tests, formulas presented by Lipsey and Wilson (2001) were used to calculate inverse sample-error weights of each standardized effect size, and bivariate correlations were calculated between the moderator variables and weighted effect sizes per the recommendation of Steel and Kammeyer-Mueller. In the test of H7, which examined publication date, the resulting correlations were large for both knowledge (r = .32, p = .00) and skill outcomes (r = .48, p = .00). Thus, H7 was supported at a similar effect magnitude to that observed by Kraiger and Jerden (2007). To determine if the effect increases over time, we also conducted hierarchical multiple regressions including quadratic and cubic terms of publication date in the second step; however, the incremental R 2 was small and not statistically significant for those terms in both sets of outcome analyses.

To examine RQ1, which compares non-students with students in class and students in lab settings, meta-analyses of all three groups were conducted. These results appear in Table 6. The majority of the research identified, 66 samples, were conducted using students in lab settings. The effects of learner control for students of both types were minimal, although the credibility interval indicates a great deal of true score variation within the lab setting group. The effect for non-students was small and negative. We also conducted meta-analyses splitting investigation of RQ1 by outcome type where sufficient k was available, but the results closely paralleled the overall analyses presented here.

Table 6 Additional moderator analyses of the effect of learner control on combined knowledge and skill outcomes
Full size table

To examine RQ2, which investigated the effect of quasi-experimentation, separate meta-analyses of experiments and quasi-experiments were conducted. These results appear in Table 6. Neither produced more than a minimal effect (δ = 0.02 vs. δ = −0.01, respectively). We also further broke down the meta-analysis of quasi-experimentation to examine the role of self-selection into condition; again, effects were minimal (δ = −0.01 and δ = 0.00). Without evidence of differential effect by sampling technique, we determined it unnecessary to conduct additional analyses of prior hypotheses and RQs without quasi-experiments included. As was done in analysis of RQ1, these analyses were repeatedly split by outcome type, but no interpretable deviations from the presented pattern were found.

Discussion

This study primarily contributes to the literature on learner control by developing and testing an expanded framework of objective learner control integrating and expanding upon the works of Kraiger and Jerden (2007), Karim and Behrend (2014), and Carolan et al. (2014). It more clearly defines each dimension of learner control than in this previous literature. We also provide evidence that experimental and quasi-experimental learner control researchers almost universally (in 96 % of studies) confound multiple types of learner control and demonstrate the importance of this fact through our moderator tests showing that different types of learner control influence the success of a training program with varying effect sizes across outcomes types, sometimes with opposing signs. In particular, skip and sequence control increase knowledge and skill gains but harm reactions. We thus encourage both researchers and practitioners to carefully consider which types of learner control are being implemented in their training programs. A single continuum of “low” and “high” learner control, although common in the literature, confounds a variety of more complex psychological effects and should be avoided in future research. We hope these results guide both researchers working in this area and practitioners seeking to implement learner control in their training programs.

Misspecification of Learner Control and Resulting Confounds

A critical finding from our survey of the literature is that learner control is rarely operationalized in a way that permits clear conclusions about its effect across contexts. Four common misspecifications were identified. First, although pace control was commonly reported as a critical type of control, it was so commonly mis-specified that it was not possible to isolate its effects meta-analytically. In all true WBI studies we identified, pace control was present in both the control and experimental conditions. Studies that did isolate learner control of pace to the learner control condition were generally not strictly WBI. For example, in one of the cleanest examinations of any learner control dimension in our database, Mayer and Chandler (2001) asked learners to watch an instructional animation but randomly assigned them to either advance the animation frame by frame (high pace control) or to watch the entire animation at once (low pace control). Although a self-initiated animation and an automatically advancing animation could feasibly be run within WBI and thus were included in the present meta-analyses, no study actually manipulated learner control this way within an authentic WBI environment. We expect purposeful restriction of pace control to be uncommon in the modern practice of WBI, but this does leave a gap in the literature.

Second, in all studies providing learner control of scheduling (i.e., time and location), this type of control was also provided in control group comparisons even when researchers labeled this “no control”. Thus, from a practical perspective, our results must be generally interpreted as holding time and location control constant. For example, if corporate WBI was previously available only by sitting at a computer in the human resources department, these meta-analyses cannot help predict the impact of allowing employees to complete that WBI from anywhere. This is also a major gap in the literature (Karim and Behrend 2014).

Third, in studies examining learner control of practice, the control group varied systematically in terms of the comparison. Specifically, in half of the studies we identified, learners with control had the ability to add opportunities for practice in comparison to the control group, whereas in the other half, learners with control had the option to remove such opportunities. The effect is sizable, although the number of studies is small (both k = 4) and in the opposite direction from hypothesized; specifically, the removal of practice opportunities was associated with greater knowledge learning. This is potentially due to the effects of learner focus. If learners are able to self-assess accurately, they can skip practice they do not need, focusing instead on areas of weakness, which is important if there is a limited total amount of time available to interact with learning material. Unfortunately, very little research is available to suggest how learners actually make decisions when presented the opportunity to utilize control, so this remains an area for future research.

Fourth, in two studies examining learner control of guidance, the use of feedback in the control group varied, confounding the presence of guidance control with the presence of feedback. When isolating these two studies, the effect was a dramatic but spurious increase in the apparent effect of control. As it is a common theme in learner control research, better care must be taken to isolate the effects of particular targeted types of control. No feature of WBI should vary across experimental conditions, especially in terms of specific pedagogical elements or learning content. Only the specific, targeted aspect of control itself should be manipulated. In addition, in two studies, guidance control was operationalized not only as we defined it but also in terms of its general presence. For example, in their learner control experimental condition, Pridemore et al. (1993) asked learners after every question: “Would you like to check your answer?” Only if the learner responded “yes” would guidance be then provided, at which point, the learner could choose what to do. In the two studies allowing this, the effect of guidance was negative. When both the confound described earlier and this ability to choose exposure to guidance were removed, the effect of guidance control became positive, which we contend is the most accurate estimate of this dimension’s effect. Providing learners the ability to choose their response to feedback is beneficial to knowledge gains. This again highlights the impact of study-level confounds when interpreting the effects of control.

These challenges provide context to a larger and more troubling problem in the learner control literature. The goal of learner control research should be to identify the effect of control alone. Often, control is confounded with additional design decisions that make identification of the true driver of learner or reaction differences impossible in a primary study. Future research in the domain of learner control must be much more cautious in isolating the effect of control (i.e., a learner’s ability to change aspects of their learning environment) from other confounding factors or Reeve’s (1933) condemnation of learner control research as “pseudoscience” will remain a valid criticism.

Theoretical Structure of Learner Control Constructs

Prior researchers have varied in their conceptualization of the structure of learner control; Kraiger and Jerden (2007) presented dimensions as independent elements of a framework, whereas Karim and Behrend (2014) presented them as a hierarchical, multidimensional construct. We promote a third view based upon three pieces of evidence. First, Cavanaugh and Landers (2014) found that learners do not necessarily utilize dimensions of learner control when they are provided and vary widely in what they do utilize. Second, Kraiger and Jerden argued that multiple learners do not necessarily perceive the same level of control even when what they are provided is objectively the same. Third, the results found in Table 2 demonstrates that researchers of learner control do not commonly implement any particular combination of types of control, although pace control does appear to be the most common component. We contend that these three results together support the distinctiveness of three major concepts: (1) objective control, which refers to the types of control objectively implemented by the designer, (2) subjective/perceived control, which refers to the types of control perceived by the learner, and (3) control behaviors, which refer to the types of control actually utilized by the learner.

In addition, although learner control is generally conceptualized as a continuum, there were an insufficient number of studies in the research literature to examine it this way. Instead, learner control was examined as present or absent. We emphasize that this was done as a coding and interpretational convenience and does not reflect any underlying dichotomous nature to these dimensions. With the mean effects by type established by these meta-analyses, researchers can better interpret more subtle variations in effect caused by specific implementations in relation to those mean effects.

Outcome Differences

One of the major findings of these meta-analyses was the sometimes dramatic difference in the effect of learner control between knowledge and skill outcomes. In general, it appears that learner control is more effective for skill outcomes than for knowledge outcomes. We suspect this to have occurred because the WBI techniques generally used in skills training typically involve more active learning than do techniques used in knowledge training. Feedback in knowledge training is usually quite general; when a learner receives a poor score on a multiple-choice assessment, the cause of this poor performance is not always clear. Ineffective learning strategies, poor-quality instruction, insufficient time on task, poor assessment design, or any of a host of other causes may have led to a particular score. This places a cognitive burden on the learner to identify the cause, which many learners find difficult to navigate successfully (Morrison and Anglin 2005; van Merrienboer et al. 2003). Because modern learner-controlled WBI of skills often permits learners to view demonstration videos of the skill they are learning, the learner may be encouraged to pause the video, recreate the steps on their own time, and play back the video as many times as necessary until the skill can be performed correctly and consistently. In contrast, an employee learning new employee guidelines can only retake quizzes or assessments to determine whether or not answers are correct, a much less active process. If true, this also suggests that learner control will be more impactful when active learning techniques are utilized, a question that cannot be tested given the descriptions of learning activities available in the current literature. Future research should explore this possibility, and researchers should be careful to fully describe all learning activities in any given training program. The greater variance in knowledge outcomes than skill outcomes supports this view; this could be the result of an unmeasured active learning moderator.

Limitations

We identified four primary limitations to interpretation of this meta-analysis. First, we focused upon WBI and training that could feasibly be delivered via WBI in this study primarily for practical reasons, but this may artificially fragment the literature. Specifically, little learner control research has been published outside the context of WBI since Kraiger and Jerden’s (2007) analysis of the literature, so we decided to exclude research on older technologies in order to maximize our chances of providing immediately actionable and more practically relevant advice for practitioners. Despite this, the framework developed here may generalize well to these alternative formats. Perhaps more critically, this framework may apply equally well to emerging training technologies, such as mobile app-based learning (Ally 2009; Motiwalla 2007). As research on learner control continues, this framework should be considered carefully regardless of the specific technologies employed.

Second, as noted earlier, the success of instructional control in influencing learning outcomes is contingent upon trainee ability to accurately self-assess, trainee motivation to utilize optional training features, and trainee awareness of those features. To the extent that any of these are low in a given study, observed effects will be attenuated from true effects. Trainee self-assessment ability is often considered a significant problem for those intending to utilize learner control (DeRouin et al. 2005), and there is evidence to suggest that learners often make inaccurate judgments of their own abilities and needs (Brown and Ford 2002; Carrier 1984). For example, a trainee who is overconfident in his knowledge of training content may utilize skip control inappropriately and harmfully. Similarly, a trainee who is unmotivated may simply ignore potentially beneficial features in an effort to reach the end of training as easily or as quickly as possible. This is of particular concern given the prevalence of lab studies in the database, because students completing training courses for extra credit in their courses may be less motivated than actual trainees. Trainee awareness of learner control features is also of concern, given that in DeRouin et al. (2004) review of the literature, only one research study explicitly described the extent to which learners were trained on learner control features. If learners are unaware of the control they have, they cannot utilize it (Kraiger and Jerden 2007). To the extent that studies are, on average, biased in any or all of these ways, the meta-analytic estimates we provide here are biased downward.

Third, one of Reeve’s (1993) prime criticisms of learner control research was the lack of time most trainees spent in training. Among those that reported time spent in training in this study, 2.80 h were spent on average, with 75 % of studies utilizing one training session or less. Cronbach and Snow (1977) recommended at least 10 sessions to stabilize learning over time and measure an intervention’s effect. Such studies are generally unavailable in the current learner control research literature. Merill (1975) also suggested that the real value of learner control was its ability to teach people how to learn, suggesting that learning would only be harmed in the short term and improve as learners with control gained the skills necessary to use learner control effectively. Investigation of such a claim would require a longitudinal approach, which is also not found in the current literature. Future research should focus on longer-term training interventions and longitudinal studies of learner control use.

Fourth, it is critical to remember that this is a meta-analysis of an instructional method (i.e., a content-delivering technique or technology, such as a lecture, simulation, serious game, or PowerPoint presentation), and not an individual difference related to instructional effectiveness (i.e., a cause of improved learning, such as meta-cognitive strategies or self-regulation). As such, this meta-analysis is only able to examine the effects of learner control as it is presently studied, and not as it is actually used in organizations or as it may potentially be used as learner control technology continues to improve. This is, in a sense, a snapshot of the current value of learner control, which is subject to change. For example, as researchers and practitioners improve the method by which sequence control is provided and communicated to learners, its mean effect may increase. Although this is a limitation in long-term interpretation, we believe its current value in providing direction to training designers and researchers investigating these concepts currently to be worthwhile. Specifically, we are able to provide a roadmap with Fig. 3 demonstrating which learner control features are most atypically studied in isolation. Skip control, in particular, is universally confounded with at least two other types of control.

Additional Priorities for Future Learner Control Research

In addition to the priorities described above, we identified two other issues of interest raised by this study. First, considering improved learner motivation is one of the key supposed advantages of learner-controlled training, reaction measures were surprisingly underreported. Although we conducted subset analyses on reactions measures by type of learner control, the number of available studies often limits the interpretability of these dimensions. Given the available results, it appears that to maximize reactions, learners should be given control of sequence and guidance. In contrast, to maximize knowledge and skill gain, learners should be given skip control, although this is likely to result in poorer reactions. This pattern of research efforts also highlights a potentially even more important set of research gaps in that other outcomes of interest, such as mid-training motivational processes, post-training affective learning outcomes such as self-efficacy, and changes in mental models due to learner control, are almost entirely missing from the existing literature. Explicit study of such process variables may help address some of the apparent contradictions revealed here.

Second, more complex relationships, such as interactions between various types of objective learner control or between learner characteristics and objective learner control, could not be tested. When initially designing this meta-analysis, we considered if scheduling control might interact with other forms of control. Specifically, if trainees are highly constrained in when and where they may complete training, we suspect they will be less likely to utilize other learner control dimensions. Considering most online training for managers asks them to complete training on their own time (Behrend and Thompson 2012), the lack of studies clearly examining time control may severely limit the generalizability of results from the learner control literature to the workplace. Personality also appears to play an important role in determining who takes advantage of learner control features (Orvis et al. 2011; Schmidt and Ford 2003), but no personality traits were consistently reported in the learner control research literature. Any range restriction in personality scores in the studies included here will attenuate the observed effects. Some researchers have even suggested a curvilinear relationship between control features and learning such that the additional of control only helps to a certain degree, at which point learning may be harmed (Behrend and Thompson 2012), but this is untestable given the current literature.

In addition, as we conducted this meta-analysis, we were surprised at how infrequently actual learner behaviors were measured in relation to control, which further obfuscates any potential interactions. Several types of control appear reliant upon other types of control to produce large effects. For example, effective advisory control requires that the learner have the freedom to choose a course of action after receiving advice. The learner must logically have the ability to control the course in some specific way to act upon that advice. The precise approach could take several forms. For example, the learner might be recommended to go back to previously covered material (i.e., sequence control), look at an otherwise unseen remedial module (i.e., supplement control), or complete extra knowledge self-assessments (i.e., practice control). However, without any of these types of control, the provision of advisory control is likely to have a much reduced impact. The current literature provides very little guidance on how control types are likely to interact given the lack of attention paid to control behaviors.

Practical Implications and Conclusion

For training designers, the most important conclusion is that the overall effects of learner control are generally small and subtle. Implementation of learner control will not produce dramatically improved learning, and the use of the types of control most beneficial to learning may harm reactions. If assessments have been developed to be part of a training program, trainees should be required to complete them. Furthermore, there is very little literature exploring the impact of learner control on transfer, and the effects on transfer are likely to be smaller than the effects on learning. As a result of this, the current pervasiveness of learner control in training is not scientifically justified.

In fact, given the results presented here, we can currently only recommend the use of sequence control. We do not recommend this because it is the most impactful but instead because it is the most consistent; it is the only dimension that has either a positive or no effect on all outcomes. A similar pattern of results was reported regarding practice control, but we felt too few studies have been conducted to be confident in its impact on reactions or skill gain. Beyond those, all other types of control studied generally bring disadvantages to either reactions or learning, and we therefore recommend practitioners carefully consider the likely effects of the control they are currently implementing or plan to implement by consulting Tables 3, 4, and 5. Types of control likely to help one outcome are often likely to harm another, so the specific goals of implementing control should be identified clearly and types of control chosen to meet those goals. For example, for those implementing knowledge training concerned about both reactions to training and knowledge gains, the incorporations of sequence, practice, and guidance control are unlikely to be harmful. Although other types of control may ultimately be demonstrated to be helpful to learning designers under certain circumstances, there is currently insufficient evidence to suggest what circumstances these might be. Pace, scheduling, and style control have not even been studied extensively enough to draw any meta-analytic conclusions about their mean effects. Although intuitively appealing, learner control is not always in the best interests of either the learner or the organization, and much additional research in organizational training settings is needed to better understand the boundary conditions of success.