Keywords

1 Introduction

Challenge-based learning is a pedagogical concept that incorporates aspects of collaborative problem-based learning and contextual teaching and learning while focusing on current real-world problems. Problems vary in terms of their structure. Jonassen [1] classifies problems on a continuum from well-structured to ill-structured. Well-structured problems have a well-defined initial state, a known goal state or solution, and a constrained set of known procedures for solving a class of problems. In contrast, the solutions to ill-structured problems are neither predictable nor convergent because they often possess aspects that are unknown. Additionally, they possess multiple solutions or solution strategies or often no solutions at all [2]. Jonassen [3] reiterates that the structure of a problem often overlaps with complexity: Ill-structured problems tend to be more complex, especially those emerging from everyday practice, whereas most well-structured problems tend to be less complex. The complexity of a problem is determined by the number of functions or variables it involves; the degree of connectivity among these variables; the type of functional relationships between these properties; and the stability of the properties of the problem over time [4]. Simple problems are composed of few variables, while ill-structured problems may include many variables that may interact in unpredictable ways. When the conditions of a problem change, a person must continuously adapt his or her understanding of the problem while searching for new solutions, because the old solutions may no longer be viable. Static problems are those in which the factors are stable over time while ill-structured problems tend to be more dynamic [5]. Hence, in order to successfully solve complex and ill-structured problems, the person involved in problem-solving must be able to view and simulate the dynamic problem system in its entirety imagining the events that would take place if a particular action were to be performed [6]. It has been argued convincingly that games can serve as situated problem-solving environments, in which players are immersed in a culture and way of thinking [7, 8].

In this article, we describe the foundations of challenge-based learning and provide an overview of the Curtin Challenge digital learning (Challenge) platform. We then present an assessment and analytics framework linked with Challenge. A case study then demonstrates the analytics capabilities focussing on learning engagement before we conclude with implications and future work.

2 Challenge-Based Learning

The term challenge-based learning arose in the U.S. in the early 2000s with the support of innovative technology groups such as Apple Education, the New Media Consortium, The Society for Information Technology and Teacher Education, and the U.S. Department of Education Office of Educational Technology. Challenge-based learning builds on the practice of problem-based learning, but with an exclusive focus on real-world problems being creatively addressed by diverse collaborative teams. In addition, several key distinctions add relevancy and urgency for students, especially when combined with game-inspired methods such as badges, levels, points, transparent goals and clear progress-related feedback in self-paced learning [9,10,11,12].

The pedagogical approach of challenge-based learning adds game-based elements, which creates increased self-empowerment for individuals in teams by making explicit the learning process and higher order goals (not the solutions), providing assessable progress indicators of group process evolution and product quality based on the PL-C-PS framework (rather than focusing on product delivery timelines and expert-only scored quality feedback as in traditional assignments), and utilising exogenous rewards, awards and recognition that go beyond the current context [13].

For example, a team selected as one of the best in the world this year for a solution in water quality, might receive award certificates and recommendation letters that enhance their resumes, increase their opportunities for advanced studies and give the team members bragging rights for their successful collaborative efforts. Game-based additions to challenge-based learning might also include engaging, fun, light-heartedness and wit embedded into self-guided learning experiences [14]; so a challenge-based approach can include these aspects of game-based learning even though the purposes of the engagement are serious for both the learners and the real-world recipients of the team-based solutions and efforts.

Online global learning challenges engage students’ curiosity and desire to learn by making central the solving of open-ended problems as a member of a self-organising and self-directing international team [15]. In particular, when delivered as a mobile learning experience using an application platform developed at Curtin University in Western Australia, such challenges can integrate twenty-first century tools, require collaboration, and assist students in managing their time and work schedules, while effectively scaling to large numbers of students.

Research on challenge-based learning is beginning to show impacts such as increased engagement, increased time working on tasks, creative application of technology, and increased satisfaction with learning [16].

3 Challenge

The Challenge platform (http://challenge.curtin.edu.au) is specifically designed to engage learners in solving real-world problems in a social learning environment, with unobtrusive data collection enabling seamless demonstration and assessment of learning outcomes. The platform is being developed to support both individual and team-based learning in primarily open-ended ill-structured problem-solving and project-based learning contexts. Challenge can also support self-guided learning, automated feedback, branching storylines, self-organising teams, and distributed processes of mentoring, learning support and assessment.

A challenge is regarded as a collection of learning artefacts and corresponding learning tasks linked to specific learning outcomes or competences to be demonstrated. Figure 1 shows four of several challenges that have been utilised by over 25,000 students.

Fig. 1
figure 1

Curtin challenge platform provides a hub of possible learning opportunities

Full size image

From a design perspective, Career, Leadership and English Challenges have been planned for higher education students whereas Global Discovery focusses on a more general audience. Career Challenge includes 14 modules including Who am I?; How do I get to know my industry?; Decision-making strategies; Resumes; Cover letters; Selection criteria; Interviews; Drive your career; Workplace rights and responsibilities; etc. Average completion time is about 1 h per module. The design features of each module contain ‘activities’ including one to three different learner interactions or ‘tasks.’ For example, the module Who am I in the Career Challenge is a collection of five activities containing learning interactions, such as choosing from among options, writing a short response to a prompt, spinning a wheel to create random prompts, creating, organising, and listing ideas, or matching items. Figure 2 shows an example activity focussing on selection criteria. Learners interact by dragging specific selection criteria to different categories of selection criteria. Immediate feedback is provided through green lines as correct relation or red line as incorrect relation.

Fig. 2
figure 2

Task example in the Career Challenge

Full size image

Authoring content for the Challenge platform requires collaboration among discipline experts, digital instructional designers, and technologists. The authoring team needs skills in systems thinking, mental models, game-based learning and digital delivery technologies in addition to the pedagogical and content knowledge of instruction in a field of knowledge. Curtin University meets this challenge by forming flexible teams of people from learning and teaching as well as the faculties and larger community to undertake authoring and implementing digital learning on the platform.

The Challenge platform is now of sufficient maturity to extend its reach beyond current students. It is envisaged that new collaborations will be established with other educational institutions that will enable instructors and researchers to share the platform and learning pathways, with learners anywhere in the world; enable new challenge pathways to be developed by educators anywhere for use by learners everywhere; and drive high quality research to inform the future of learning.

4 Analytics in Challenge

Research on learning analytics has drawn a lot of attention over the past five years [17]. Learning analytics use static and dynamic information about learners and learning environments—assessing, eliciting, and analysing it—for real-time modelling, prediction, and support of learning processes as well as learning environments [18]. Only recently, serious games analytics has been introduced which focuses on improving game-play and game design as well as optimising learning processes and outcomes [19]. Serious games analytics converts learner-generated information into actionable insights for real-time processing [20]. Metrics for serious games analytics are similar to those of learning analytics and ideally include the learners’ individual characteristics (e.g., socio-demographic information, interests, prior knowledge, skills, and competencies) and learner-generated game data (e.g., time spent, obstacles managed, goals or tasks completed, navigation patterns, social interaction, etc.) [20,21,22].

The application of serious games analytics opens up opportunities for the assessment of engagement within game-based learning environments. The availability of real-time information about the learners’ actions and behaviours stemming from key decision points or game-specific events provide insights into the extent of the learners’ engagement during game-play. The analysis of single action or behaviour and the investigation of more complex series of actions and behaviours can elicit patterns of engagement, and therefore provide key insights into learning processes [13].

The data traces captured by the challenge platform are highly detailed, with many events per learning activity, which when combined with new input devices and approaches brings the potential for measuring indicators of physical, emotional and cognitive states of the learner. The data innovation of the platform is the ability to capture event-based records of the higher frequency and higher dimensional aspects of learning engagement, which is in turn useful for analysis of the effectiveness and impact on the physical, emotional and cognitive layers of learning caused or influenced by the engagements. This forms a high-resolution analytics base on which people can conduct research into digital learning and teaching as well as into how to achieve better outcomes in scalable digital learning experiences [23].

The process of turning session log files and process stream data into indicators has been recently summarised in Griffin and Care [24] which also notes several precursor research projects with results related to digital learning. Further, a process of exploratory data analysis is required based on post hoc analysis of real people using an appropriately designed digital space to learn. The growing field of learning analytics focused on learning and learners (as opposed to teaching, institutional progress, curriculum and other outcomes) is exploring and expanding the knowledge base concerning the challenges and solutions of the layered and complex analyses required nowadays for a better understanding of the impact of digitally enhanced learning spaces on how people learn—we refer to this as analytics for learning.

For the case study described next, a basic educational data mining approach has been utilised [25]. Raw data of the relevant Challenge and cohort were selected and pre-processed including cleaning and matching with external data sources (e.g., student background information). Next, data were transformed focussing on time-based events linked to specific learning activities and related performance. Simple natural language algorithms were applied to open-text responses (including word count, use of language). Standard regression analyses were applied to answer the research hypotheses.

5 Case Study on Learning Engagement

This case study sought to investigate the dynamics of learning engagement in a challenge-based digital learning environment using a data analytics approach. The context of the present study is set in the Curtin Challenge. A learner interacts with Challenge content by pointing, clicking, sliding items, vocalising, taking pictures and drawing as well as watching, listening, reading and writing as in typical digital learning environments.

Learning engagement is generally regarded as the time and effort an individual invests on a specific learning activity [26]. Further, learning engagement is a multidimensional concept and understood as the individual’s ability to behaviourally, cognitively, emotionally, and motivationally interact with learning artefacts in an on-going learning process [27]. A generally accepted assumption is that the more students engage with a subject matter or phenomenon in question, the more they tend to learn [28]. This assumption is consistent with the theory of self-regulated learning [29] and concepts of engagement [30]. Accordingly, learning engagement is positively linked to desirable learning outcomes or learning performance [31]. Several studies focussing on learning engagement support the assumption that higher engagement of a learner corresponds with higher learning outcomes [32]. However, most of these studies have been conducted in face-to-face learning environments. Accordingly, a confirmation of these findings in digital learning environments is still lacking.

In light of previous empirical findings on learning engagement [33,34,35,36,37], we expect that learning engagement is positively related to learning performance in a challenge-based digital learning environment. Attributes of learning engagement in such a learning environment are conceptualised through several actions: (a) launching a specific activity (task), (b) spending active time on the task, (c) entering a written response, and (d) finishing a task. The learning performance measured in this study is computed by the number of correct answers in a subset of tasks designed with embedded feedback to the student. The hypotheses of this study focus on the attributes of learning engagement and its relation to learning performance specifically in the Career Challenge. We assume that launching specific activities (tasks) is related to the learning performance in challenge-based digital learning environments (Hypothesis 1). Further, we assume that spending active time on tasks is related to learning performance (Hypotheses 2). Also, we expect that the length of written responses is related to the learning performance (Hypothesis 3). The final assumption focusses on the relationship between finishing tasks and learning performance (Hypothesis 4).

5.1 Case Method

The data set of the Career Challenge consists of 52,675,225 rows of raw data containing information of NC = 8951 students (3571 male; 5380 female) with an average age of M = 25.72 years (SD = 6.64). In a period of 24 months (January 2016–January 2018), students spent a total of 10,239 h interacting with the Career Challenge. The students in the sample stem from various backgrounds and study programmes as well as.

Raw data from the Career Challenge were cleaned and transformed into a transaction data set in which each row represents an event of one user. The dependent variable learning_performance (LP) was computed as the number of correct answers in an activity. The variables reflecting attributes of learning engagement were computed as follows: launching_task (LT) as the number of activities started by a student; time_on_task (TT) as the duration in seconds spent in an activity; written_response (WR) as the number of words submitted by a student; finishing_task (FT) as the number of activities finished by a student.

5.2 Case Findings

In order to test the above presented four hypotheses, regression analyses were computed to determine whether attributes of learning engagement (i.e., launching task, time on task, written response, finishing task) were significant predictors of learning performance in challenge-based digital learning environments.

Table 1 shows zero-order correlations of attributes of learning engagement and learning performance for the Career Challenge. All correlations were significant at p < 0.001. High positive correlations were found between launching task (LT; M = 6.73; SD = 8.95) and learning outcome (LP; M = 8.38; SD = 13.19), time on task (TT; M = 4118.09; SD = 6623.88), as well as written response (WR; M = 166.92; SD = 284.62). Moderate positive correlations were found for written response and learning outcome as well as time on task. Low positive correlations were found for the remaining variable combinations.

Table 1 Zero-order correlations, means and standard deviations of attributes of learning engagement and learning performance for the Career Challenge
Full size table

The linear regression analysis for the Career Challenge is presented in Table 2, yielding a ∆R2 of 0.713 (F(4, 8950) = 5568.79, p < 0.001). Clearly, the number of activities started by a student (LT; β = 0.80, p < 0.001) positively predicted the learning performance. In addition, the number of activities finished by a student (FT; β = 0.04, p < 0.001) and the number of words submitted by a student (WR; β = 0.13, p < 0.001) positively predicted the learning performance. In contrast, the duration students spent on a task (TT; β = −0.09, p < 0.001) was inversely related to learning performance.

Table 2 Regression analyses predicting learning performance by attributes of learning engagement for the Career Challenge
Full size table

In sum, the four hypotheses are accepted for the Career Challenge, confirming significant relationships between attributes of learning engagement and learning performance.

5.3 Case Discussion

The analytic results showed that learning engagement in challenge-based digital learning environments is significantly related to learning performance. These findings support previous studies conducted in face-to-face situations [34, 38, 39]. Significant attributes predicting the learning performance of the student appeared to be the number of activities started and the number of activities finished by a student. This is a reflection of active engagement with the learning environment [33]. At the same time, better learners seem to spend less time on a specific task in the Career Challenge. This may be interpreted as a reflection of existing prior knowledge or a progression towards an advanced learner [40]. Another significant indicator predicting learning performance in the Career Challenge was the number of words submitted in open-text activities. On a surface level, these findings are also related to studies conducted in writing research and clearly reflect the impact of the variation in learning engagement [36, 41].

Limitations of this case study include the restricted access of student data, for example, course load, past academic performance, or personal characteristics, for linking additional data to the reported engagement and performance measures. Combining such additional data in the future will provide a more detailed insight into the multidimensional concepts to be investigated. Second, the Career Challenge does not presently include an overall performance measure which has been validated against an outside criterion. Accordingly, a revision of the learning and assessment design should include additional or revised measures which follow accepted criteria or competence indicators. However, without the externally validated benchmarks, there is sufficient available data which can be used to improve the existing learning design through algorithms focussing on design features and navigation sequences of learners [42,43,44]. Third, as we included the analysis of open-text answers in our analysis model, this approach is limited by the overall potential of the simple approaches used in natural language processing (NLP). Further development of a future analysis will include a focus on deeper levels of syntactic complexity, lexical sophistication, and quality of writing as well as a deep semantic analysis compared to expert solutions [45, 46].

6 Conclusion

The Challenge platform is being developed to support both individual and team-based learning in primarily open-ended ill-structured problem-solving and project-based learning contexts [47]. The platform can also support self-guided learning, automated feedback, branching story lines, self-organising teams, and distributed processes of mentoring, learning support and assessment [48, 49].

The data traces captured by the Challenge platform are highly detailed, with many events per learning activity. The data and analytics innovation of the Challenge platform is the ability to capture event-based records of higher frequency with the potential to analyse higher dimensional aspects of learning engagement, which we believe may be in turn useful for analysis of the embedded learning design’s effectiveness and impact on the physical, emotional and cognitive layers of learning caused or influenced by digital engagements. The data from the challenge-based learning platform forms a high-resolution analytics base on which researchers can conduct studies into learning analytics design [44, 50]. In addition, research on how to achieve better outcomes in scalable digital learning experiences is expected to grow [23, 49].

There are multiple opportunities arising from analytics of digitally delivered challenge-based learning. Analyses of the learning performance transcript, even when automated and multileveled, is a mixture of conditional and inferential interpretation that can utilise several frames of reference while adding layers of interpreted evidence, insights concerning the complexity and additional dimensionality to our understanding of the performance and our ability to re-present the performance in the light of our understandings [48]. Practitioners, for example, learning designers, may use the detailed data traces to inform changes required in the design of individual activities or the flow of the story line [44]. Tutors may use the analytics data to monitor and adjust interactions with specific modules or tasks in real-time. For educational researchers, the detailed trace data can provide insights into navigation patterns of individual learners and linking them with individual characteristics or learning performance. Data scientists may use the same data to apply advance analytics algorithms using A/B testing or other analytics approaches.

Future research will focus on the analysis of several large extant data sets from the Challenge platform. Currently, the possibility of adaptive algorithms based on learning engagement and learning performance are being investigated. Such algorithms will enable meaningful microanalysis of individual performance as well as personalised and adaptive feedback to the learner whenever it is needed.