Keywords

1 Introduction

Self-regulated learning (SRL) refers to the process where learners actively and adaptively adjust their cognition, emotion, and behavior toward their goals [1]. SRL processes, in general, consist of three cyclical phases: (1) preparatory, where learners analyze the task, set goals, and make plans, (2) performance, where learners execute the plan, monitor and control the processes, and (3) appraisal, where learners evaluate their performance based on self- or external feedback and adapt their goal and plans [2]. SRL provides a valuable theoretical framework for understanding the interactions between cognition, metacognition, motivation, and emotion during learning [2].

Increasingly, researchers are viewing SRL as events that unfold over time rather than static skills [3,4,5]. Understanding and researching SRL from this conceptual perspective requires fine-granularity learning activity data which can be provided by computer-based learning environments (CBLE), due to their capability to record each of the learners’ actions unobtrusively. Approaches for micro-level analyses are necessary to obtain insights about SRL from such moment-to-moment data [4, 6, 7]. Researchers have explored SRL using multiple approaches, such as knowledge engineering [8], sequential pattern mining [9], lag-sequential analysis [10], statistical discourse analysis [11], and process mining [12, 13]. The current study employs an emerging method, epistemic network analysis (ENA) [14], for the in-depth analyses of SRL processes.

We apply ENA to trace data captured from Betty’s Brain, a CBLE designed to foster student self-regulation [15]. We aim to investigate the relationship between students’ self-regulated behavior in Betty’s Brain and their performance, and the affordances of ENA for the study of student behaviors in CBLEs. Results illustrated how students engage in three main categories of actions required to regulate their open-ended problem-solving [16]: information seeking, solution construction, and solution assessment. ENA highlighted differences in behaviors between low and high performers, allowing us to identify opportunities for further analyses and refinements of Betty’s Brain.

2 Related Work

2.1 Investigation of Self-regulated Learning Using Trace Data

Much of the work measuring and studying SRL relies upon self-reported, out-of-context questionnaires (e.g., [17]). Such methods assume that SRL involves static skills. It may capture the global level of self-regulation, but trace data, such as computer action logs and think-aloud data, may better reflect specific SRL strategies in context [18].

Various analytical approaches have been applied to the micro-level analysis of SRL using trace data. Knowledge engineering was used to identify the different help-seeking strategies students employ while using an intelligent tutor [8]. Sequential pattern mining was applied to evaluate the effectiveness of SRL scaffolding during the study of scientific phenomena [9]. Lag-sequential analyses were used to understand the differences in self-directed speech and self-regulated behaviors for children with language disorders [10]. Statistical discourse analysis was utilized to investigate how sequences of cognitive, metacognitive, and relational activities impact later cognition in a collaborative writing task [11]. These methods have shown potential in revealing the micro-processes of SRL. However, their results only highlight local behavior patterns.

In contrast, process mining depicts a holistic SRL process, where actions have directional connections [12]. This technique has been applied [13] to compare processes between groups who did/did not receive metacognitive prompts when studying topics in educational psychology. However, process mining does not allow a global statistical test for the difference between groups and different weighting for individuals [19]. These shortcomings may be overcome by ENA, which provides both networked visualizations of the data, facilitating qualitative interpretation, and statistical tests.

2.2 Using ENA to Study SRL with Trace Data

ENA has been used to investigate SRL behaviors using a range of data types including trace data [20, 21], qualitatively coded questionnaires [19] and interviews [22]. ENA was combined with process mining and clustering, to provide a rich interpretation of SRL [19,20,21]. We limit our discussion to studies that used trace or analogous data.

Gamage, et al. [22] used ENA to compare participation of two groups of MOOC takers: multiple MOOC completers, and first-time MOOC taker. They applied ENA to qualitative coding of log sheets and interviews to identify cognitive (e.g. watching a video) and social (e.g. use or discussion tools) tasks students performed. While they did not use trace data, their codes were analogous to trace data produced by MOOCs.

Trace data produced by LMSs has been used to conduct ENA analyses [20, 21]. ENA was used to compare the use of SRL related actions, e.g., goal setting, making plans, work on a task, evaluation, and reflection, between the top and bottom decile of students using assessment performance [20]. Another use of ENA was to compare student behaviors for different course taking strategies (e.g. reading the e-book, viewing learning resources, taking quizzes and assignments) [21]. In both studies, ENA was combined with process mining [20, 21] or agglomerative hierarchical clustering [21].

Fig. 1.
figure 1

The Betty’s Brain interface. Here the student is looking at a quiz’s results.

Full size image

3 Betty’s Brain

Betty’s Brain (Fig. 1) is an open-ended CBLE that provides students opportunities to develop and apply SRL processes as they build causal models of scientific phenomena [16]. Trace data in Betty’s Brain captures the performance and appraisal phases, i.e., how learners process the task, evaluate their performance, and adjust strategies [2, 23].

Students create a causal map of scientific concepts with associated links to model a scientific phenomenon (e.g., climate change), to teach a computer agent, generically called Betty. This causal map represents Betty’s Brain, i.e., what she has learned about the topic. To produce this map, students acquire knowledge of the subject by using hypertext resources, translate that knowledge into causal links between selected concepts (e.g., global temperature increases ocean temperature) and test their map by having Betty take quizzes (see Sect. 4.2 for more details about possible student actions).

4 Methods

4.1 Participants and Procedures

Data for this analysis were collected from 98 sixth graders in an urban public school in south-eastern United States. This school serves 700 students in grades 5–8 (40% underrepresented minorities, and 8% enrolled in the free and reduced lunch program). The study was conducted in classrooms in December 2018 and lasted seven school days (50 min per day). Students answered a pre-test (1), received Betty’s Brain training (2), used the system to study climate change (3–6), and answered a post-test (7).

The pre- (M = 6.26; SD = 2.66) and post-tests (M = 9.18; SD = 3.28) assessed students’ knowledge of climate change and causal relationships. They were identical in form and content and consisted of seven multiple choice items and three short answer items with a maximum score of 18. Learning gains (M = 0.25; SD = 0.24) were calculated as (post-test scores − pre-test scores)/(18 − pre-test scores) to account for differences in pre-test scores. The maximum score on the pre-test, 13, showed no ceiling effect. A median split was used to divide students into two groups based on their learning gains: low performers (<0.26) and high performers (>= 0.26).

4.2 Contextualizing Actions

In Betty’s Brain, students can perform a set of actions to build their causal map. They can read a hyperlink resource, work on their causal map by adding or deleting a concept, adding a causal link between two concepts, or editing an existing link. In addition, they can ask Betty to take a quiz (generated by the system) which Betty will answer using the students’ causal map. Students can then view the quiz’s results to make inference about the correctness of their causal map. As an alternative to having Betty take a quiz, students can use dropdown menus to ask Betty to explain the causal relationship between two concepts. While Betty’s answers to such questions are not graded, they allow students to better understand how concepts are related to each other in their map.

To study SRL behaviors, the different types of actions were further contextualized based on their duration and coherence. First, quiz result viewing and resource reading actions were contextualized as short vs. Long, where a short action is usually too short for the student to acquire the information presented by the resource. Quiz result viewing actions were labeled as long or short, based on whether their duration was higher than 2 s. Reading actions were labeled long or short, based on whether their duration was greater than 10 s. Both time thresholds were selected based on prior research [24]. In addition, long readings were labeled as old or new based on whether the page has been previously accessed; providing insights about whether the student was exposed to new information (new reading) or was revisiting information (old reading).

Long reading of old pages and links edits were labeled coherent vs. Incoherent, based using Coherence Analysis [24]. The students’ actions generate information that can be used to guide subsequent actions. For example, reading a hypertext resource provides information about the causal links between different concepts, and quiz results provides information about missing or incorrect links. If a student performs a sequence of actions where a first action informs a second one, the second action is considered “coherent”. For example, if, after reading a resource discussing two concepts, the student adds (or edits) a link connecting those concepts, the add (or edit) link action is considered coherent with the reading action. In contrast, if a student adds a link between two concepts without previously reading a resource related to those concepts, this addition is considered “incoherent”. Similarly, viewing a quiz result provides information that can lead to reading a related resource (coherent reading) or editing a related link (coherent addition, or coherent editing). Coherence analysis is operationalized based on information stored within the students’ log trace data and does not rely on human judgement.

It is important to note that a coherent action does not necessarily imply that the action is part of the correct solution, only that it is informed by a prior action. In addition, neither short reading or short viewing of quiz results counted towards considering a future action as coherent since short action are indicative of searching for relevant information rather than acquiring new information.

Each action type was classified as part of one of three categories of actions reflecting processes relevant to SRL (Table 1): information seeking, solution construction and solution assessment. Those categories were taken from a model aligning Betty’s Brain’s actions to a framework for problem-solving in open-ended learning environments [16]. Any action not related to one of the three categories was excluded from our analyses.

Information seeking includes all action types related to reading one of the hypermedia resource pages provided within Betty’s Brain. It includes new reading, short reading, and coherent and incoherent reading actions. Solution construction includes action types related to building or editing the causal map. It includes adding or deleting a concept, coherent or incoherent additions and revisions of a causal link. Solution assessment includes actions related to evaluating the correctness of the causal map. It includes asking Betty to take a quiz, long and short viewing of a quiz result, viewing an ungraded quiz question, and asking Betty a causal question.

Table 1. Selected action types, their classification as one of the three categories of open-ended problem-solving actions and frequency of occurrences across student groupings.
Full size table

4.3 Epistemic Network Analysis

We applied Epistemic Network Analysis (ENA) [14] to our data using the webtool (version 1.7.0) [25]. Each data point corresponded to one of the 98 participants in our study who were grouped into high or low performers, as discussed earlier.

Our ENA models used a moving window of three actions (each action plus the two previous actions) to generate the networks. This window size was selected to account for situations where students might perform an intermediate action between two connected actions. For example, a student might perform a coherent reading followed by an incoherent reading before adding a link related to the first reading. In such a case, there should be a link between the first coherent reading and the following coherent addition of a link, regardless of the superfluous incoherent reading. Initial exploration of the data investigated the use of a larger window size of four. This change did not affect general trends in the networks and the smaller window size was selected.

ENA was used to investigate the relationships between action types both within each of the three categories of open-ended problem-solving actions, across each pair of categories and over the three categories combined. In each case, we first generated a network including all 98 participants to describe the average co-occurrence of action types. Then analyses were conducted to investigate differences in behaviors between the low and high performing students. Results from these analyses were interpreted to provide insights about how behavioral patterns relate to performance within Betty’s Brain.

First, networks were generated for each of the categories of problem-solving actions: information seeking; solution construction; and solution assessment. Each network was analyzed to identify how different groups of students might approach each category. Second, networks combining two categories were generated to investigate differences when transitioning between categories of actions: information seeking and solution construction; solution construction and solution assessment; solution assessment and information seeking. Finally, networks including all three categories were generated to investigate patterns spanning the full problem-solving process.

For all analyses, a means rotation was applied to align the means of both student grouping (low and high performers) along the X axis. Statistical significance in network differences between the groups were computed using Mann-Whitney on the X axis.

5 Results

5.1 Information Seeking

Fig. 2.
figure 2

Information seeking networks for all participants (left) and performance groups (right).

Full size image

As the network including all participants illustrates (left of Fig. 2), short reading actions are central to information seeking behaviors. They often co-occur with coherent, incoherent, and new reading actions. This is unsurprising as short readings are both very frequent (21.34% of all actions) and indicative of a student searching through resources for the page(s) that may contain information they currently seek. The network also suggests it is somewhat rare for long reading actions (coherent, incoherent, and new reading) to co-occur with each other. Suggesting that actions of other types, whether short readings to search for information or actions related to other categories of actions (e.g. solution construction), act as intermediate actions between long readings.

Comparison of information seeking networks (right of Fig. 2) showed no statistically significant difference between low (Mdn = −0.04) and high (Mdn = 0.28) performers (U = 1340, p = 0.32, r = 0.12) despite short readings more often co-occurring with coherent readings for high performers and with incoherent readings for low performers.

5.2 Solution Construction

Fig. 3.
figure 3

Solution construction networks for all participants (left) and performance groups (right)

Full size image

Co-occurrences of solution construction actions are centered around adding new concepts to the map (left of Fig. 3). Adding a new concept is most strongly connected to deleting a concept and adding a new causal link (coherent and incoherent). The network shows few other strong connections, besides the connection between coherent revisions of a link and the coherent addition of a link. Most cases of co-occurring actions can be interpreted as one edit that requires multiple actions; e.g., adding a new link to the map might first require adding new concepts or deleting incorrect links (coherent revision).

A significant difference was observed in networks (right of Fig. 3) between low (Mdn = −0.19) and high (Mdn = 0.37) performers (U = 1750, p < 0.01, r = 0.46). Adding a new concept is central to building the causal map, however high performers showed stronger connections to coherent addition of new links, whereas low performers showed more behaviors combining adding and deleting concepts, indicating they might be unsure of how to build their map. This suggests that low performers have more difficulty in identifying the appropriate concepts to build their map, which may partially explain their lower performance: adding and deleting concepts rather than adding links and testing them. High performers show more behaviors combining coherent revisions (i.e., making corrections to their map) and additions of links (both coherent and incoherent – indicating that high performers also made mistakes when building their maps).

5.3 Solution Assessment

Fig. 4.
figure 4

Solution assessment networks for all participants (left) and comparing groups (right)

Full size image

Co-occurrences of solution assessment actions (left of Fig. 4) show a strong connection between having Betty take a quiz and long quiz views. Interestingly, it was rare for short quiz views to co-occur with quiz taking, suggesting that after having Betty take a quiz, students often spend a reasonable amount of time viewing the quiz’s results. Actions related to quickly browsing results (short viewing) are most strongly connected to long views of quiz results and might indicate students going back to quickly review a previously studied results or searching for some results that they wanted to study in more detail. No significant difference was found between networks (right of Fig. 4) for low (Mdn = −0.04) and high (Mdn = 0.19) performers (U = 1023, p = 0.21, r = 0.15).

5.4 Information Seeking and Solution Construction

Figure 5 (left) shows the co-occurrence of information seeking and solution construction behaviors for all students. While adding a concept to the causal map appears to be connected to all types of reading actions to some degree (with its strongest connection to short readings and its weakest to new readings), deleting a concept is weakly linked to information seeking (with its connection to short readings being the strongest).

For actions related to adding a causal link to the map, both coherent and incoherent addition are connected to short readings, but the connection is stronger for coherent addition. As expected, coherent addition is connected to coherent reading, but this connection is not as strong as the one to short reading. Incoherent addition is rarely connected to incoherent reading. This might be partially explained by the fact that a link added based on an incoherent reading would still be labeled as coherent since it is informed by the content of that incoherent reading. However, co-occurrence of incoherent reading and coherent addition also appear to be uncommon, therefore, it is generally less frequent for an incoherent reading action to co-occur with solution construction.

Fig. 5.
figure 5

Networks combining information seeking and solution construction for all participants (left) and performance groups (right)

Full size image

As expected, coherent and incoherent additions of a link are connected to adding concept actions, which are themselves strongly connected to information seeking (both coherent and incoherent). Adding a concept appears to be an intermediate step between reading and adding a link to the causal map. In some cases, this process might lead to adding an incoherent link, implying the link did not appear in the pages the student read. Revisions of causal links generally have weak connections to information seeking actions, with the strongest connection between coherent revision and short readings.

A significant difference was observed (right of Fig. 5) between low (Mdn = −0.32) and high (Mdn = 0.53) performers (U = 713, p < 0.01, r = 0.41). High performers showed stronger connections between coherent solution construction (addition and revision of links) and information seeking (short and coherent readings). No differences in incoherent solution construction actions (incoherent addition/ revision of a link) were observed. Behaviors related to adding and deleting concepts co-occurred more frequently with short readings for low performers. They also showed slightly stronger connections between deleting a concept and coherent reading. This suggests low performers did not understand what they were reading, and this lack of surety resulted in adding and deleting concepts more often instead of adding links after adding concepts.

5.5 Solution Construction and Solution Assessment

Co-occurrences of solution assessment and solution construction actions (left of Fig. 6) are centered around the actions of having Betty take a quiz and long quiz viewing. Both action types are well connected to most of the solution construction actions. This could be related to either students validating their map by having Betty take a quiz or students modifying their map based on their interpretation of the quiz results.

There was a significant difference (right of Fig. 6) between low (Mdn = −0.41) and high (Mdn = 0.52) performers (U = 614, p < 0.01, r = 0.49). High performers showed stronger connections between coherent additions and revisions of causal links, and the main solution assessment actions (having Betty taking a quiz and long view of quiz results). Network connections for incoherent addition or revision of a link appeared to be stronger for low performers. However, these differences were small. Low performers showed stronger connections between adding or deleting a concept and the main solution assessment actions. They showed stronger connections between asking Betty a causal question and all solution construction actions. However, none of those connections were strong. This might be partially explained by the low frequency of asking Betty a causal question throughout the problem-solving process (1.03% of actions).

Fig. 6.
figure 6

Networks combining solution construction and solution assessment for all participants (left) and performance groups (right)

Full size image

5.6 Solution Assessment and Information Seeking

Fig. 7.
figure 7

Networks combining solution assessment and information seeking for all participants (left) and performance groups (right)

Full size image

Co-occurrences of solution assessment and information seeking actions (left of Fig. 7) were mainly associated with three solution assessment action types (having Betty take a quiz, and short and long viewing of quiz results) and two information seeking actions (short and coherent readings). All three solution assessment action types show a similar pattern where the strongest connection is to short reading, followed by coherent reading. However, the strength of those connections varies across action types. The strongest connections to short or coherent reading are from long viewing of quiz results, followed by having Betty take a quiz, with the weakest for short viewing of quiz results.

Overall, those patterns are consistent with expected behaviors. Having Betty take a quiz is expected to lead to actions related to viewing its results, which then informs information seeking behaviors, especially if Betty answered a question incorrectly or could not answer a question that was asked of her. Information seeking may start with short readings as the students search for a resource aligning with the quiz results, which should then lead to a coherent reading action. Few connections were observed between solution assessment actions and incoherent reading. It may be that short readings (i.e., searching for relevant information) act as an intermediate action between viewing quiz results and incoherent readings (indicating the students end up on the wrong page because of their lack of understanding). Actions related to asking Betty to answer a causal question did not appear to frequently co-occur with information seeking actions.

A significant difference was observed between networks (right of Fig. 7) for low (Mdn = −0.03) and high (Mdn = 0.27) performers (U = 892, p = 0.03, r = 0.26). However, a visual inspection of the networks did not reveal any important differences in connections bridging solution assessment and information seeking. Rather, differences appear to be local with high performers showing strong connections within solution assessment and within information seeking for low performers.

5.7 Combined Analysis of Open-Ended Problem-Solving Processes

Fig. 8.
figure 8

Networks combining all actions for all participants (left) and performance groups (right)

Full size image

While the network (left of Fig. 8) provides a global view of the co-occurrences of action types it does not highlight any particular set of multiple strongly connected actions spanning the three categories of actions. The most strongly connected actions are short readings and long viewing of quiz results. This might be because they are the two most common action types, both are central to their own category (information seeking and solution assessment respectively) and are actions that can be repeated multiple times to achieve a goal – for example, browsing multiple resources (short reads) to find the appropriate one and viewing the results of many different questions (long view of quiz results) for the same quiz. Both short readings and long viewing of quiz results are also actions that we expect students would perform regardless of whether they make coherent use of the information they acquire. In contrast, there doesn’t appear to be one central solution construction action with most of them well connected in the network.

There was a significant difference (right of Fig. 8) between low (Mdn = −0.31) and high (Mdn = 0.44) performers (U = 1686, p < 0.01, r = 0.40). The difference mainly aligns with the expected strategic problem-solving behavior rather than unexpected behaviors. High performers showed stronger connections for many of the expected interactions across action categories. They showed stronger connections between short and coherent readings, which were in turn connected to coherent additions of links. While information seeking actions did not show stronger connection with adding a concept in the map for high performers, they showed a stronger connection between adding a new concept and coherent link addition. Coherent link addition was also more strongly connected to having Betty take a quiz and to long viewing of quiz results; themselves more strongly associated with coherent revision of a link and coherent readings.

Low performers showed stronger connections related to a few unexpected problem-solving behaviors. In general, they showed stronger connections spanning the three categories of actions for adding and removing concepts. Both were connected to short readings (information seeking) and long viewing of quiz results (solution assessment).

Two action types, short readings and long viewing of quiz results, appears to be points of divergence in the behaviors of the two groups. Both co-occurred more frequently with adding and deleting a concept for low performers and with the expected coherent problem-solving behaviors for high performers.

6 Discussion

6.1 SRL Behaviors of Low and High Performers in Betty’s Brain

In this paper, we generated a set of epistemic networks to illustrate the co-occurrences of actions related to three main categories of open-ended problem-solving actions: information seeking, solution construction and solution assessment and interpreted them as SRL processes used by the students. Our analyses investigated overall co-occurrence for all students as well as differences across students based on their performance.

In line with prior studies [12, 26], ENA revealed that high performers showed stronger connections related to the expected SRL problem-solving process. Searching through resources (short readings) was more often connected to finding the relevant one (coherent readings). When constructing the causal map, high performers showed stronger connections between adding a new concept and adding coherent links. They showed stronger connections to coherent revisions of existing links. When assessing their solution, they showed stronger connections between quiz taking, quiz result views, and coherent responses to those results (readings and adding or revising a link). While it is not surprising that high performers showed stronger connections related to the expected SRL process, ENA was an effective tool for revealing those connections.

Perhaps most surprising was our observation that, while high performers show more connections related to coherent actions, connections for incoherent actions showed no or weak differences between high and low performers. In other words, the high performers were better at implementing efficient problem-solving strategies, but both groups had difficulties inhibiting inefficient behavior.

One way low performers differentiate themselves is their behavior related to adding and deleting concepts from the causal map. ENA showed that the co-occurrence of these actions with both information seeking (short reading) and solution assessment (long quiz result views) was more frequent for low performers. This observation raises questions about the reasons why low performers add and delete concepts in the map. Concepts are well identified in the hyperlink resources and adding a concept is achieved through a fixed menu of concepts that are needed to build a correct map. Unlike causal links, which are inferred by students and might need to be revised, it should not be necessary to delete a concept. One possible explanation m be that low performers are unable to fully understand the resources, leading to uncertainty and therefore, adding and deleting of concepts. Further investigations may be required to better understand this phenomenon and its association with poorer performance.

Our analyses of the networks also suggested differentiations in actions co-occurring with asking Betty a causal question. Because this action type is infrequent (1.03% actions), its connections in the networks are weak. However, low performers showed stronger connections than high performers. This result was surprising considering past research [27], using differential sequence mining, suggested that high performing students were more likely to combine asking Betty a causal question and following up by asking Betty to explain her answer. This raises questions related to how students made use of this feature in our study and why it was mainly associated with low performance.

6.2 Factors Influencing the Interpretation of Epistemic Networks

While ENA was an effective tool in our investigation, there were factors that influenced how each of the networks were interpreted. Most important was the difference in the frequencies of different action types. Some types were either very frequent (e.g. short readings) or uncommon (e.g. asking Betty a causal question). Such differences in frequency can impact the visual representation of the network. For example, short reading actions tend to be central to whichever network includes it and its connections are usually the strongest. Visually, the thickness of the connections between other action types might be impacted by the thickness of the connections associated with short readings. While we decided to include short readings in every relevant network, we also experimented with networks that excluded them. We observed that doing so increased the thickness of other links and made the networks easier to interpret. Mello and Gasevic [28] observed a similar effect where “the exclusion of the dominant code led to an entirely different configuration in ENA.” Further research might be necessary to formally investigate when it is acceptable to exclude dominant codes and to assess the impact of such exclusions on ENA results.

Another factor we observed is related to the statistical significance of the difference between networks. In our analyses, we wanted to systematically investigate differences within a specific category of problem-solving actions (e.g. information seeking alone) and across multiple categories (e.g. information seeking and solution construction). For this purpose, we generated multiple networks using different sets of actions types. While analyses of statistical significance allowed us to say whether there was a difference between two networks in their entirety, it did not allow us to identify whether a specific subset of action types had the most impact in driving this difference. For our purpose, this was important when comparing networks across multiple categories.

In such cases, we were interested in knowing whether the connections between those two categories were different across groupings. However, the generated networks would still include connections between actions within the same category. For example, the networks generated to investigate the co-occurrences of solution assessment and information seeking actions (Fig. 7) showed a statistically significant difference. However, visual inspection of the network showed that stronger connections were local to action types within the same category. I.e., the strongest connection for high performers was between two solution assessment actions and the strongest connections for low performers were among three information seeking actions. In such a case, the statistical significance of the difference between the two networks was not a useful tool for us to investigate differences in behaviors that spanned those two categories. Further work should investigate whether masking specific connections in the networks, a feature available in the R version of ENA might allow for a more targeted comparison of how different groups of students navigate the transition between categories of open-ended problem-solving actions. Alternatively, it might be beneficial to complement ENA with other methods focusing on the identification of local behavior patterns such as sequential or differential pattern mining [27]. ENA allows a holistic test between groups, while the other methods can examine differences in single pairs of actions.