Keywords

1 Introduction

Even though the recent influx of tablets in primary education goes together with the vision that educational technology empowered with learning analytics will revolutionize education, empirical results supporting this claim are scarce [1]. Specifically, advances are expected in adapting learning materials to the needs of individual students, leading to enhanced educational effectiveness [2, 3]. Learning analytics are expected to play and important role in driving adaptive learning and are often conceptualized by the distinction between embedded and extracted learning analytics [4]. On the one hand, embedded analytics refer to cases where the data is used directly by the learning technology, for example providing different practice assignments to different students based on their ability profiles. On the other hand, extracted learning analytics refer to instances in which data are made available for different actors, such as teacher or students. Teacher dashboards are an often-used example of extracted analytics [5]. Dashboards can be conceptualized as new instruments that help teachers to improve their daily practice. However, we know very little about how teachers are using dashboards and how this affects their pedagogical actions such as the feedback they provide or the instruction they give.

A way to theoretically ground teachers’ dashboard usage is through the distributed cognition theory. This theory states that instruments can indeed support professionals, when these instruments fit seamlessly into the activities of a professional [7]. Extensive research in domains ranging from aviation to medicine shows that the connection between instruments and the professional’s routine is of great importance for the successful usage of new tools [8]. For example, a new tool in an aircraft must fit seamlessly into the daily routine of the pilot and his crew to prevent accidents. In classrooms dashboards can be considered a ‘new’ instrument for teachers to support them in selecting effective pedagogical actions [6]. Where the distributed cognition theory provides a research paradigm to view the use of instruments during professional functioning, the Verberts’ learning analytics process model can be used to investigate how teachers use dashboards specifically in their daily classroom contexts.

Verberts’ learning analytics process model specifies the stages users go through from interpretation of the data on the dashboards towards meaningful actions [5]. Four stages are distinguished in this learning analytic process model. First, in the awareness stage the user becomes aware of the dashboard and the data available. Second, in the reflection stage the user interprets the data by asking questions and evaluating the relevance of these questions. In the third, sense making stage the user answers the relevant questions to further understand the value of the data. Finally, in the impact stage, the user’s understanding of the data is employed to change his/her behavior. Hence, this model applies to teachers and their use of dashboards in their teaching practices [5]. Teacher dashboards are directed at teachers to better understand students’ ability, learning process and progress. Often these dashboards represent information about students’ progress on different learning goals and show correct and incorrect answers students have given on assignments [5]. In the awareness stage teachers become consciously aware of the data in the dashboards. For example, they explore which information is shown. Next, in the reflection stage, teachers start asking themselves questions about the data, such as “how can I see if students are understanding the material?”. In the sense making stage teachers try to answer their questions, and try to understand how the data are informative for their teaching and how they relate to their pedagogical actions, for example, how do the data show that a student is struggling which would call for feedback or additional instruction. Finally in the impact stage teachers determine pedagogical actions that respond to the data in de dashboard. Pedagogical actions are interventions teachers take to support students’ learning, for example providing additional instruction to improve individual students’ progress.

Generally, teachers constantly make decisions leading to pedagogical actions [9]. These pedagogical actions are based on teachers’ pedagogical knowledge base that consists of knowledge, skills, perceptions, and personal characteristics and entail knowledge on both student and class level. Important knowledge elements are understanding of individual students’ abilities, students’ domain knowledge and skills, but also common developmental problems students face during learning and how they are indicated by particular errors students make. Furthermore, knowledge on the class level deals with social dynamics within the group and understanding of the knowledge and skill development at the group level. All these knowledge elements can be used to select appropriate pedagogical actions. Information on the dashboards can add to teachers’ knowledge base on both student and class level. Therefore, when teachers go through the stages of the learning analytics process model to interpret data on the dashboards, it is likely that they activate their pedagogical knowledge base. To understand how dashboard data affect teaching, it is important to understand which additional knowledge teachers activate to understand the data and reason towards pedagogical actions.

Both on a class and student level, teachers can use data to adjust pedagogical actions such as instruction and feedback. Teachers in Dutch primary schools often follow the direct instruction model [10]. In this model a lesson consists of 7 phases. First teachers present the general topic of the lesson and assess students’ prior knowledge in the introduction phase. Second, in the goal setting phase, teachers elaborate on the learning goal of the lesson and their expectation of students’ learning. Third, during the instruction phase, teachers give class-wide instruction adjusted to the class’s knowledge and skills. Fourth, in the guided practice phase, students practice together with the teacher. This stage is important for teachers to determine if all students understand the instruction provided. Fifth, in the independent practice phase, students work on practice assignments individually. Sixth, during the independent practice phase, teachers may give extended instruction to low ability students. After the extended instruction, all students are working in the independent practice phase and teachers provide help to individual students using a range of pedagogical actions. Often they provide additional instruction or they give students feedback. Feedback is defined as individual support, which helps the student progress. It can be directed at the task, person, progress, metacognition, or social aspects of learning [11]. Teachers’ pedagogical actions can also entail selecting different learning material for a student, or changing the pace for students depending on their needs and progress. The seventh and last phase of the lesson is the reflection phase, in which the teacher reflects on students’ practice and progress.

The dashboard information can have different functions in different phases of the lesson. For example, if a teacher sees that a number of students are making similar mistakes during independent practice, he/she can give additional instruction to this particular group. However, if this information is provided during guided practices, the teacher might change the instruction. Moreover data on learning phase may be viewed in a different light during the guided practices phase as compared to during the independent practice phase. For example, if a particular student is much slower compared to the other students, in the guided practice phase where the strategy is discussed together this could indicate a problem with prior skills and knowledge, whereas during the independent practices phase this could indicate an individual problem, for instance this student is using a less effective strategy. Additional insight provided by clicking in the dashboard to show the type of errors and mistakes this student is making can support the teacher to investigate the students’ errors and define an appropriate pedagogical action.

To conclude, extracted learning analytics in the form of dashboards provide teachers with concurrent information about students’ abilities, progress, performance, and errors made. This information can be useful to adjust pedagogical actions and teaching behavior, but only when teachers are aware of the data and able to interpret the data properly and translate this understanding into appropriate pedagogical actions. Consequently different stages of the learning analytics process model are a prerequisite for effective teacher usage of dashboards in different phases of the lesson. Accordingly, this study explores how teachers go through the different stages of the learning analytics process model in their use of extracted analytics, thereby increasing our understanding of the usage of dashboards by teachers in the classroom context. The following research questions are examined:

  1. 1.

    How often do teachers consult the dashboards during a lesson and in which phases of the lesson?

  2. 2.

    Which pedagogical knowledge do teachers activate to interpret the data on the dashboards?

  3. 3.

    What pedagogical actions do teachers take after consulting the dashboards?

  4. 4.

    How do teachers’ teaching practices change in response to the usage of dashboards?

In this study, the awareness stage of the learning analytics process model is operationalized by how often teachers consult the dashboards during lessons. The reflection and sense making stage are combined by assessing which pedagogical knowledge teachers activate to interpret the data provided on the dashboard. The pedagogical actions that teachers take after consulting the dashboards provide first insights of the potential impact dashboards can have. Finally, changes in teaching practices are determined by comparing teachers’ pedagogical actions after consulting the dashboard to teachers’ pedagogical actions that are initiated without dashboard consultation. It is important to note that consultation and pedagogical actions are directly observed in the classroom context, whereas the reflection and sense making are assessed through a stimulated recall interview with the teachers.

2 Method

2.1 Sample

In total, 38 teachers of 8 different primary schools participated in this study. 30 teachers were female and 8 were male. The participating teachers each taught a different class, ranging from Grade 2 (8-year-old students) to Grade 6 (12-year-old students). On average teachers had 19 years of teaching experience and 2 years experience with tablet education. Each teacher was observed during a 50-minute mathematics lesson, dealing with the topics of the math curriculum the school follows. Teachers agreed to participate in the study and were interviewed directly after the lesson.

2.2 Adaptive Educational Technology

The adaptive educational technology used in this research is called ‘Snappet’. This technology is mainly used for mathematics and spelling across primary schools in the Netherlands. The mathematics and spelling assignments in ‘Snappet’ are comparable to those used in traditional paper workbooks. This educational technology operates on tablet computers and features both adaptive assignments (embedded analytics) and dashboards (extracted analytics). Children receive immediate (knowledge of results) feedback after finishing each assignment. Next to pre-selected assignments that are the same for all students in a class, the technology features adaptive assignments, which are adjusted, automatically to students’ performance levels. The technology uses a derivative of the Elo rating system to adapt assignments to the current ability level of the individual student [10]. The system uses the algorithm to model the probability of a student answering a question correctly. The algorithm calculates a student’s ability score, which is the representation of a student’s ability on a particular learning objective. The ability score represents an expected outcome for a given assignment with a specific difficulty level. Once the student finishes an assignment, the ability score is re-calculated using the difference between the expected and actual outcome. Based on the ability score the next assignment with matching difficulty level is selected.

Teachers use this technology in a blended educational scenario in which traditional instruction is combined with practice on the tablet computer. Class wide teacher instruction plays an important role in this scenario. After the teacher has explained a new topic to all students, students first practice with the same pre-selected assignments. In the next phase, students work with adaptive assignments. This adaptive practice supports individual practice at the students’ own ability level.

The dashboards.

The technology captures real-time data of learner performance, which are concurrently displayed to the teachers on dashboards. The system includes three different dashboards for teachers. The lesson overview dashboard indicates the performance of students on the pre-selected assignments, see Fig. 1. Teachers can monitor this dashboard to see the progress of the individual students. Green blocks indicate that a student has answered an assignment correctly. Orange blocks denote that a student eventually answered the question correctly after one or more incorrect attempts. Finally, red blocks indicate that the student did not manage to give a correct answer. As this dashboard is updated concurrently during the class it also provides information on students’ pace. The class overview dashboard provides an overview of the performance of the students compared to all other students using the system, indicating to which norm group each student belongs (10% best students, 20% best students, etc.). Finally the progress dashboard is used when students work on adaptive assignments. This dashboard indicates which students are progressing on their learning goals, are stable, or slow down in their progress.

Fig. 1.
figure 1

Teacher dashboard lesson overview.

Full size image

2.3 Measurements

The observations.

To examine how often teachers consult the dashboards during a lesson classroom observations were performed by trained research assistant that were seated in the classroom. They observed the teacher’s tablet or computer screen and were logged in the adaptive educational technology being able to see the teachers’ dashboard. Every time the teacher consulted a dashboard, the observer wrote down the time, made a screen shot of the dashboard and coded the pedagogical action that followed. Pedagogical actions were classified as: no action, feedback, instruction, adjustment of learning materials, or adjustment of pace, see Table 1. Feedback actions were actions in which the teacher gave information to the students about their learning process. Instruction actions were instances where the teacher provided additional instruction to one or more students. Adjustment of learning materials included actions in which the teacher customized the learning materials for an individual student or a group of students. Adjustment of pace included allowing one or more students to work shorter or longer on a particular section of learning materials. The Cohens’ Kappa on all categories was acceptable to good, ranging from .71 to .90. Finally, in case an action was directed at an individual student, the research assistant also wrote down the ability level of the student that was addressed. This was done by using classroom plans indicating the ability level of children based on the seating arrangement.

Table 1. Pedagogical actions
Full size table

Stimulated recall interviews.

After each observation, the research assistant discussed all dashboard consultations in a stimulated recall interview with the teacher. The teacher was asked to indicate which knowledge he/she used to assess the data in the dashboard. By means of a grounded analysis the teachers’ answers were classified in the following categories: knowledge of the student, characteristics of the student, progress of the student, error analysis, knowledge of the class, characteristics of the class, and agreements with the class, see Table 2. The Cohens’ Kappa was acceptable to good, ranging .69 to .94 for different categories.

Table 2. Teacher pedagogical knowledge activation
Full size table

Children’s mathematics ability was determined by using the national standardized mathematics assessment, CITO Mathematics [CITO Rekenen-Wiskunde]. Students were divided in three ability levels. The high ability group represented the top 25%, the middle ability group contained the middle 50% and the low ability group represented the lowest scoring 25%.

3 Results

3.1 Awareness: Dashboard Consultation

The first research question addressed how often teachers consult the dashboards during a lesson. In the 38 lessons that were observed, teachers consulted the dashboards a total of 317 times. On average, teachers looked at the dashboards 8.34 times per lesson with a standard deviation of 5.22 times. There was quite some difference in how often teachers consulted the dashboards, ranging from 2 to 22 times per lesson. Three groups of teachers could be distinguished: −1 SD and below (between 0 and 5 dashboard consultations in a lesson) consisting of 13 teachers, between −1 SD and +1 SD (between 6 and 10 dashboard consultations) consisting of 15 teachers, and +1SD and above (between 10 and 22 dashboard consultations) consisting of 10 teachers. These groups are further distinguished as the low, medium and high consultation group.

With respect to the positioning of dashboard consultations during the 7 phases of the direct instruction model, 69% of the dashboard consultations were during the independent practice phase, see Fig. 2. Teachers also consulted the dashboard during the reflection phase to evaluate how the class performed. During other phases consultation was minimal.

Fig. 2.
figure 2

Percentage of dashboard consultations during the phases of the direct instruction model

Full size image

3.2 Reflection and Sense Making: Pedagogical Knowledge Activation and Data Interpretation

The data from the stimulated recall interviews showed that teachers indeed reflect on the data when consulting dashboards. Teachers did activate their existing pedagogical knowledge to interpret the data in the dashboards. For example, teachers would activate knowledge of a particular student to interpret why his pace was different from the other students. Based on the teacher’s knowledge that this student was often very accurate, he could determine if the current information on the dashboard was different than expected and create new meaning. Figure 3 provides an overview of the types of pedagogical knowledge teachers activated. Knowledge of the student (66 of the 317 dashboard consultations) was activated mostly to understand the data in the dashboards. Furthermore, teachers often made an analysis of the type of errors students made (56 dashboard consultations) and of students’ progress (55 consultations) to determine which particular type of support a student needed. Teachers also activated pedagogical knowledge on progress of the class (46 consultations), knowledge of the class (37 consultations) and agreements with the class (34 consultations) to interpret the data and determine pedagogical actions. Teachers mostly activated knowledge on the individual student level, namely 60% of the time versus 40% knowledge at class level.

Fig. 3.
figure 3

Percentage of types of pedagogical knowledge activated.

Full size image

On average teachers activated 4 different knowledge types during a lesson. We examined if the dashboard consultation rate was associated with the diversity of the activated knowledge. A three-way ANOVA analysis indicated that there was a significant difference between teachers in the low, medium, and high consultation group, F(2,35) = 30.94, p = .001. Post-hoc Bonferroni analysis indicated that there were significant differences between all three groups. Teachers in the low consultation group on average activated 2.54 different types of knowledge, teachers in the medium consultation group activated 4.00 types of knowledge, and teachers in the high consultation group activated 5.00 different types of knowledge. Additionally a difference was found between the three groups in the types of knowledge that teachers activated. Low consulting teachers mostly relied on knowledge of student and class. Medium and high consulting teachers also engaged in error analysis and used progress information of both class and students more frequently.

3.3 Impact: Pedagogical Actions

The actions that followed dashboard consultation are outlined in Fig. 4. The action that was most likely to follow after a teacher looked at the dashboard, was providing feedback to a student (N = 118). For example, “you are doing really well, keep this up Ann”. Often teachers provided additional instruction to the class or to a particular student (N = 90). For example, “you need to remember to add that number”. Adjustment of learning materials (N = 17) and adjustment of pace (N = 10) were less frequently taken pedagogical actions. About a quarter of the dashboard consultations (N = 82) were not followed by any explicit teacher action. Of the actions that were performed following dashboard consultation, 50% was directed at individual students, 7% at a group of students, and 43% at the class level.

Fig. 4.
figure 4

Percentage of types of pedagogical actions taken after dashboard consultation

Full size image

On average teachers activated 3 different types of pedagogical actions during a lesson. We determined whether the number of consultations of the dashboards was associated with the diversity of the pedagogical actions. Possibly teachers that consulted the dashboards more frequently, were also more likely to show more diversity in pedagogical actions. Indeed a three-way ANOVA analysis indicated that there was a significant difference between the low, medium, and high consulting teachers F(2,35) = 20.31, p = .001. Post-hoc Bonferroni analysis indicated that there were significant differences between all three groups. Low consulting teachers on average took 2.08 different actions, medium consulting teachers took 2.80 different actions, and high consulting teachers took 3.90 different actions. Additionally a difference was found between the three groups in the types of pedagogical actions. Low consulting teachers mostly gave additional instruction or did not engage in any action. Medium and high consulting teachers gave feedback more often and high consulting teachers also adjusted materials and pace.

3.4 Sense Making: Relation Between Knowledge Activation and Pedagogical Action

In order to further understand teachers sense making proces, the relation between knowledge activation and the three most likely following pedagogical actions were assessed. Table 3 shows the correlations between activated knowledge types and pedagogical actions. Instruction was related to activation of knowledge about the class or students which indicated that teachers need to augment the information in the dashboard with existing knowledge to be able to determine which instructional actions are appropriate. Feedback actions were driven by agreement with the class, for example, “you are working according to our agreement, well done!” Also error analysis and progress of students drove feedback actions. This indicated that individual feedback actions were supported by data on mistakes made and progress and augmented by reasoning on the types of mistakes and meaning thereof. Finally, in cases where no action was taken, teachers often simply confirmed that the class was making appropriate progress or they assessed errors and felt no immediate need to intervene.

Table 3. Correlations between pedagogical actions and teacher activated knowledge
Full size table

3.5 Prolonged Impact: Changing Teacher Behavior After Dashboard Consultation

Teachers naturally initiate pedagogical actions during their lessons and now they also initiate pedagogical actions after dashboard consultation. To determine whether teachers alter their practices due to the dashboard information, we examined if teachers’ pedagogical actions after dashboard consultation were directed at students with different abilities compared to teacher actions that were initiated naturally (i.e. without dashboard consultation). We found that indeed there was a marginally significant difference χ2(2, 329) = 5.84, p = .054. Naturally initiated teacher actions were mostly directed at low ability student, which (see Fig. 5). However, after dashboard consultation teachers supported medium and high ability students more often. The dashboard information seemed to guide teachers to support students that normally would have received less support.

Fig. 5.
figure 5

The division of pedagogical actions directed at different ability groups

Full size image

4 Conclusion

This study examined how teachers use teacher dashboards, a form of extracted learning analytics, during mathematics and spelling lessons. The results showed that teachers were aware of the dashboards and consulted them on average over 8 times in a 50-minute lesson. However, a large diversity in consultation was found. Low, medium and high consulting teachers were distinguished. As expected dashboard consultations were mostly occurring during the independent practice phase of the lessons, but also during the reflection phase the dashboard was consulted. Typically in the independence practice phase teachers were giving additional feedback or instruction to students. In the closure phase teachers would provide additional feedback reflection on the student or class progress as input to discuss problems students’ faced.

Teachers indeed reflected on the data and activated additional pedagogical knowledge to interpret the data as suggested by Verberts’ learning analytics process model [5] and Roelofs’ pedagogical knowledge bases model [8]. Knowledge on the individual student level was activated more often, but also knowledge on the class level was used by teachers to make sense of the dashboard data. In line with these results, 50% of the pedagogical actions following dashboard consultation were directed at individual students, about 7% was directed at a small group, and 43% at the class level. At the individual student level, pedagogical actions taken were feedback and providing students with additional instruction, whereas at the class level teachers most often gave additional instruction. Surprisingly, about a quarter of the dashboard consultation were not followed by any explicit teacher action. This seems to indicate that the dashboard was also used as a tool by which teachers confirm their own assessment of students’ and class progress.

The analyses showed that Verberts’ learning analytics process model can be used to progressively study teachers’ use of dashboard data. Teachers, who consulted the dashboard more often, also activated more and different types of pedagogical knowledge to interpret the data. Consequently, they also engaged in more diverse pedagogical actions. In line with this development, teachers who view the dashboards more often also analyzed students’ errors and progress more often. This was associated with more feedback and adjustment of students’ pace and learning materials. This suggests that more diverse teaching practices are associated with awareness, reflecting and sense making of the dashboard data. Moreover, behavior change was evidenced by a shift in the type of students that were directed by teachers’ pedagogical actions. Middle and high ability students received support more often after teachers looked at the dashboard. We expect that the data on the dashboard highlighted the need for support for these groups of students. Possible consequences of this shift in teacher attention an important focus of future research.

Overall, this study shows that teachers were indeed using the dashboards and this seemed to influence their daily teaching practices. Interpreting our results in the light of the distributed cognition theory, we can conclude that information in the dashboard connects to teachers’ professional routine and teachers are indeed able to successfully usage these new tools. The stages of Verbert’s learning analytics model support the analysis how teachers use dashboards. The data drove reflection and sense making and teachers used their existing pedagogical knowledge to come to new understandings, which lead to pedagogical actions. This study indicates that there were changes in teacher behavior due to using the dashboard. These actions can potentially support educational effectiveness, but future research needs to investigate this further. Moreover, developments in teacher usage of dashboards over time as well as the role of experience and possible interactions with professional skills need to be explored in future research.

Consequently, we conclude that dashboards seem to impact the way teachers teach. The diversity among teachers as indicated by the differences between low, medium and high consulting teachers and related differences in reflection, sense making and impact could be indicative of a development in the usage of the dashboards, but this hypothesis needs future research to be tested. Naturally more research is needed to further explore the way teachers are using dashboards and to come to a more profound understanding of the associations found in this exploratory study. Yet there are ample opportunities to improve the human-technology interaction and the usage of dashboards. As indicated by the learning analytics process model, it starts by making teachers more aware of the data. Moreover, training teachers to interpret data with their existing pedagogical knowledge might help teachers forward. Moreover, as suggested by the distributed cognition theory, a good connection between the instrument (dashboard) and the repertoire of the professional can support successful usage in daily classroom practice. This connection can be improved by adding new services to the existing dashboards, for example highlighting important dashboard information and making potential pedagogical actions more explicit with recommender services.