Keywords

1 Introduction

There is a need to better integrate research on learning analytics dashboards (LAD) and open learner models (OLM) [1]. LADs aim to make data actionable, commonly using traditional bar charts, line graphs, tables, pie charts, network graphs [2]. Learner models are dynamic models of learning that allow personalisation; OLMs externalise this model to aid learner decision-making [3]. Whilst some OLMs use traditional visualisations, many use other methods: Fig. 1 outlines examples. Skill Meters 1&2 show knowledge level in the filled part of the meter. Bullets use fill in the bullet. Graph has positive data on the right of the axis; problems on the left. Grid uses colour to show understanding. Table 1 lists competencies in columns from weak to strong; a dot in a cell indicates strength of each competency. Table 2 ranks understanding. Radar Plot portrays learning across a curriculum by fill and position. Histogram shows knowledge from weak to strong. Word Clouds have strong competencies in larger text on the left; weak competencies on the right. Treemap 1 shows competency by size of the corresponding area; Treemap 2 uses colour (size shows number of problems). Circle also uses colour. Network and Hierarchical Tree have hierarchical structures similar to that shown by indenting sub-topics in Skill Meters 1, Table 1, or zooming in Treemaps 1&2. Pre-requisites and Concept Map show corresponding relationships.

Fig. 1.
figure 1

Outline of layout of some common types of open learner model visualisation

Full size image
Table 1. Understand/would use view; easily identify well known/not well known.
Full size table
Table 2. Typical responses for some of the visualisations claimed as most likely to be used.
Full size table

In multiple view OLMs, skill meters tend to be viewed more if they are an option, though all views are accessed [4, 5, 11]. Nevertheless, whilst skill meters were popular amongst Fig. 1 screens when considering ‘what to work on next’, pre-requisites and hierarchical tree were anticipated more useful for that purpose [12]; concept maps were more effective than skill meters to synthesise an overview in a controlled study [13]; and a simple ranked list was favoured over other views (including concept map, pre-requisites, hierarchical tree) in an experimental study [10]. Individuals may also use different views depending on reason for viewing [5]. This suggests a need to further investigate the relative usefulness of different views. Studies have typically been with single systems, so whilst several views have been compared, some have not yet been contrasted. As a first step, we follow approaches where screens were designed to gauge interest in options before deciding which to implement [14, 15], but we instead take visualisations from a range of EXISTING OLMs. These are from our own OLMs (for accessibility), but we use views that are similar to those commonly deployed.

2 Evaluation

38 students from School of Information Sciences, University of Pittsburgh, accepted an email invitation and were compensated $20. 16 had previously used Grid. The Fig. 1 screens were shown. Likert scale questionnaires (strongly agree 5-strongly disagree 1) included space for comments. Differences in learning data (concepts, skills, knowledge level, competencies) were not highlighted, to avoid responses based on this.

Results: Each view had some expecting to use it, and some not. All anticipated using multiple views (mean 8.5; median 8; range 3–14). Table 1 shows Skill Meters 1&2, Tables 1&2, Treemaps 1&2, Bullets, Graph, Pre-requisites Map most easily understood: at least 30 claiming ‘I understand the purpose of [view]’ (agree/strongly agree); then Grid, Histogram, Network, Hierarchical Tree, Concept Map, with 27–29. ‘In a system with many visualisations I would use [view]’ had Skill Meters 1&2 as most likely (>70%); and at least half chose Graph, Grid, Table 2, Network, Pre-requisites Map, Hierarchical Tree, Concept Map. For the two items ‘I could easily identify topics I know well/do not know well using [view]’: at least half responded positively for well known topics for all views; only Word Clouds and Tree map 2 had less than half for topics not known well. Skill Meters 1&2 scored especially high for both.

Most views attracted positive (+) and negative (–) comments (C). Table 2 provides typical examples, often indicating the positive uses of detail as well as negative perceptions of too much detail. As domains often use a hierarchical structure, to further highlight individual differences in preferences, Table 3 shows the hierarchical views chosen by the ten students expecting to use the least views (3–6) overall (mean 1; median 1; range 0–2). The most popular, Skill Meters 1, was selected by only half; Hierarchical Tree, by three; a Treemap, by two. Three participants anticipated using no hierarchical view, but each of these selected Table 2, which CAN show structure in topic labels (e.g. dashes before sub-topics, as in the screen in the study). The other 28 students (7 or more views) chose at least one hierarchical view (mean 3.7; median 4; range 1–6). A combination of Skill Meters 1, Hierarchical Tree and Network would cover these participants’ preferences; omitting any one of these would leave only one student with no preferred hierarchical structure. (Considering all participants together, values for expecting to use hierarchical views were: mean 3; median 3.5; range 0–6.)

Table 3. Hierarchical views expected to be used by those using the fewest views overall (3–6).
Full size table

Discussion: OLMs use not only traditional methods of information visualisation often found in LADs (see [2]), but also other options, from simple displays for a quick overview, to highly structured views that include information about relationships. This paper identified that, apart from Radar Plot, Word Clouds and Circle, all views were claimed to be understood by at least two thirds (and only Word Clouds by less than half). For a multiple-view OLM, over 70% of participants anticipated using Skill Meters 1&2; and at least half, Graph, Grid, Table 2, Network, Pre-requisites Map, Hierarchical Tree, Concept Map. These nine visualisations were also considered easily usable to identify well known and less known topics by at least half of the students.

Skill Meters 1&2 were the top views for all four questionnaire items. This echoes findings that skill meters are used frequently in practice when amongst the options available [4, 5, 11]. However, initial results looking at the same views to identify ‘what to work on next’ revealed different preferences (Pre-requisites Map, Hierarchical Tree) [12], compared to findings here for well known/less known topics (Skill Meters 1&2), though over half also stated they would use Pre-requisites Map and/or Hierarchical Tree in a multiple-view OLM. Furthermore, of those who expected to use fewer views overall, only half anticipated using the hierarchical Skill Meters 1. We therefore recommend considering providing each of the above (Pre-requisites Map, Hierarchical Tree, one of Skill Meters 1&2) to the extent that the domain structure allows.

A controlled study found a concept map more effective than skill meters for synthesis of an overview [13]; a ranked list was used more often than other views in an experimental study [10]. Our Concept Map had three quarters claiming it understandable, and half stated they would use it in a multiple-view OLM. Table 2 positions topics on five levels, similar to a ranked list, and is considered understandable by as many as for Skill Meters 1&2, and over half stated they would use it. We therefore further propose considering Table 2 and Concept Map. However, we also had participants finding the above views difficult. Comments revealed perceptions of too much detail; others found detail useful for specific purposes: identify knowledge (Table 2) or conceptual relationships (Concept Map), form a study plan (Pre-requisites Map), navigation (Hierarchical Tree). This supports multiple views for different users, or individuals for different goals. Students can explain why they use different views at different times [5], indicating they understand the relative benefits and limitations FOR THEM; and consistent with OLMs where other views are used as well as skill meters [4, 5, 11]. Whilst we largely support using multiple views, when an OLM has a specific purpose, we suggest a view similar to one of the above (e.g. concept map for an overview of understanding of conceptual relationships; pre-requisites to plan activities).

Since the range values show most views had some students expecting to not use them, we also consider alternatives from the remaining views that had at least half anticipating using them and stating they would be able to identify known and less known topics: Graph, Grid and Network. We do not suggest Network or Graph if only one view is to be employed, since these were not the most used views in their respective deployed OLMs [4, 5]; and Network shares the structure of Hierarchical Tree, with Graph very similar to Skill Meters 2 - both already recommended above (with higher scores). However, these may be useful as additional views to provide greater choice. Thus, we propose the above 8 visualisations be considered as options in a multiple-view OLM. Grid is more difficult to compare, since 16 participants were already familiar with it. This may have inflated the values, but Grid has been successfully used in practice in a single-view OLM [7], and may be helpful when space in an interface is limited (e.g. where the OLM is displayed together with course content). Bullets are an interesting case: they were claimed to be more easily understandable than any other visualisation, scored well for identification of known/less known topics, but had less than half expecting to use them. For this reason, we suggest Bullets for cases where space is especially restricted as they will likely be understood, and offer information in a similar manner to Skill Meters 2. However, we would not propose Bullets as a replacement where Skill Meters 2 (or 1) can be used. Finally, Treemap 1 could be considered for a large, many-layered hierarchical domain, also to efficiently use available space. (These three views may also be useful as additional options in a multiple-view OLM.) Table 4 summarises our initial suggestions, to be supplemented with any additional visualisations appropriate for the specific context.

Table 4. Visualisations suggested for consideration for single and multiple-view OLMs.
Full size table

Because domains are often structured hierarchically, we further consider responses relating to these views. The median for anticipated use of hierarchical views was 3.5. Skill Meters 1, Hierarchical Tree and Network together would satisfy all 28 participants anticipating using at least 7 views; removing any would leave only one person with no preferred option. However, amongst the ten choosing only 3–6 views overall, four would have no favoured hierarchical view. Three of these expected not to use any of the hierarchical views, but these all opted for Table 2, which can be configured to show hierarchy levels - albeit not within the hierarchical structure (as (sub-)topics are simply ranked). Nevertheless, including Table 2 may raise awareness of the hierarchal structure, and lead to a better understanding of the full hierarchical view(s). In this case, only one student (who chose a Treemap) would have no preferred view.

Participants predicting use of a feature does not necessarily mean they will use it in practice [16], but offering visualisations that students can anticipate easily using may lead them to try to interpret the information in context. This paper therefore offers a STARTING POINT for OLM and LAD designers who are selecting learning visualisations to incorporate. Further research into these combinations in use can then be undertaken.

3 Summary

Building on findings that students may use different views when there are multiple options in an OLM, we studied reactions to typical OLMs to explore relative benefits and drawbacks of view combinations. The screenshots were largely judged understandable, though there were differences in expected use. Combining views designed to fit the purpose of viewing with ones previously successfully used, also taking into account our findings here, is a step towards providing useful alternatives in future e-learning systems, as well as for considering options when only one view is preferred.