Keywords

1 Unfolding Scenarios

This chapter weaves theory and practice issues through unfolding scenarios.

It’s a sunny day and Paulo is a new lecturer in the Digital Education team and he’s excited to be contributing to the development of new online subjects and reviewing completed subjects. He’s at his desk tasked to conduct post-teaching reviews of blended/fully online subjects. He’s a little apprehensive about all that he is privy to on the LMS (Learning Management System), and he’s familiar enough with some of the built in LMS learning analytics (LA) reporting features. He can view all the content of discussion forums, the back end of the LMS activities and log-in times and dates, the IP addresses, log-in device types and the assessment grades; essentially he has access to everything about student and staff online activity! His son is a student at the university and is enrolled in the online subject he’s about to review. Paulo is keen to view his son’s performance as his son recently told him he was sure to have done well. So what stoty might the LMS data tell?

Diana is a first year student at the same university, and her data analytics show that she’s from a low socioeconomic group and first in her family to go to university. The LMS sends her an automated welcome message on her first day at university and alerts her to the fact that it will check in with her to see how she’s going on a weekly basis and that she has a specially assigned university assistant called Sophia (unbeknownst to her, Sophia is a chatbot) to assist her with any questions 24×7. Diana is delighted to have received such a caring message and feels empowered and excited to be commencing her first year.

2 The Promises and Challenges of Big Data and Learning Analytics for Higher Education

Learning analytics (LA) in education is a newly and rapidly growing field and emerging discipline (Long & Siemens, 2011) regaled at the 1st International Conference on Learning Analytics and Knowledge in 2011 (Lak’11, 2011). With the emergence of a ‘dataistic paradigm’ (Kitchin, 2014b; Ruckenstein & Pantzar, 2015), LA was predicted to reach ‘maximum impact’ in 3–5 years in 2015 (Johnson, Adams Becker, Estrada, & Freeman, 2015, p. 6). Indeed, the UK has been a significant leader (Sclater, 2014) in the field, and universities can learn from the innovations and challenges of early adopters and innovators. The 2017 Handbook of Learning Analytics (Lang, Siemens, Wise, & Gašević, 2017) is testimony of a rapidly emerging multidisciplinary field (Lang et al., 2017; Williamson, 2017).

Learning analytics promises improved student success and retention (Greller & Drachsler, 2012; Siemens, 2013) and uses Big Data (large digital data sets) (Dyche, 2012) to analyse learning and engagement. The allure of educational learning analytics has been appropriated from business by using learner profiling ‘to build better pedagogies, empower students to take an active part in their learning, target at-risk student populations, and assess factors affecting completion and student success’ (Johnson et al., 2015, p. 12). We have impactful developments in personalised adaptive e-learning (for examples, see Pardo et al., 2018; Pardo, Jovanović, Dawson, Gašević, & Mirriahi, 2019), quality learner experiences and greater collaboration given that students’ digital interactions can be mined at increasingly affordable costs (Greller & Drachsler, 2012; Johnson, Adams, & Cummins, 2012; Johnson et al., 2013). Educational digital environments provide readily available dynamic and unobtrusive data tracking to provide learner feedback and comparisons with others, support and early warning systems, for example (Greller & Drachsler, 2012, p. 43). Digital profiling can be used for various purposes by various stakeholders from individual learners, educators, designers, administrators to external business organisations (such as the Pearson Education Group). Ultimately, our quantified and measured profiled digital selves are promoted – knowingly or unknowingly – to influence how we learn, consume and behave based on data-driven algorithms, algorithmic educationpar excellence.

There are indeed valuable benefits of LA that have been promoted to educational stakeholders as a way to improve and influence learning ‘consumption’ and ‘behaviour’ from the field of economic management using business intelligence and web analytics (Buckingham Shum & Ferguson, 2012). For example, the educational uses of learning analytics across academe can range from academic analytics (institutional analysis), action analytics (that require an action) to predictive analytics (that predict behaviours and outcomes, e.g. which students are likely to fail or succeed) (Greller & Drachsler, 2012; Long & Siemens, 2011). Gathering data and analysing data with the aim of optimising performances and improving outcomes is the basis of LA, and at first glance this is to be lauded. However, there are consequences also in that LA relies on tracking our digital imprints based on Big Data that is gathered, analysed, stored, interpreted and acted upon (Dyche, 2012; Spector, 2013) using algorithmic methods to detect patterns previously impossible (Romero & Ventura, 2013) – providing profiles of ‘quantified selves’ (Lupton, 2016). The promises for educational contexts is that to describe, diagnose and predict behaviour based on a ‘quantified’, ‘measured’, ‘audited’ and ‘surveilled’ self can improve and personalise learning. Proactively Prinsloo (2017) reminds us that ‘once we have this Big Data, it behooves us morally to use it for improved educational purposes’ that enhance students’ academic study success (see, e.g. Ifenthaler, Yau, & Mah, 2019).

At its core LA is concerned with the question, ‘Can we tell from your digital profile if you’re learning?’ (Buckingham Shum, 2014). This seems reasonable and innocent enough; however, it is vital for university researchers, educators, ethicists, lawyers and administrators to engage critically (Boyd & Crawford, 2012; Kitchin, 2014a; Selwyn, 2014, 2015) with the assumptions and consequences of LA that translate learning into numbers and visual patterns. Aptly, Lodge et al. (2017) provide a compelling discussion about how learning is inferred (and the limitations of LA) from big data sets across disciplines where there are ‘different conceptualisations of learning’ – echoing the importance of being measured in what LA can(not) achieve.

Taking on an ethics of care (Noddings, 2013) requires a criticality (hooks, 1994) in that our starting point has to be care for and care of the being done to, our learners and the consequences of their ‘quantified selves’, as well as care around LA processes. As Morris (2017) compellingly says, ‘We don’t get to stop asking questions about why and whether of our teaching simply because the digital provides algorithms that approximate answers’. Ultimately, in being measured by numbers, we are (con)figured by technology, algorithms and data through the constant scrutiny of algorithmic education and Big Data technology vendors.

This chapter shakes up the spaces around what, why and how we need to be measured in adopting LA given its performativity regimes (Ball, 2016) so that it can be ‘transformative’ and ‘carefully thought through’ (Long & Siemens, 2011). The argument is that we need LA policies and practices that are transparent, relational, co-designed and reflexive, which have at their heart social justice ethical and legal frameworks where power imbalances amongst stakeholders are confronted openly and critically.

Consequently, LA needs to be addressed in terms of its underlying performative politics (power) across ethical and legal aspects and knowledge production and across broader ecologies of learning (beyond the merely digital). This chapter provokes discussion around these issues by troubling what and how we measure (and what we don’t), to open up LA as situated in contingent specific complexities and circumstances towards questions and implications for theory, methodology and practice towards a more nuanced and measured LA.

This chapter considers the politics and consequences of Big Data and LA as it (per)forms digital profiles to promote critical and rigorous considerations of adopting LA by university stakeholders across leaders, policy makers, educators, technologists, lawyers, ethicists, programmers and students. Firstly, the ethical and legal issues arising about data are considered – given that any use of LA should start from a robust ethical basis. Secondly, the knowledge that LA produces (automated, recursive, productive) and its uses are discussed, to thirdly suggesting richer learning ecologies. The chapter concludes with a brief discussion of developing a culture of LA and implications and considerations for universities.

3 Ethical and Legal Frameworks of Big Data and Learning Analytics

There are many ‘disciplinary and ethical critiques’ surrounding data about, data for, data from and data by, which are about power imbalances amongst LA’s stakeholders (Buckingham Shum & Ferguson, 2012, p. 18, original italics). Consequently, ethical frameworks and legal frameworks around data and what constitutes data are paramount, for data is never neutral; it never speaks all on its own (Kitchin, 2014a); data is always part of larger performative assemblages that enact worlds and meaning into being. Data carries agendas that too often are not examined (Perrotta & Williamson, 2016). So issues around learning analytics’ data need to address the nature of the data, their selection, analysis and use; the data’s viewers and custodians; the data’s longevity and storage; and data betrayal (Lupton, 2016), its privacy and concerns of commodification beyond a learning institution, for example, to potential employers or LMS and Big Data vendors. Vitally, transparent legal and ethical frameworks need to underpin the data protection and privacy of every individual’s profile.

The data collected may already exist within automated systems such as the LMS or have specialised data mining programs and databases used to extract information from the LMS of an educational institution (Nelson & Creagh, 2013). Data sets may be analysed internally within an institution with a data/learning analytics unit or by external companies. Either way, data collected include an individual’s institutional engagement and profile (Sclater, 2014). Whilst this type of surveillance may be considered for the better good of the student and the institution, Orwellian fears may well be cause for concern when an individual is unaware of the data collection – ultimately it is ‘stealth data’ (Spector, 2013). Campbell, DeBlois, and Oblinger (2007) also raised issues in using data analytics around privacy, profiling, data sharing and data stewardship. Long and Siemens (2011, p. 38) also highlight the need to leverage data ‘associated with tracking students’ and ‘learning options based on deterministic modelling’. These are critical issues that need addressing across complex ethical and legal domains, and require institution-wide discussions.

There are also legal violations in using ‘stealth data’ without a person’s consent. Beattie, Woodley, and Souter (2014) in a critically titled paper Creepy Analytics warn of the legal and ethical dangers inherent in undisclosed data collection that constitute a ‘violation of trust’ and ‘academic freedom’ – here we have ‘data as a commodity’. There are power imbalances between data ‘seers’/‘viewers’/‘controllers’/‘interpreters’ and ‘receivers’. The power relationships are uneven. Indeed, online university students were significantly aware of being visible, traced, and tracked online where ‘the LMS space and its permanence, visibility, and longevity raise significant complex traceability and surveillance issues for students and lecturers’ (Al-Mahmood, 2012, pp. 24–25). Greller and Drachsler (2012) remind us of ‘the double-edged sword’ that learner data may not necessarily benefit learners, but rather the dynamic data availability adds to the power of educational institutions, governments and commercial organisations ‘to increase manipulative control over students, employees, and citizens, thereby abusing LA as a means to reinforce segregation, peer pressure, and conformism rather than to help construct a needs-driven learning society’ (p. 54). Clearly, there are significant ethical and legal implications around data for all stakeholders to consider, but significantly the voices of our students (Sclater, 2015) need to be at the heart of an ethics of carefulness in being measured.

Encouragingly, a few universities have adopted codes of practices to proposals for Student Charters (e.g. Beattie et al., 2014; Drachsler & Greller, 2016; Ferguson, Hoel, Scheffel, & Drachsler, 2016; Greller & Drachsler, 2012; Nelson & Creagh, 2013; Pardo & Siemens, 2014; Prinsloo & Slade, 2016; Sclater, 2014; Sclater & Bailey, 2015; Slade, 2016; Slade & Prinsloo, 2013). Vitally, we need ‘the highest ethical standards’ ‘based on open, transparent, participatory, accountable, shared, and ethical principles of inquiry’ (Stevens & Silbey, 2014, online). The Asilomar document outlines the six pivotal principles about data use: ‘Respect for the rights and dignity of learners; beneficence; justice; openness; the humanity of learning; and continuous consideration of the ethical dimensions of learning research’ (Asilomar Conference, 2014). Inevitably, there are challenging ethical, ideological and epistemological assumptions about data (Slade & Prinsloo, 2013, p. 1510).

Paulo discovers that his son, Dimitri, had only logged in once a week into the subject’s LMS and had only made a handful of contributions to the discussion forum.

4 Knowledge Production: Algorithmic and Datafied Education and Its Consequences

How might we produce new or novel approaches to interpret and understand data patterns and produce new knowledge? The challenges of LA are in how we model learning interactions based on Big Data with a ‘transformative potential’ to inform learning and decision-making at an individual and institutional level (Long & Siemens, 2011). Currently, ‘algorithmic’ paradigms underpin many analyses (Perrotta & Williamson, 2016; Williamson, 2017) to lead to ‘actionable intelligence’ (Campbell et al., 2007). For example, in the case of the Signals system at Purdue University, the ‘actionable intelligence’ is about ‘guiding students to appropriate help resources and explaining how to use them’ (Arnold, 2010). The Purdue Signals analytics are based on a student success algorithm to provide a student facing dashboard of red, yellow and green lights to indicate where the student is at and suggest actions. (And even in this visualisation, there are cultural biases and accessibility issues). There are various levels of analytics for various ‘actionable intelligence’ outcomes ranging from academic analytics to learning analytics. For example, Buckingham Shum and Ferguson (2012) have developed a sophisticated and rich range of LA from social learning network analytics, social learning discourse analytics, social learning disposition analytics, social learning content analytics to social learning context analytics to offer visualisations and recommendations to support learning. LA’s promises are in their power for prediction based on large-scale data sets and various statistical and mathematical algorithms where Big Data is translated into reflective and predictive discourses to assist decision-making through the ‘the quantified self” (Lee, 2013; Lupton, 2016) or ‘the quantified institution’.

Further, Lodge et al. (2017) highlight the fraught dangers and reductionisms in LA as it stands in privileging quantification data and not capturing the multiplicity and complexity of learning, with a ‘bias towards quantification, and that implies that anything that cannot be measured is not worthwhile’ (Lodge et al. 2017, p. 389). The question is: What can(’t) numbers do? And what can digitisation never capture? How do we capture engagement beyond digital clicks and algorithmic overtaking? So the caution here is that we need to be measured in understanding the limitations of the specific LA developed and in use and what underpins the learning pedagogy of the algorithmic weightings and data captures.

Dimitri has received weekly emails from the university alerting him to the paucity of his online subject contributions, which have increased his anxiety greatly.

Diana has been actively participating in the LMS weekly discussions and has received weekly encouragement reminders by Sophia, based on her LA data.

4.1 On Algorithms and Data

Learning analytics’ power is to predict and enable student success and retention (Colvin et al., 2015) through decision-making algorithms, but therein lie their potential dangers when relying on merely quantitative data without qualitative information for ‘aligning and regulating performance and behaviour of individual teachers or learners against a statistical norm’ (Greller & Drachsler, 2012, p. 47). Phillips et al. (2011) have shown the limitations of using quantitative web analytics and the need to include rich qualitative data to extend understanding of how participants learn and use LMS environments. Despite the predictive promises of LA, there are ethical problems in judgments made about an individual learner based on machine or human interpretation of machine-generated algorithmic data. This could ‘potentially limit a learner’ (Greller & Drachsler, 2012, p. 48; Lodge et al. 2017) in terms of ‘confirming old-established prejudices of race, social class, gender, or other with statistical data, leading to restrictions being placed upon individual learners’. Some studies show the greater the engagement with peers in a learning community is what produces the strongest positive effect on learning and can better predict more successful student outcomes (Macfadyen & Dawson, 2010), but learning can never be captured from solely digital environments. By monitoring clicks and digital spaces, we miss the richness of other learning that happens beyond formalised digital learning environments to the infinite array of formal and informal physical environments.

The diversity and complexity of learning, judgments made about learning impact and what data is used will always mean that ‘the reliability of a LA-supported learner profile and its usefulness to the learners will remain questionable’ (Greller & Drachsler, 2012, p. 48). Basing judgments merely on numbers alone limits the possibilities of interpreting learning in its distributed nature beyond just the digital space alone (Long & Siemens, 2011). Further, as Buckingham Shum and Ferguson (2012, p. 19) advocate, we ‘must engage fully with questions around the academic, pedagogical and ethical integrity of the principles for defining such patterns and recommender algorithms, and who is permitted to see them within the set of stakeholders’. Learning is complex and so much more than simple models can capture. And just as significantly, there are problems with assuming that a digital log in pertains to a single student, for students may work on group activities under one log in and so on; this is what Greller and Drachsler (2012, p. 49) refer to as ‘enmeshed identities’.

Data itself is never generated free of theory, and no data can ever be objective – it ‘is always framed’ through a particular worldview (Kitchin, 2014a). The issue with analytics is it can ‘present the illusion of automatically discovering insights’ and removing human bias (Selwyn, 2015). Yet there are infinite algorithms that can be used which are ‘imbued with particular values and contextualized within a particular scientific approach’ (Kitchin, 2014a, p. 5). Different algorithms play different politics in that the methods used on the same data set will result in different outcomes and ‘actionable intelligence’, and with this come invisible biases and assumptions. Buckingham Shum and Ferguson (2012, pp. 18–19) alert us to the politics at stake here: ‘who is defining the measures, to what ends, what is being measured, and who has access to what data?’. Further, who has the evaluation skills and competencies to interpret complex sets of data and patterns in our universities, and how can these be developed and sustained? So how will our universities evolve algorithmic educational expertise and training to build not only data analyses capacities but interpretative capacities towards open, reflexive and transparent, multidisciplinary understandings of learning (Lodge et al. 2017)? Some universities have progressed this significantly, such as the UK’s Open Universities New Media Institute which started in 1995 to grow to 70 research staff in 2012, who are leading the discipline of analytics (Buckingham Shum & Ferguson, 2012). Newly established centres in Australia have drawn expert staff from there to establish the Connected Intelligence Centre at the University of Technology Sydney, in Australia, which conducts transdisciplinary cutting-edge research on using data analytics to improve student learning, but other universities await such centres.

Dimitri is troubled by the alert emails as he has contributed significantly in the f-2-f discussions and had even set up a subject debating session, as well as a weekly exam study group. Perplexed by the ongoing emails, he now finds the LMS space with its formulaic structure a burden.

Meanwhile Diana has been telling her parents about how lucky she is to have her own personal university assistant, Sophia, and how it has motivated her to set up an online study group given that she doesn’t have a lot of time to be on campus physically.

4.2 On Interpretation and Contextualisation

Interpretation of data analytics requires contextualisation and multidisciplinary framing (Lodge et al. 2017) as data needs to be presented to stakeholders with the capacity to be understandable with the view to ‘surprise’, ‘compel’ or ‘motivate change’ (Macfadyen & Dawson, 2012, p. 161). Hence the usefulness of learning analytics’ data and its usability will depend on how it is presented to various stakeholders and definitions of success. Exploring meaningful patterns is one way through this, with explanations. However, patterns alone do not convey much without an integration of social theory and deep contextual knowledge (Selwyn, 2015). Data patterns can provide a starting point for additional analysis, along with other data sets. Another issue in data framing is the comparative aspects of comparing a person with others in a subject or course cohort by sheer algorithmic reduction – humans are ‘way more complex, contingent and messy… to be reduced to formulae’ (Kitchin, 2014a). Overall, there are limitations to quantitative analysis that ‘need to be complemented by other approaches’ (Kitchin, 2014a, p. 9). A LA health barometer might be well placed around ‘the quality of debate around what technology renders visible and leaves invisible’ (Buckingham Shum & Ferguson, 2012, p. 19). So assuming that we can get multiple and accessible LA data interpretations, how might these be communicated, and how might a LA culture be adopted, for better or for worse? Ultimately, any interpretations must be contextualised, but developing and communicating a LA culture becomes essential for LA to have any traction.

Paulo wonders if his son or for that matter any of the other students have access to the data to which he is privy and if it would make any difference to their grades. He wonders if they have been consulted about what might be important for them to assist with their learning?

Diana wonders about how Sophia seems to know so much about her and how she seems to provide just the right suggestion to her at the most opportune times. Diana likens it to ‘the Goldilocks effect’ – just right!

4.3 On Communicating and Adopting a Learning Analytics Culture

How LA analyses are presented should depend on the various stakeholders, and within a learning institution will typically involve strategic, administrative and academic staff, as well as tutors, laboratory demonstrators and students. The aim is to contrast perception versus actual behaviour (Greller & Drachsler, 2012) to provide ‘actionable intelligence’ towards desired actions on some level for some stakeholder. To what extent is this achievable? An example of actionable intelligence is the little-used large-scale LA intelligence in the strategic decision-making process around the LMS (Macfadyen & Dawson, 2012). Audiences matter and need to be engaged in how LA can be communicated effectively. Educators’ voices have been considered in works by Corrin, Kennedy and Mulder (2013, p. 204) to explore lecturers’ needs, perceptions and disciplinary differences of LA’s potential to show that there was a ‘considerable amount of skepticism and confusion over the utility of learning analytics’ by lecturers and that the LA dashboard reports had a ‘disconnect’ aspect to them, and they also expressed ‘concerns about the level of skill and time required to adequately engage with learning analytics in a useful way’. Whilst there are various ways that data can be presented to students and staff, with commonly used dashboard traffic light adoptions (Arnold, 2010), the assumption is that ‘actionable intelligence’ is achievable and decipherable by students and for that matter usable and understandable by lecturers. There is need for more promising work on what would constitute accessible and meaningful information such as that by Olmos and Corrin (2012) and the SNAPP visual developments by Dawson, Tan, and McWilliam (2011).

Further, there are concerns with how LA may ‘disempower learners, making them increasingly reliant on institutions providing them with continuous feedback, rather than developing meta-cognitive and learning-to-learn skills and dispositions’ (Buckingham Shum & Ferguson, 2012, p. 19). Learners can also stress over endless comparison with their peers (Long & Siemens, 2011, p. 36). In a study of learner LMS experiences (Al-Mahmood, 2012), perceptions about the LMS platform certainly meant that students were aware of performing the good student online, or at the other extreme loathed the online platform which curtailed the length of time spent in its spaces.

Dimitri realizes that he could decide to post more often to the discussion forum and maybe the email alerts telling him he has only logged in once would cease.

Diana has been wondering about other students and if they too have an individual digital mentor/assistant assigned to them. She is reluctant to ask but assumes they do (but only equity and at-risk students have had this feature assigned to them).

Hence we need to be careful and measured then with what we communicate and how we communicate any LA. We need to be cognisant not to fall into using ‘economically motivated data analyses’ and a ‘language of economics’ with LA audiences (Baym, 2013, p. 14).

For LA to be adopted by a university, there needs to be strategic vision of the complex issues involved. The work by Lodge et al. (2017); Norris and Baer (2013); Davenport, Harris, and Shapiro (2010); and Davenport, Harris, and Morison (2010) all provide strategic organisational approaches and cycles for insightful consideration. Above all, whatever the platforms chosen, whatever data chosen and whatever the ways to interpret data and feedback cycles, there needs to be considerable investment by the organisation in creating, building, sustaining and evaluating a culture of analytics (Corrin et al., 2013; Norris & Baer, 2013; Davenport, Harris, & Morison, 2010, Davenport, Harris, & Shapiro, 2010). Whilst the culture of LA adoption may seem innocent enough, there are further matters of epistemological concerns to consider.

By the final weeks of his subject, Dimitri is becoming more anxious as he now has a strong sense of being under surveillance and is feeling harassed. He simply wants to plead, ‘please let me be. I have enough study and work to do’.

Diana is progressing well and managing to stay up-to-date with everything given the reminders and actionable intelligence provided by Sophia. She’s feeling confident about the final semester exam given that she has achieved 50% of her marks already in assignments and tests.

4.4 On Data Reduction and Knowledge Production

By deconstructing and simplifying learning into algorithms, and focusing on selective data, LA can easily fall into simplistic solutionism (Selwyn, 2015) and data ‘seductions’ (Kennedy, Corrin, & de Barbara, 2017) where learning becomes a ‘spectacle’ (Ball, 2016). Fanghanel (2012, p. 24) so pertinently asks, ‘To what extent is this visualization (measuring and displaying of performance) informative? …. What is actually happening in practice is erased from these public computations’. We need to therefore be measured when using LA that are solely based on reductionist approaches as learning is far richer and messier and ‘a contestable process’ (Selwyn, 2015, p. 75). Greller and Drachsler (2012, p. 52) also highlight the limitations of merely considering LMS digital data given that ‘learning’ occurs ‘in a lifelong and diverse ecosystem’, where ‘an exclusive data view on single elements may provide a stimulus for reflection but not a sound basis for assessment’. Big Data may provide some insights, but they are inevitably limited to specific types of knowledge that deal mainly with quantification rather than measuring quality. Hence LA will always need contextualisation relative to other information.

We will always still need to put theories and context back into numbers and patterns to make sense of the world. Learning will always be more complex, and knowledge production belies values and worldviews that will need to be made explicit – ultimately LA is multidisciplinary (Lodge et al. 2017). For better or for worse, perhaps a humbler claim of LA might be to be a ‘blunt indicator’ (Lodge & Lewis, 2012) to be used with other information and indicators of student success. As Booth (2012, Online) highlights, ‘the adoption of learning analytics … must be informed not only by what can be measured but also by what cannot. There will be limits in what learning analytics can do. …. not every aspect of learning can be captured by the powerful tool that analytics promises to be. Sometimes learning is ineffable!’. The best that we might aim for is to use ‘multiple methods for assessing learning’ (Booth, 2012). Hence we need to consider broader ecologies of learning.

Dimitri has managed to submit all his subject assignments online and has worked very hard on them and is hoping for a good mark.

Diana discovers that her other friends don’t have a Sophia mentor assigned to them, and she feels very special to have had Sophia’s personalised guidance and direction.

5 Ecologies of Learning: Towards a Measured Learning Analytics

At the heart of LA, students’ learning and engagement need to be considered in their ecological richness which extends learning beyond mere digitised Big Data. By focusing on only digital LA, we lose all the other connections and complexities of a student’s learning assemblages nuanced in the physical and the immeasurable. This ecological view might provide for a reflexivity where we are open and transparent about how we come about analysing and understanding learning and the limits of LA. Armed with relational and reflexive stances that highlight our politics, ethics and values, we might then use an ‘ensemble approach’ to LA to ‘build multiple solutions’ (Kitchin, 2014a, p. 2). Reflexivity means that we shake up the ‘uncritical faith in numbers’ (Baym, 2013, online), adding ‘qualitative sensibilities and methods to help us see what numbers cannot’ (Baym, 2013, online). Only then might we be able to be measured about the degree of trust we place in LA and algorithmic education, for if we are not careful, we risk reducing ‘humanity to quantity’ (Ball, 2016, p. 1054).

Paulo knows how much effort his son has put in to getting to university and studying. He’s saddened to see that despite all the work his son has put in, the LMS analytics highlight Dimitri as an at-risk and borderline student.

6 Implications for Universities Adopting Learning Analytics

This chapter has explored the power and politics of Big Data and LA to move it beyond simple solutionism approaches so that universities can evolve rich and rigorous dialogues, policies and processes with all of their stakeholders. If LA is to have impact to do what it promises, then significant further research is required underpinned by robust ethical and legal frameworks that are based on codesign by all stakeholders. For example, the Open Universities in the UK (Johnson et al., 2015) have used participatory design and implementation of their LA with a university-wide engagement and awareness. Students are automatically opted in for analytics but with the possibility of opting out. Their eight principles ‘ensure openness and transparency around the data being collected and what is being done with it.… Learning analytics is seen as an ethical process with the student at the heart of it. Learners are not wholly defined by the data the University has about them which can never reflect the totality of the individual’s experience’ (Sclater, 2014, p. 55). The timely work by Corrin et al. (2019) also provides an updated and important contribution by LA leaders. Whatever we evolve, student voices and participatory designs have to be at the heart of how LA might progress, beyond vendor-produced products. Students as active co-designers and engaged stakeholders (Ifenthaler & Schumacher, 2016) can provide vital insights and ways to deal with the power imbalances of being acted on. Figure 2.1 encapsulates some of the vital steps in the codesign and measured possibilities of LA. Let us open up and extend a measured LA then that is more inclusive, transparent, negotiable and accountable for all stakeholders.

Fig. 2.1
A diagram depicts the 13 steps involved in codesign and measurement possibilities from determining potential solutions for problems to the establishment of stakeholder involvement.

Learning analytics model on being measured

Full size image

Dimitri has received 49% for his subject and is aghast. He had told his dad that he would be bound to pass everything. After all, he had worked night and day on this one subject in which he was enrolled and completed extensive reading research for the major assignment. Deeply troubled he decides to go and see the lecturer in person on campus. As he makes the trip across the other side of town, he rushes across amber lights and is hit by a speeding car, despite using his cane for his vision impairment.

Diana knows Dimitri as he was in her online study group that she established. She is terribly saddened to know this news. She wonders why he was not provided with a Sophia just like she was, but it turns out he did not alert the university to his vision issue, as he was a highly independent learner, and so was not picked up by the LMS system as an equity student.

What this chapter has sought to highlight is that there may well be value to a well-executed and considered LA that is measured in its claims, open, inclusive, transparent and reflexive, but that there are significant areas that need to be evolved. Any LA adoption requires engagement across all stakeholders in rigorous discussion to suit the requirements of each university. The considerations below raise important questions for universities to provoke robust and deep discussion amongst all stakeholders:

  1. 1.

    What is your university’s vision and purpose of LA use? (This question should be revisited frequently and cyclically by any institution.)

  2. 2.

    How does your university envision a successful LA adoption?

  3. 3.

    At what cost is LA to your institution (e.g. platforms and programs, financial, digital storage, expertise, impact, benefit, dangers, contingencies, etc.)?

  4. 4.

    How will your university implement LA at the macro, meso and micro levels? Who will be responsible? How will you recruit for expertise?

  5. 5.

    What is your university’s conceptualisation of learning to assist in how your LA could look and what it would capture and what would constitute data?

  6. 6.

    How will your university incorporate qualitative data to the quantitative LA analyses?

  7. 7.

    How will your university facilitate interpretive capacities and negotiated understandings of LA outputs for stakeholders to ensure meaningful analyses?

  8. 8.

    How will your university’s LA promote self-regulation and metacognitive awareness for students and other stakeholders?

  9. 9.

    How will your university establish stakeholder buy-in?

  10. 10.

    How will your university include student voices, needs and permissions?

  11. 11.

    How will your university navigate guiding policies and ethical and legal frameworks, locally and globally?

  12. 12.

    How will your university address data ownership and privacy laws and permanency and longevity?

  13. 13.

    How will your university navigate educational LA data and analyses via outsourcing to commercial vendors versus internal to your educational custodians? Who will be your data custodians?

  14. 14.

    How will your university ensure that your data analysts/scientist/educators are also trained in and understand educational learning theories?

  15. 15.

    How will your university ensure that practitioners of LA have a code of conduct around data privacy?

  16. 16.

    How will your university establish opt-in or opt-out options for stakeholders?

  17. 17.

    How will your university build digital capacity around LA?

  18. 18.

    How will your university evolve digital literacies to include LA and its consequences for educators and students and beyond? Who will teach and promote this?

  19. 19.

    What stories and whose stories will be told via your LA and for whom and by whom?

  20. 20.

    How will your university select data visualisation and interpretations of LA dashboards?

  21. 21.

    How will your university ensure your duty of care for all?

  22. 22.

    How will your university critically evaluate your LA adoption, its value, benefit and sustainability?

  23. 23.

    How will your university ensure an ethics of carefulness in your institution, and what would that look like for your institution? What does your responsible and responsive LA look like?

In conclusion, whilst LA is a rapidly evolving field that more recently is showing more nuanced use in adaptive personalised e-learning development, for example (and this too could be argued is problematic in what it enforces upon learners and prevents them from exploring, e.g. serendipitous learning), we need humbler claims that incorporate the ethics of the data, and we need richer data to inform/transgress/challenge algorithmic reductions and their power in the age of algorithmic education. We need to consider whose narratives get told through LA and all the other learning narratives that don’t get told (or seen) to move beyond singular, simplistic, reductionist accounts that LA falls into that we might empower learner agency to create agendas where our students use LA on their own terms and in their own ways – openly, knowingly and judiciously – to see LA as merely one glimpse of a quantified digital self, amongst much richer ecological selves. As Prinsloo (2017 online) captures it so potently, we need a ‘responsible learning analytics’ that ‘is found in the nexus between their stories and ours. We cannot afford to ignore the fact that it is their data, their aspirations, their learning journeys and that our data collection, analysis and use may not tell the whole story’ – so let’s be measured.

A few months later, as Dimitri recovers, he receives an email from his lecturer, alerting him to an error in the LMS involving a typo by his tutor and that his grade is in fact 94% and not 49%. This was instigated as a follow-up by his dad Paulo who was puzzled by his son’s mark as he knew how hard he had studied. Sometimes it really is useful to be privy to certain information!

At the end of the year, both Diana and Dimitri are delighted to be invited to be student representative voices on the Learning Analytics Evaluation Committee that meets monthly at their university.