1 Introduction

The fast pace of work nowadays intensifies the pressure on citizens to make quick, essentially instantaneous decisions without giving much consideration to the consequences, and sometimes with ethical dilemmas or integrity matters involved. A pertinent issue in tertiary education in the twenty-first Century is academic dishonesty among students (Jones 2011), and one factor that leads to students’ misconduct is the prevalence of information technology (IT) whereby students may easily perform unethical or with low-integrity (Manly et al. 2015). To maintain students’ academic integrity and ethics (AIE) in classrooms, lessons, tutorials and learning activities in helping them understand the meaning and importance of behaving ethically and with integrity are common practices in many tertiary institutions. Yet, despite these efforts, even with enforcement of students’ declarations and severe penalties for misconduct, cases of plagiarism, disregard for intellectual property, data fabrication and breaching of rules still arise with alarming regularity (Jones 2011; Simola 2017). It seems the problem lies in that students do not digest the notion of AIE but learn for some other reasons (e.g., required by teachers, etc.). Hence, we introduce the element of gamification in education to improve motivation and student engagement (Domínguez et al. 2013). By gamifying the learning experience with augmented reality, students can turn the abstract conceptual knowledge of AIE into real understanding. Meanwhile, IT is employed through the project to combat the issue worsened by itself.

To offer actionable intelligence for both the teachers and students, there is a need to conduct learning analytics (LA). According to The Society for Learning Analytics Research (SoLAR, www.solaresearch.org/), learning analytics is defined as “the measurement, collection, analysis and reporting of data about learners and their contexts, for purposes of understanding and optimizing learning and the environments in which it occurs.” It is an area that is evolving and continuing to gain traction within the higher education sector (Johnson et al. 2015) and it is envisioned as an effective, efficient way to assess student responses, provide immediate feedback, and make adjustments in content delivery and format. Those invested in the field of LA see its potential to foster personalized learning environments that adapt to the learning behaviors of students (Johnson et al. 2013). In the present project, the use of mobile technologies for blended learning helps to provide large quantities of learning data which can be reviewed and analyzed by automated means, thus providing the prerequisites for learning analytics.

According to Papamitsiou and Economides (2014), integration and meaningful triangulation of different data sources can improve the accuracy of a learner profile which leads to evidence-based improvement and adaptation of learning content. In this project, as students engage in the learning activities and share their experiences, views and opinions, large amount and different types of data (e.g. from clickstream, pre-/post-discussions, and questionnaire) can be captured. The sources range from physical worksheets to online activities through the AR APP (called Mobxz), as well as the learning management systems (LMSs) in use—Blackboard or Moodle.

2 Methodology

2.1 Augmented Reality (AR)

Augmented Reality (AR) is an emerging, state-of-the-art technology characterized by incorporating multi-media information such as virtual images, graphics, videos and audio clips into the physical world (Yuan et al. 2008; Johnson et al. 2016). It aligns well with the concept of situated learning and the process of contextualization of learning enhances new knowledge application (Johnson et al. 2011, 2016). It is evident that AR can support learners in visualizing abstract concepts, e.g. in science disciplines where the learners’ senses of presence, immediacy, and immersion in 3D learning content, etc. can be actualized (Wu et al. 2013). In order to motivate students to learn and stimulate them to think about ethical scenarios in real-world settings, this project thus adopts AR, coupled with mobile technologies to bring scenarios of AIE to real-life situations for students. We have created mobile learning paths on campus called “Trails of Ethics and Integrity (TIEs)” whereby students are immersed in various ethical dilemma situations via AR applications; students produce different responses and then play out the consequences of their ethical decisions. There are also subject-specific or functional TIEs in which students face cases of ethical dilemmas specific to their disciplines and professions.

2.2 Trails of Integrity and Ethics (TIEs)

This project uses sharply focused scenarios on ethical dilemmas coupled with AR and mobile technologies to design the TIEs where scenarios are triggered in different locations and students are challenged to review the arguments presented and reflect on their ethical choices. Contents of all scenarios are designed based on the “3C model” as advised by Kuhlmann (2009), i.e. each decision making scenario has to include a Challenge (for challenging student’s understanding or assumptions), Choices (for students to make decisions), and Consequences (for providing feedback and encouraging student’s self-reflection).

All TIEs are deployed using a mobile application Mobxz (renamed as AR Learn in late 2016 with a new user interface) which is specifically designed for learning trails deployment and is capable of supporting both iOS and Android smart devices. Students bring their mobile devices to walk through a trail and activate the learning activities within each checkpoint scenario through marker-based (e.g. QR code scanning, image recognition) and marker-less (e.g. GPS location mapping) AR technologies (Pence 2010) that are built-in in this mobile app.

While one general and five subject-specific/functional TIEs have been developed and in operation since the start of the project in late 2014, this paper reports the findings from the learning analytics on the TIE-General supplemented with results gathered from a functional TIE called the TIE-Hall Tutor.

2.2.1 TIE-General

The TIE-General is the first trail established by the project’s host institution in early 2015; it covers ethical scenarios relevant to students in all disciplines. It is a short trail consists of four checkpoint scenarios, namely (1) plagiarism, (2) fabrication of data, (3) ethical use of library resources, and (4) citation and common knowledge. Each checkpoint scenario relates to a specific physical location within the campuses of the four participating universities in Hong Kong. This trail is normally used as an outdoor teaching and learning activity in classes where the issues of AIE will be discussed. With the project team gaining experience as the project progresses, in the 2016/17 academic year, the TIE-General has also been piloted as one of the induction activities for undergraduate freshmen at new students orientation.

2.2.2 TIE-Hall Tutor (TIE-HT)

To enhance residential education, a resident master, with the help of previous hall tutors (HTs), used role-playing games (RPGs) as the overarching framework to embed situated learning pedagogies in the design of TIE-HT. As hall tutors, they are expected to be role models, hold residents accountable to Hall Rules and Regulations and report any irregularities of residents’ behaviours. They also act as peer counsellor, floor manager and team builder. The role-playing cases, designed based on the 3C Model (Challenge, Choices and Consequences model) (Kuhlmann 2009), reverberated deeply with these ‘hall leaders’. The TIE-HT is part of a training programme to strengthen the conceptual understanding of ethics and integrity for newly appointed HTs. This student-led design of mobile learning trail of integrity and ethics (TIE) is developed and executed within a simulated hall environment to enable the new HTs to have a better insight of carrying out their role ethically and with integrity towards their peer residents.

2.3 Data Collection and Analysis

2.3.1 Population

A total of 658 undergraduate students from all disciplines from four Hong Kong institutions participated in exploring the TIE-General during the period from October 2015 to December 2016 and a total of 46 undergraduate student Hall Tutors from one of these four institutions participated TIE-HT in either August 2015 or August 2016.

Ethics clearance for data collection and analysis was obtained from the host institution of this project.

2.3.2 Learning Analytics

The usual condition was that the individual instructors would provide some explanation on AIE concepts, according to the subject disciplines of the respective classes. The pre-trail data were normally collected from individual students before or during the class, so students in the different classes would be exposed to varying explanations of AIE issues given by their respective class teachers. Then the students would be set onto the TIE-General to complete the activities. Instead of a class, TIE-General was also used for undergraduate freshmen to help them get familiar with their new universities and campuses as one of the induction activities. The clickstream data were gathered in the same manner as described earlier.

2.3.3 Clickstream Data

Students accessed the learning contents for each checkpoint scenario through their mobile devices. The choices that students made and the time they spent on each part of the scenario were recorded. Clickstream Java Script snippets (timestamp) were inserted within each scenario for data logging and the timestamp logs were automatically gleaned in the AR APP. Figure 1 shows the timestamp log that are recorded within each checkpoint scenario. As the content of each checkpoint scenario is designed based on the 3C model, there are total numbers of 4 timestamps, viz: 3 timestamps recording at the beginning of the Challenge, Choice and Consequence session and 1 timestamp at the end of Consequence session. The time taken for is compiled by calculating the differences in timestamp logs. The duration of time represents the degree of difficulty for students to understand the challenges, make the choices and reflect the consequences. Moreover, the sequence of choices made by the users is shown from the clickstream data, some of which were eliminated and missed if the Wi-Fi connection was lost and the mobile devices were shared among others participants during the trails.

Fig. 1
figure 1

An illustration of the design of a learning trail with the timestamp log indicated for calculating the time taken for challenge, choice and consequence

Full size image

2.3.4 Text Mining of Pre-/Post- Commentaries/Discussions

Students’ reflective responses on the understanding of the concepts of AIE or issues related to AIE before and after the trail become input for text mining analytics. Pre- and post- trail discussion data are collected from a variety of methods and means ranging from the AR APP to Learning Management System (LMS) discussion boards to hardcopy worksheets. This approach is to maximize the number of responses in light of the practicalities of collecting data from different groups of students. The data were collected with a voluntary survey before or after the trails which may lead to the loss of data. Students are allowed to write in either Chinese or English or a mix of these two languages. The data are then imported into a Bilingual Text Mining platform (learning analytic algorithm) which is developed by one of the partner institutions (Li et al. 2015). A list of AIE keywords (mixed Chinese and English keywords) per trail is provided to the text mining platform for analysis. The system searched for predefined keywords and key phrases within the responses and the frequency of these keywords/key phrases was then generated. The generated frequency indicates the understanding of different concepts corresponding to academic integrity and ethics.

2.3.5 User Experience Survey (Quantitative and Qualitative)

A specifically designed user experience survey was used to collect perception data from students with regard to the usefulness of the TIEs on the learning of the abstract concepts of AIE. The survey data was collected via the AR APP, after the students had completed the trail. The survey questions solicited responses in a 5 point Likert Scale, (5 = strongly agree, 1 = strongly disagree) so quantitative data collected could be presented in a chart. Qualitative feedback was also collected together electronically with the user experience survey.

3 Findings and Observations

3.1 TIE-General

3.1.1 Clickstream (Time of Task and Decision on Ethical Choice)

As students engaged in the learning activities on the TIE-General, all clicks and button-press events were recorded. Two sets of results, time on tasks and decisions on ethical choices, were extracted from the clickstream data and analyzed. By tracking the time spent on making decision, it can reflect the difficulty of taking ethical action. Students in general spent more time to understand the challenge of the scenario than on deciding on the choices and learning the consequences, except for the “ethical use of library resources” scenario (Fig. 2). Apart from text and graphics in all scenarios, a video (¬1.5 min) was included in the consequences of two scenarios—“citation and common knowledge” and “data falsification”. It provides simulation of scenario facilitating students to imagine the real-life cases happened to them. However, the average time spent in the consequence part for these two scenarios were only 33.01 and 28.59 s respectively which is way shorter than the video duration. It is not difficult to deduce that students did not go through all the content materials in the consequence part.

Fig. 2
figure 2

Time on task based on the 3C model, i.e. challenge, choice and consequence, for each checkpoint scenario in TIE-General

Full size image

Figure 3 shows students’ decisions on ethical choices in each checkpoint scenario. As the scenarios were designed in such a way that there is no absolute right or wrong answers, but as dilemma for students to make a decision on an ethical choice, a split is expected in the ethical decision. The corresponding choices are listed in Table 1. The analysis show that except for the scenario on “citation and common knowledge” where over 75% of the students had selected the same choice in their first attempt, other scenarios had around 50% of students selecting the same choices in their first attempt. This result tallies with that for the time on task with students only spending 13.03 s on deciding a choice in the “citation and common knowledge” scenario. It implies majority of students can converge on ethical choice of making proper citation while students show split inclination in other scenarios.

Fig. 3
figure 3

Decision on ethical choices of the four checkpoint scenarios in TIE-General

Full size image
Table 1 Summary of options corresponding to different scenarios
Full size table

3.1.2 Text Mining

In order to ascertain the effectiveness of the TIE-General in increasing students’ understanding of AIE concepts, bilingual (Chinese and English) text mining based on students’ written commentaries and discussion on the understanding of AIE was conducted. The pre-trail question was “what is your understanding of Academic Integrity and Ethics?” while the post-trail question was “what have you learnt about Academic Integrity from the learning trail?”. There were twelve predefined keywords/key phrases which were used for text mining for the TIE-General. In Fig. 4, the frequency of some keywords appearing in students’ post-trail responses increased, e.g. “academic integrity/honest/honesty” (12.2% increment), “cheat/cheating” (39.0% increment), “cite/citing/citation” (53.8% increment), “respect” (14.9% increment), and “responsible/considerate” (41.5% increment). This reflects students’ emphasis in the types of vocabulary related to the checkpoint scenarios, particularly prominent is in the surging number of occurrence for keywords “cite/citing/citation” which are related to the “citation and common knowledge” scenario. With respect to Ethics, there is a slight increase in the keywords “respect” and “responsible/considerate” showing a shift slightly to concrete words that are related to the scenarios. On the other hand, the number of occurrences for the words “plagiarism/plagiarize” decreases in the post-trail responses.

Fig. 4
figure 4

A comparison of students’ pre- and post- discussion/commentaries before and after the learning trail and intervention

Full size image

3.1.3 Quantitative and Qualitative Results From User Experience Survey

Although not all students had completed the survey, the response rate was around 88%, ensuring that the results were representative of the students involved in TIE-General.

Quantitative results of the survey (Fig. 5) showed that students like the learning trail (Q6, 75% agree/strongly agree, Fig. 5) and 76% of students found that the learning trail using the mobile app had made learning AIE more interesting (Q3). The majority of the students (>71%) found the mobile app easy to use and is clear and understandable (Q1 and Q2). However, only around 50% of students agree/strongly agree that the Wi-Fi connection is stable.

Fig. 5
figure 5

Quantitative results of user experience survey

Full size image

Students also provided qualitative feedback on their learning experience after exploring the TIE-General, and these were classified in aspects of “Experience/Content” and “Technical”. In line with the quantitative results of the survey, major of students stated comments in the aspect of “Experience” and “Technical” such as easy to use, the learning trail experience engaging and interesting, and the content very useful. It implies our application can raise their awareness towards AIE. Through walking the trail physically, abstract ideas of AIE were introduced to students constructively with real-life examples. Students also commented on the instability of the Wi-Fi connection. It hindered the loading time of the scenario contents during the learning trail which reduced students’ attention while they were waiting for the consequence part in the trail. Some of the students suggested including more checkpoints and challenging options. Samples of direct quotes from students are shown in Table 2.

Table 2 Qualitative comments on the learning trail experience from student participants
Full size table

3.2 TIE-HT

In this detective/role-playing game-based trail, HTs were asked to discover and reflect on the cases within the storyline of finding the reasons why a student resident, named Desperate was hospitalised when the HT was not in the residence hall. They had to first select a method (out of four) to look for clues on the cause of the incident. The clickstream result shows that over half (57%) of the HTs chose to go directly to Desperate’s room first while only 9% of the HTs chose to look over Desperate’s Facebook account to find the clues (Fig. 6).

Fig. 6
figure 6

Hall Tutors’ first choice of methods to look for the reasons for the incident

Full size image

HTs also had to state their feelings and reflect on their roles and responsibilities as HT during and at the end of the TIE-HT. Figure 7 shows HTs’ immediate response to the questions in the scenario. The words “shocked”, “sorry”, “guilty”, “scared”, “anxious” were prominent in the word cloud (Fig. 7, left) indicating a higher frequency of these words in the HTs responses on immediate feelings. The HTs were shocked and felt guilty with what had happened to that student resident who was hospitalized due to the HT’s unawareness of a potential problem in advance. The results demonstrated the awareness of the consequences for not being taken immediate action for any instance and reminded them the ethical role as HTs such as team builder and counsellor to their peer residents (Fig. 7, right). In addition, the HTs revealed in the debriefing session that immersing themselves into the trail can reinforce them the importance of being a role model and upholding hall regulations. They also expressed that the trail is useful because it allowed them to place themselves into emergency situation and to acquire experience to solve the unexpected problems.

Fig. 7
figure 7

Word cloud on the answers to the questions in the scenario: (Left) What do you feel after listening to some residents’ comments or on seeing the scene? (Right) What role do you see yourself in the cited situation?

Full size image

4 Discussion

Mobile learning activities in this project are conducted in real-world environment as a form of situated learning to help students connect learning to real-life context. To evaluate the effectiveness of the TIEs, triangulation of different sets of data is adopted and these datasets include user experience survey, qualitative feedback, clickstream data, and text mining of pre-/post-trail discussion. All responses and data gathered are analysed to ascertain the effectiveness of these mobile learning trails in enhancing students’ awareness to AIE issues.

The number of research studies in the area of learning analytics using clickstream data collected from mobile devices is still limited (Chan et al. 2015) and single source of learning analytic is not sufficient enough to make a conclusive claim. Hence, by triangulating data collecting from user experience survey, qualitative feedback, clickstream data, and text mining of pre-/post-trail discussion with larger sample size, we could examine students’ interactions and exploration patterns within the AR learning activities using the tracked data. Although the preliminary data of this project shows feedback mainly in the view of user experience, analysis of the clickstream data collected suggested that the decision-making time of choosing ethical action in various dilemma situations can reinforce students’ reflection.

Analysis of students’ pre-/post-trail commentaries or reflective texts detected encouraging increase and emphasis in the types of vocabulary or concrete words related to trail scenarios. These findings indicated that situated learning using AR with mobile devices has been effective for students to learn abstract concepts of AIE.

Quantitative survey results and qualitative feedback allowed designs refinement to incorporate elements more conducive to improving student learning. We are confident that by deploying the TIEs using AR applications with mobile devices, students’ interests in learning about issues of AIE have been aroused.

Akin to other projects that incorporate innovative technologies into pedagogies to enhance student learning, our project has experienced variations and at times, surmounted difficulties in order to get the relevant data to ascertain the effectiveness of the project. Hence we tried to be flexible in adapting to the situations and environments to ensure that the data collected were reliable and consistent. The following paragraphs provide some insights into the matter.

Although 658 undergraduate students explored versions of TIE-General within the reporting period of October 2015 to December 2016, the 4 datasets might not have been collected for each of these participating students. Exploration of the TIE-General was conducted primarily as a teaching and learning activity for classes when the concepts of AIE were discussed.

This project uses mobile technologies coupled with AR, and students conduct the exercises on the move when they go around the respective campuses. As described previously, the quantitative survey data highlighted the importance of Wi-Fi stability for maintaining students’ interests to learn via this new system. More importantly, Wi-Fi stability and reliability also affected the response rate of the server as well as the accuracy of the clickstream and survey data collected. While we counted the individual students in each class that had gone on the trails as participants, the activities were usually conducted in groups of two to three students, hence the clickstream data collected were not for each individual student, but rather for each entire exploration of the TIE-General by one class. While for the freshmen induction session, the students did the trail exercises in groups of twos and threes. After that, students completed the quantitative survey and provided the post-trail reflections as the final activity on the trail, and they were expected to do these individually. Whether the TIE-General was administered as a teaching and learning activity for a class, or as an induction activity at new students orientation, participation was entirely voluntary. A student could go on the trail without engaging in any of the activities, nor completing the survey, not even providing their pre/post discussion/reflection on AIE matters. Nonetheless, from the volume of data that we had collected within the reporting period, we were assured that an average of 80% completion could be guaranteed for each run of TIE-General by a class/induction and the missing and incomplete data were taken out of the sample before any analysis could be done on them.

As described earlier, a specific TIE development is the training of resident hall tutors (HT) through their exploration of the TIE-HT. As these tutors are expected to be student leaders and role models to the hall residents, the TIE-HT is designed to help the HTs internalise ethical issues related to the residence hall and assume their role as HTs through reflections during and after the trail. This mobile learning trail on ethics and integrity had greatly facilitated HTs to immerse themselves in the given scenarios in informal settings. With the role-playing game (RPG) style incorporated into a set of sequenced stages of a story-line within the AR APP, HTs were highly motivated with a desire to complete the story and they unconsciously engaged themselves deeply through the excitement of the game. Interpretation of both qualitative and quantitative results together had illustrated that through the learning trail the HTs have recognized the importance of behaving as role models to their peer hall residents.

With the advent of IT and its adoption into pedagogies with the aim to enhance student learning, a major advantage not available previously is the accessibility of large amount of data, conveniently collected electronically while students are engaged in the teaching and learning, or even assessment, activities. This abundance of data, brought forth due to the use of IT in education, has facilitated the study of learning analytics, providing educators with means and new educational research methodologies to look at student learning from very different perspectives (Gibson and Ifenthaler 2016). Yet how to help students learn more effectively is not an exact science. The physical and cyber environments are interacting with human beings which are the most complicated element to assess. Hence to check for the effectiveness of any learning activity deploying IT, learning analytics data from different sources should be collected, analyzed and triangulated in order to draw a reliable conclusion. As explained in the previous sections, this is the approach we have adopted for our current project.

The positive learning outcome of the TIE-General and TIE-HT suggests that this form of situated learning using the latest AR and mobile technologies can be adopted and applied to a wider scope of residential education and the academic curriculum and co-curriculum.

This paper has reported on the findings based on learning analytics collected for two groups of students. The first group of some 600 students were undergraduates that had explored the TIE-General, while the other group of 46 new hall tutors had participated in training with the TIE-HT. Both of these TIEs are developed within the first two years of a three-year, Hong Kong-government-funded mobile AR research project. Drawing together quantitative and qualitative data obtained from mobile app clickstreams, text mining of pre- and post-trail commentary and user experience surveys, it has provided preliminary evidence of learning benefits to students.

Putting aside the novelty of this kind of “Pokemon Go like” learning approach which appealed to students in general, our study found that students were very engaged in the contextualised learning facilitated by mobile AR, as reflected both in their quantitative and qualitative responses to the user experience survey. Furthermore, a number of students commented specifically on the gamification aspects of the TIE-General and TIE-HT, which was specifically designed as a game. This is in line with the research literature on gamification and game-based learning when it is introduced to learning activities (Gibson and Webb 2015; Dicheva et al. 2015), and that the deployment of mobile AR helps to provide an engaging environment that motivates student learning (Lee 2012).

As far as students benefitting from this form of situated learning is concerned, we can draw some insights from the time students spent on the learning exercises on the trails with the choices they had made as recorded via the AR app. Having spent time on the challenges, reviewed the choices and made the decisions, then considered the consequences of their choices helped students linked abstract concepts to their real-life situations. The fact that variations in students’ ethical choices tallied with the time spent on tasks suggested increased motivation with this form of learning. The increase in the use of AIE related vocabulary in the post-trail reflective texts further signified that students had made progress in their understandings of these abstract concepts. The TIE-HT in particular, the situated role-playing game had helped transformed hall tutors’ understandings from rules to daily responsibilities by behaving as role models and team builders, they could gain peer acknowledgement and respect from the hall residents. Hence our findings support the literature in that learning in context is more effective when students can apply what they have learned in class to real-world environments (Dunleavy et al. 2009; Lave and Wenger 1991).

4.1 Limitations

The current study has a few limitations. This paper only reports the results of two trails developed from this project, TIE-General and TIE-HT in the first two year of development, from piloting to production. During this time, these two trails were developed iteratively with appropriate data collection mechanisms being implemented as the researchers gained experience in the trail design and in the assessment of the trails’ effects on student learning. While the results are encouraging, they can be strengthened by extending both the length of the project and the number of participants, and with more data from other subject specific TIEs collected and analysed for comparison. Despite the fact that we have upgraded to version 2 of the AR app for all the TIEs, with an enhanced server and better user interface, the motivational benefits of the trails seemed to have been compromised by limited Wi-Fi coverage at times. While it is envisaged that IT continues to advance, we have to continue to investigate for possible solutions.

4.2 Future Studies

The project has been given a one-year extension from June 2017. With the remaining duration of the project, the team plans to consolidate and expand on the TIEs for full implementation. With six TIEs piloted, three in ‘normal’ operation, and two more under development, it is logical to consolidate them such that the TIE-General can be used to introduce initial concepts and rules of AIE to freshmen at orientation when they first enter our institutions. In the future, more data, with larger student numbers, will be collected under this work-in-progress project to further elucidate the learning experience towards more concrete and personalised understanding of AIE. The subject-specific and functional TIEs, like the TIE-HT can then further re-inforce ethical behaviour and professional standards in the discipline classes or for targeted training. It is envisaged that in this way, the TIEs, will contribute even more substantially to students’ understandings of AIE by having the immersion in learning more comprehensive.

5 Conclusion

At full implementation, selected trails are expected to operate perpetually in the partner institutions. It is anticipated that in time, similar trails might be transplanted to, or set up in other tertiary institutions in Hong Kong. With enough evidence showing students can effectively learn the concepts of AIE using the TIEs, others in the education sector, such as secondary schools can be encouraged to get involved.