1 Introduction

“History of Western Architecture” is an important architecture design course that introduces the history and advancements of architecture design concepts, contemporary architectural theories and representative buildings in western countries [52]. In the course, architectural knowledge is instructed and evaluated using visualized materials to cover the factors that influenced the development of architecture in terms of nature, society, politics, culture, science and technology and economy. Moreover, students are scheduled to observe some real buildings or models to learn the details of architecture design with specific features.

The purpose of the architecture course is to equip the students with the capability to absorb the architectural knowledge and spirit from historical theories and different genres of buildings, and to understand what the architecture from different periods of time represents [52]. In this way, the students can potentially incorporate those features in the creation of their own designs. Ideally, architecture courses are taught in a way so that the students can visit and experience world-renowned examples of architecture in person, while the teacher provides one-on-one sessions to help students develop critical thinking. However, architecture design courses can hardly be expected to create such a learning environment. Furthermore, it is a fact that architecture classes often fail to facilitate active thinking and information researching abilities, as students tend to just try to memorize the architectural knowledge and features for the class.

With the advances in technology and the extensive adoption of mobile devices, mobile technology has become an indispensable part of our life. More importantly, it enables learners to step outdoors and into real-life learning contexts. Researchers have suggested that learning is much more meaningful in real-world contexts than in the classroom [6, 32, 68]. The advancement of Augmented Reality (AR), which can integrate real-world objects with digital information via mobile devices or particular equipment, further provides a novel form of application that has attained widespread attention and interest.

On the other hand, Burleson [7] indicated that awareness and reflective technologies could be instrumental in developing students’ meta-cognitive ability and enhancing their learning achievement, creativity, and self-actualization. Providing learners with clear feedback can especially help their self-reflection, improve their learning performance, and solidify their understanding [37]. Panjaburee et al. [46] further proposed a learning diagnostic system which offered learners immediate feedback during the learning process and found that the approach could bring better learning results. Apparently, immediate feedback given to learners is of great importance [41]. In particular, several studies have reported that, during the learning process, formative assessment is critical to significantly increasing learners’ learning achievement [2, 4, 20, 47]. Bell and Cowie [2] pointed out that formative assessment could assist teachers to understand learners’ progress, modify instruction and give timely feedback, as well as simultaneously improving learners’ concentration and learning effectiveness [2, 44, 51]. The feedback contributes to meaningful learning during the learning process, which is why scholars define learning feedback as an indispensable part of formative assessment.

To enhance learners’ performance in the History of Western Architecture course, this study proposes an AR learning system based on a formative assessment strategy. The assessment method serves as a road map and an assisting tool which provides feedback to students about their learning and then prompts them to engage in self-reflection and more effective learning.

1.1 Research purpose

This study aimed to investigate the effect of applying formative assessment strategies that are known to be effective in mobile learning activities. To achieve this objective, an AR-based formative assessment interactive u-learning system was developed for use in a mobile learning activity. Such learning strategies have been reported as being effective in web-based learning environments [59, 60, 62]. Moreover, the learning achievements, motivation and cognitive load of the students who participated in the learning activity were analyzed to demonstrate the usefulness of this approach.

1.2 Research questions

In line with the aforementioned research purpose, comparisons of the learning outcomes of the students who learned using the AR-based u-learning system with the formative assessment mechanism and the conventional AR-based u-learning system were made by investigating the research questions listed as follows:

  1. (1)

    In the different learning modes (i.e., the AR-based u-learning system with the formative assessment mechanism and the conventional AR-based u-learning system), does formative assessment in an AR environment have an effect on the learning outcomes?

  2. (2)

    In the different learning modes, do the influences on the students’ cognitive load differ?

  3. (3)

    In the different learning modes, does formative assessment in an AR environment have an effect on learning motivation?

2 Literature review

2.1 Context-aware ubiquitous learning and augmented reality

Derry and Lajoie [15] employed the concept of situated learning, indicating that knowledge is constructed through interactions between the learner and the real-world learning contexts. Under this assumption, learning is ineffective if it fails to be combined with real-life situations. Therefore, this teaching style emphasizes that learners be involved with real learning contexts, so that abstract knowledge can be related to the real-world scenarios to help strengthen the learners’ knowledge building. Hwang and Chen [30] indicated that ubiquitous learning happens without the restrictions of time and space. The adoption of mobile devices or other technologies is not, however, requisite in such a description. Mobile learning refers to learning practices in which students attain anytime and anywhere learning via mobile technology and wireless networking. The major characteristics of ubiquitous learning are permanency, accessibility, immediacy, interactive situating of instructional activities, and adaptability [1, 9, 27, 39, 58].

The notation of context awareness was proposed by Hwang [27], who aimed to provide users with the information they needed with the aid of sensing elements and wireless networking, based on users’ needs and their physical locations. In recent years, a variety of context-aware technologies have been developed, such as Radio Frequency Identification (RFID), Global Positioning System (GPS), Quick Response (QR) codes and augmented reality (AR). These technologies have received increasing attention in the field of mobile learning. To access more learning content for students via mobile devices, environmental sensing and location recognizing technologies are required to offer learning content to learners by location recognition. The context-aware ubiquitous learning approach was developed by integrating wireless, mobile, and context-awareness technologies to detect the situation of learners in the real world and provide adaptive support or guidance accordingly [8, 9, 12, 33].

Researchers have further indicated that AR technology, a context-aware technology that is able to present the integrated real-world objects and corresponding digital information on mobile devices, can draw students’ attention to the learning materials on mobile devices and facilitate their active learning experiences when compared with traditional lecture courses [34, 40, 67]. Moreover, AR technologies have been used in guiding tours and artwork teaching to enhance students’ learning motivation, concentration and confidence levels in learning the subject [16, 24, 69]. Nevertheless, using AR technology does not necessarily translate into a positive effect on students. Previous studies have pointed out that without proper tools or strategies, students’ learning effects may not increase to the extent that reaches researchers’ and teachers’ expectations, even if new technologies like mobile learning systems or Augmented Reality techniques are applied [10, 12, 13, 58]. Therefore, assisting learners to achieve the greatest possible learning effectiveness through effective instructional strategies implemented in the AR learning environment is essential if the learners are to understand important issues [16,17,18, 31].

2.2 Immediate feedback and online formative assessment

Many studies have indicated that a good formative assessment design is a key factor in understanding students’ learning achievement when teachers are conducting their teaching activities [11, 19, 21, 45, 49]. In past decades, formative assessment has become a popular issue in classroom assessment, and many studies have contributed opinions and viewpoints for formative assessment. For instance, Hooshyar et al. [22] argued that formative assessment is a systematic process of constantly collecting learning evidence, which can be used to check students’ progress, adjust their learning pace, and thus help them reach the learning goals and objectives. Ng [44] indicated that formative assessment is the process of evaluating the interactions between the teacher and the students, helping the teacher give immediate feedback to the students while teaching.

In addition, several studies have further indicated the importance of giving students prompt feedback [5, 40, 57, 60, 65]. In particular, the immediate feedback provided by monitoring the learning progress for a task enables students to promote the development of self-regulated learning skills and contributes to deeper learning engagement [28, 42, 63, 70]. For example, van der Kleij et al. [56] claimed that immediate feedback in user interface design provides students with more effective feedback for improving learning effectiveness. Wu et al. [65] combined concept maps with an instant feedback mechanism and implemented a ICMLS system that enabled students to not only receive immediate feedback on learning tasks, but also to enhance their learning performance. Moreover, Sun et al. [53] also emphasized that immediate feedback is beneficial for learning motivation and achievement. The positive effects brought about by the feedback speaks of why such assessment is helpful for students’ learning [43, 44, 48]. Hence, scholars define feedback as an essential component of formative assessment due to its benefits to students’ learning.

Formative assessment plays an important role in conventional teaching environments and digital learning environments alike. Many studies have indicated that students’ learning achievement can be raised significantly when formative assessment-based teaching strategies are included in digital learning environments [20, 22, 29, 36, 50]. Hwang and Chang [29] developed the Formative Assessment-based Mobile Learning (FAML) guiding system, and integrated it into an elementary local culture course. FAML allowed students to perform Web-based formative assessment online, and adjusted the ongoing instruction and learning. The study found that the guiding mechanism could help improve students’ learning effectiveness. Lin and Lai [38] developed an online formative assessment-based virtual learning system and applied it to an electronic commerce course. They indicated that the system not only improved the learners’ motivation, but also facilitated their further learning.

To guide students to deal with learning tasks on their own, Wang [61] developed a formative assessment module which was part of the Web-based Assessment and Test Analysis (GPAM-WATA) system. In the GPAM-WATA system, several important Web-based formative assessments were included, such as “repeated answering”, “no answer-providing” and “immediate feedback provision”. The study showed that the GPAM-WATA system provided good support to the online learners by helping them recognize the problems, and motivating them to clarify their misconceptions and master the course. Several later studies have further pointed out that similar strategies have strong potential to help students learn better on field trips, such as mobile learning or context-aware ubiquitous learning activities [10, 22, 41]. Therefore, in this study, the formative assessment strategy with “repeated answering”, “no answer-providing” and “immediate feedback provision” was adopted to support students to learn in AR-based ubiquitous learning activities.

3 AR-based interactive ubiquitous learning system with the formative assessment strategy

In this study, an AR-based ubiquitous learning system based on a formative assessment strategy has been developed for conducting context-aware u-learning activities for an architecture course. Figure 1 shows the learning scenario of using the AR-based learning system in the field. The students interact with the learning system via a tablet computer with wireless networks.

Fig. 1
figure 1

A student uses the AR-based u-learning system in a real-world context

Full size image

During the in-field learning process, students learn with the formative assessment mechanism, which guides them to answer a series of questions for each learning target based on their observations following the guidance provided by the learning system. For example, when students are guided to observe a building, there could be several learning targets on the building, such as the windows, statues at the entrance, roof, walls and pillars (see Fig. 2). When observing each learning target, the students need to answer a series of questions which guide them to make detailed observations. For instance, they may be asked, “Which of the following statements regarding the ‘South transept rose window of Chartres Cathedral’ is false?” with options provided such as “The window includes 24 panes of exquisitely detailed scenes”, “Mary is known as the mystic rose or the rose without thorns, and is the figure implied in rose windows”, etc. During the learning process, the learning status of individual students, which is represented as a summative score, is shown on the lower part of the tablet computer screen. The score is computed by taking their efficiency of completing the learning tasks into account.

Fig. 2
figure 2

Interface of the AR-based learning system

Full size image

The flow of the AR-based formative assessment mechanism is given in Fig. 3. First, students are asked to use the AR program via the mobile device to identify a specific real-world learning target and then to observe that target with access to relevant digital learning material. They are also guided to observe the details of the target, based on a series of related questions raised by the learning system. There are three key features in this formative assessment mechanism, namely “repeated answering”, “no answer-providing” and “immediate feedback”. “Repeated answering” implies that the students are asked to answer questions randomly selected from an online test item bank. It is possible that a question is asked repeatedly unless the student correctly and continuously answers the same question three times. “No answer-providing” means that the learning system does not provide correct answers when students submit answers; instead, some hints (“immediate feedback”) are provided to encourage them to find the answers or to confirm their answers on their own.

Fig. 3
figure 3

Flow of the AR-based formative assessment interactive u-learning process

Full size image

4 Experiment design

4.1 Participants

A total of 39 college students were randomly assigned to the two groups in the experiment, with 19 assigned to the experimental group, who were taught with the AR-based formative assessment interactive u-learning mode, while the remaining 20 made up the control group and were taught with the AR-based u-learning mode. During the research process, the participants’ privacy was protected by hiding their personal information. Moreover, they were informed that their participation was voluntary and that they could withdraw from the study at any time.

4.2 Research tools

The measuring tools in this study included the cognitive load measure, learning achievement tests, and the questionnaire for measuring the students’ learning motivation.

The pre- and post-tests were developed to evaluate the learning effectiveness of the students. The test sheets were developed by two experienced teachers for expert validity. The pre-test aimed to evaluate the students’ prior knowledge of architecture history and design principles before participating in the learning activity. The post-test aimed to evaluate the students’ knowledge on design concepts of Gothic architecture. The pre-test and post-test both contained 25 multiple-choice items, with a perfect score of 100.

The motivation questionnaire was developed by referring to the Motivated Strategies for Learning Questionnaire (MSLQ) developed by Pintrich and DeGroot [48]. It consisted of 19 items in a five-point Likert scale, including items such as “I am interested in the learning content of this course”, “I believe that I will have better learning achievement than my classmates” and “I think the learning content of this course is important”. The Cronbach’s alpha value of the questionnaire was 0.77.

The cognitive load survey developed by Hwang et al. [35] based on the cognitive load measures proposed by Sweller et al. [55] was adopted for measuring the cognitive load of individual students. In this study, two cognitive load dimensions were measured, mental load and mental effort. Mental load is concerned with the cognitive load caused by the amount of information presented to the students simultaneously, while mental effort is related to the cognitive load caused by the way the learning content is organized or by the adopted learning strategy [55, 66]. It consists of eight items on a seven-point Likert rating scheme, five for mental load and three for mental effort, where “7” represented “strongly agree” and “1” represented “strongly disagree”. For the mental effort and the mental load dimensions, the Cronbach’s α values were 0.86 and 0.85, respectively. These values show good reliability in internal consistency.

4.3 Experimental procedure

Before the experiment, the two groups of students took a one-semester course on the history of western architecture. Figure 4 shows the flow chart of the experiment. At the beginning of the learning activity, an orientation was given to introduce the learning environment and the learning tasks. Moreover, the students took the pre-test and filled out the pre-questionnaire. During the learning activity, the students in the experimental group were instructed to use the AR-based u-learning approach with the formative assessment mechanism to accomplish the learning goals when interacting with the website; on the other hand, those in the control group used the conventional AR-based u-learning without participating in the formative assessment strategy. Both versions of the activity contained the same background story, learning missions and learning content. The time for the students to complete their learning missions was 90 min. After the learning activity, the students took the post-test and filled out the post-questionnaire to measure their learning achievements, attributes, cognitive load, self-efficacy and any change in their learning motivation. Finally, statistical analysis was performed on the generated quantification.

Fig. 4
figure 4

Experimental design

Full size image

4.4 Data analysis

The Statistical Package for Social Science (SPSS) was used for the data analysis in this study, including the computation of the t test and one-way ANCOVA results for analyzing the pre-test and post-test scores, and the learning motivation ratings. The independent variable was the two learning strategies (i.e., the AR-based u-learning approach with the formative assessment mechanism and the conventional AR-based u-learning approach). The dependent variables were the students’ learning achievement, cognitive load, and learning motivation. The alpha was established a priori at the 0.05 level.

5 Results

5.1 Analysis of learning achievement

To evaluate the effectiveness of the proposed approach, an experiment was conducted on a college architecture course in Taiwan, as shown in Table 1. Before performing the experiment, the results showed that there was no significant difference between the two groups (t = 0.60, p > 0.05).

Table 1 Descriptive data and one-way ANCOVA of the pre-test/post-test results
Full size table

After the learning activity, the results of one-way ANCOVA analysis using the pre-test scores as the covariate and the post-test scores of learning achievement as the dependent variable are shown in Table 1 and indicate that the students in the experimental group had better learning achievement than those in the control group (t = 2.29, p = 0.028 < 0.05), with a medium to large effect size (Cohen’s d = 0.73). This implies that the augmented reality interactive u-learning system based on a formative assessment strategy mode benefited the students more than the conventional augmented reality-based u-learning mode.

5.2 Analysis of cognitive load

In this study, two cognitive load dimensions (i.e., mental load and mental effort) were measured. Table 2 illustrates the t test result of the cognitive load ratings of the two groups, showing that there was no significant different in mental load (t = − 1.22, p > 0.05). This finding is consistent with previous studies as the mental loads of the two groups of students did not significantly differ because the students in both groups learned with the same learning subject materials and tasks [14, 51, 60]. On the contrary, the results indicate that the experimental group exerted significantly greater mental effort than the control group with t = 2.60 (p = 0.01 < 0.05) with a large effect size of d = 0.83. This result is consistent with previous studies, as moderate mental effort is necessary for students to maintain good learning performance, which could be achieved by using effective learning strategies [24, 26].

Table 2 The t test results of mental load and mental effort for the two groups
Full size table

5.3 Analysis of learning motivation

Table 3 shows the t test results of the learning motivation of the two groups. This results showed a significant difference between the learning motivation pre/post-questionnaire ratings of the experimental group (t = − 2.93, p < 0.01); moreover, the difference between the learning motivation pre/post-questionnaire ratings of the control group was also significant (t = − 2.63, p < 0.05). The experimental results showed that the students in both groups who learned with the AR-based learning system were able to improve their learning motivations, no matter whether the formative assessment mechanism was employed in the system or not. Accordingly, this implies the potential for augmented reality (AR)—based learning to motivate learning due to its enjoyable interaction nature and experience [12, 17, 34, 41, 67], as well as the positive effect of the formative assessment strategy which enables students to engage in their learning.

Table 3 Paired t test results of the learning motivation pre/post-questionnaire scores of the two groups
Full size table

6 Discussion and conclusions

In this study, an AR-based formative assessment interactive u-learning system was proposed for conducting a u-learning activity in architecture courses. An experiment was conducted to evaluate the performance of the proposed approach in a university architecture course. The experimental results showed that, in comparison with the conventional AR-based u-learning system (w/o formative assessment strategies), the proposed approach significantly improved the students’ learning achievements. This result indicated that the AR-based formative assessment learning strategy was helpful for the integration and internalization of knowledge; meanwhile, it facilitated students’ higher learning effectiveness. Several previous studies have shown that using formative assessment provides opportunities for students to take increasing responsibility for their own learning and to actively build their understanding, while also helping them learn to respond to feedback and provide feedback to others [3, 20, 22, 28, 44, 50]. Crisp et al. [14] also found that formative feedback facilitates students’ engagement, and improves their learning development and achievement [20].

On the other hand, it was found that the students who learned with the AR-based formative assessment strategy exerted greater mental effort, meaning that those strategies might give students higher mental pressure during the learning process. Cognitive load represents the load that performing a particular task imposes on the learner’s cognitive system [23, 54]. Mental load, on the other hand, is the load that forms while the learner is engaged in performing high-level thinking and challenging tasks or facing a large amount of learning content, while mental effort refers to the learning strategies adopted or how the learning content is organized and presented [55, 64]. From the experimental results, no significant difference was found between the mental loads of the two groups, which is reasonable since they learned with the same learning tasks and content. In the meantime, the experimental group scored significantly higher than the control group on mental effort (the mean of the mental effort was 4.93 for the experimental group and 4.23 for the control group), implying that the formative assessment approach engaged the experimental group students in learning tasks with reasonable challenges, which could be the reason why they had better learning performance, as indicated by several previous studies [25, 28].

In terms of learning motivation, this research showed that the learners could be attracted and motivated to learn by both AR-based learning modes; meanwhile, it was shown that effective formative assessment feedback could promote the students’ motivation to engage in self-regulatory processes, implying that they were likely to use the feedback in revising their learning activities to solve the tasks efficiently, and also perceived it to be useful. This finding conforms to that reported by Hwang and Chang [29], who indicated that the lead-in of learning guiding strategies could encourage students to engage more in the learning tasks. Several previous studies have also reported similar findings of providing learning guidance to students in technology-enhanced learning environments [45, 57]. In addition, the fact that the students became more engaged and inclined toward self-directed learning in the formative assessment activity could also be the reason why their learning achievements were significantly improved. That is, by better facilitating learning motivation among learners, students who participated in the formative assessment strategy outperformed those who did not.

Although the proposed formative assessment mobile learning approach is effective in helping students improve their learning performance in the architecture design course, there are still difficulties in the validity of the research design for AR-based learning environments. First, the experiment was conducted in an architecture design activity. To apply the approach to other applications, some modifications to the approach might be needed. Second, the sample size in this study was not large owing to the teaching reality in the selected university. To infer the findings of this study to courses with a larger number of students, further studies might be needed.

Despite these limitations, it is not difficult to apply the present approach to other design courses which need to situate students in in-field observations. The teachers only need to replace the learning materials and test items in the system with those for the new applications. Currently, we are working on extending this study by including a personalized feedback mechanism to provide additional avenues for brainstorming and in-depth discussion. These improvements should create further opportunities for students to practice and sharpen their own critical thinking skills and assist them in improving their learning and performance; meanwhile, it is worth investigating an interactive process involving both students and teachers, and students’ learning behaviors in the future.