Keywords

1 Introduction

According to the report of ministry of education in Taiwan, learning Japanese as a second foreign language (JFL) has gradually increased because not only young people love Japanese popular cultures, but also Taiwan is one of the top consumers of Japanese pop culture, likely due to factors such as geographical proximity and the two country’s colonial history. It is worth mentioning, manga (i.e. Japanese comic) is form of these popular cultures that derive from Japan, especially among young adults. The hype and enthusiasm of manga is tremendous not just in Japan, but throughout the world [2]. Generally speaking, the adolescents’ consumption of manga cannot be regarded as mere entertainment, the notion of manga can provide an emotional intimacy, a visual representation of conversation, and a believable social context for students’ own identities as future workers [5, 17, 20]. In the past two decades, manga has begun to receive more scholarly attention from the standpoint of popular culture studies and literacy education [3, 13]. This is due to the fact that the graphic representation and ideologies contained in imported manga may have a more powerful cognitive effect on the group of youths than any formal educational process they undergo. For example, Khurana [9] considered manga as an effective tool for media literacy instruction. Ogawa [14] used educational manga in English language classrooms in a Japanese university to indicate learning and motivational benefits. After a postcourse survey, it revealed that positive overall responses from students with regard to both language and content learning. Adams [1] reported that high school students’ reading skills are influenced and heightened due to reading manga.

With the rapid development of technology, college and university students own laptops, and an increasing number of them are purchasing tablets, smartphones, and other handheld devices [19]. Meanwhile, since the state-of-the-art technology development makes students’ learning habits change, publishers are offering an increased number of textbooks in digital format, called Electronic-book (E-book ), to affect student’s adoption, along with the broad content of E-book, such as text, text-speech, music, sound, and animation [10]. Lin [11] investigated that the features of E-book enhance the motivation of students while reading foreign languages. Chou [6] attempted to find out Taiwanese undergraduate students’ E-book reading attitudes in both first language (L1—Mandarin) and second language (L2—English) and explored factors that may play a role in explaining students’ E-book reading attitude in L2. The results showed that the students demonstrated a slightly more positive e-book reading attitude in L2 than in L1 and indicated that if a reader has a positive reading attitude in an e-book environment when reading in his or her L1, this same attitude can be transferred to an L2 context. Yin et al. [21] analyzed on students’ learning behaviors comprise an important thrust in education research and used the E-book system BookLooper to record students’ learning behaviors in their daily academic life. The paper found that a number of learning behaviors have a significant relation with students’ test scores and showed that the number of pages read correlated with students’ scores. Shimada [18] proposed a method to analyze preview behaviors of students using a learning management system (LMS) and an E-book system. The paper collected a large number of operation logs from E-books to analyze the process of learning and reported that students who undertake good preview achieve better scores in quizzes.

In this pilot study, we developed a manga-based interactive E-book with the combination of the effectiveness and advantages of E-book and manga to construct a useful and practical tool for academic learning and promote learning motivation while learning Japanese. On the basis of the Dual-Coding Theory [15] and Cognitive Theory of Multimedia [12], this pilot study utilized an eye tracking as a tool to explore how the effect of the manga-based interactive E-book on students’ visual behaviors while reading Japanese language. An eye-tracking technology has been widely applied to studies on human visual behaviors and adopted eye trackers to study the process of reading [16]. There are over ten different types of eye movements, of which the most important ones are fixations, saccades, and smooth pursuit. When the eyes stop to focus it is called a fixation and the movements between these fixations are called saccades. Besides, areas of interests (AOIs) and gaze-related variables are often used when recording visual behavior with eye tracking technology. Researchers often define AOIs according to research questions and main areas of concern before the experiment to observe the relationship between eye movement variables and the main areas of interest of the experiment. The AOI definitions and the visual information of the material are interdependent: the AOIs of various materials and the research questions are defined accordingly. Based on the defined AOIs, the pilot study adopted two common eye movement variables, Number of Fixations (NOF) and Total Contact Time (TCT), used in visual behavior observations on each AOI. The two variables are common models for processing eye movement data [4, 8]. In addition, the third variable, called Number of Clicks, was adopted to count the number of annotations they were clicked to present the meanings of words or phrases in the sentence.

2 Research Questions

This pilot study intends to examine the relevance between different prior knowledge and students' visual attention, the experimental analysis is conducted to observe students’ visual patterns. In order to probe in-depth into how students learn Japanese language in the classroom with multimedia materials, we conducted a pilot study that examined students’ visual attention in terms of their eye-movement patterns as they were given a conversation presentation with manga-based interactive E-book design in the classroom. Therefore, this study proposes three research questions as follows:

  1. 1.

    How would university students distribute their visual attention to a manga-based interactive E-book with annotation text–picture formats?

  2. 2.

    How would university students make use of annotations to realize what the words and phrases mean in manga-based interactive E-book?

  3. 3.

    How do university students with different prior knowledge backgrounds differ in their outcome of reading comprehension?

3 Methodology

3.1 Participants

A total of 6 university students, with Japanese competence background, were selected from a mid-level Japanese course in Taiwan. According to the pretest scores, the 6 university students were categorized into high and low prior knowledge (PK) groups. All participants passed the eye tracking calibrations. They all consented and were included in an eye tracking experiment with a Japanese conversation used in manga-based interactive E-book reading comprehension .

3.2 Stimuli

The reading stimuli material was prepared a manga-based interactive E-book presentation on the topic of “daily conversation” for the pilot study. The manga-based interactive E-book consisted of 8 slides showing various textual annotation–picture conversation. The content and the design of the interactive E-book presentation were constructed by a content expert who is an Associated Professor of the relevant area and a Japanese education researcher.

3.3 Apparatus

An EyeNTNU-120 eye tracker with a sampling rate of 120 Hz (sampling 120 times per second) was used to track each participant’s eye movements while they read the material about the context of scenario. While collecting the movement data, a chin-rest was used in the experiment to reduce the occurrence of invalid or inaccurate data. The error rate of EyeNTNU-120s eye measurement is less than 0.3°, which is sufficient for this experiment. SPSS software were further utilized to store and analyze the eye movement data.

3.4 Data Collection

In order for the participants to be familiar with the software, the researcher gave them a short orientation and overview of the experiment. A paper-and-pencil pretest was used to measure each participant’s PK about Japanese competence. All participants received the same pretest and wrote down their answers on paper. The participants were asked to rest their chin on the chin rest while the EyeNTNU-120 eye tracker camera was directed to their right eye. Participants have gone through a nine-point calibration process to ensure data accuracy. After passing an eye tracking calibration, the experiment started by letting the participants view the arranged stimuli with graphical and textual information shown on a computer screen. No time limit was set for the task. Each participant’s eye movements were tracked and recorded by EyeNTNU-120 during the whole reading process. After finishing reading the stimuli, participants received a posttest immediately.

4 Results and Discussions

4.1 Results of Total Contact Time and Number of Fixations

Independent sample t tests were employed to examine whether there was any significant difference in the participants’ viewing behaviors as follows: (1) total contact time (TCT) and (2) number of fixations (NOF) within the graphic AOIs and with the text AOIs between the higher and lower PK groups. If a significant result was found, an effect size of Cohen’s d [7] was then further calculated. The results in Table 1 revealed that the high PK group has more total contact time on the graphic AOIs than the low PK group with a large effect size (t = 2.86, p = 0.046, d = −2.34) and the low PK group has more total contact time on the text and annotation AOIs than the high PK group with a large effect size (t = −2.79, p = 0.049, d = 2.28) and (t = −3.72, p = 0.021, d = 3.03), respectively. However, with respect to total contact time on the whole reading behaviors, no significant difference was found between the two PK groups. It showed that the two PK students paid the same attention to and put the same mental effort into reading the whole texts and graphics in the learning material. Meanwhile, Table 1 showed that the low PK group has more number of fixations on the text AOIs than the high PK group with a large effect size (t = −3.97, p = 0.017, d = 3.24). However, with respect to number of fixations on the graphic AOIs, no significant difference was found between the two PK groups.

Table 1 Eye tracking measures compared between the high and low groups
Full size table

4.2 Results of Number of Clicks on Annotations

The Table 2 showed that the familiarity with annotations is significantly different between the two PK groups with a large effect size (t = −4.75, p = 0.009, d = 3.88). It means that the low PK group clicked annotations for the need to realize the meaning of words or phrases in Japanese conversation. Due to this pilot study, the number of click on annotations was considered as an indicator to promote the outcome of reading comprehension in the posttest.

Table 2 Independent sample t-test of number of clicks on annotations
Full size table

4.3 Paired-Samples t-Test of Pretest and Posttest Scores

As shown in Table 3, through a paired-samples t-test, pretest had a significant difference from posttest for the low PK students. It means that the posttest score is higher than pretest for low PK students after reading manga-based interactive E-book . That is, a manga-based interactive E-book promoted a reading comprehension in this pilot study.

Table 3 Paired-samples t-test of pretest and posttest scores
Full size table

5 Conclusions

This pilot study was aimed to employ eye tracking technology to examine how learners with different levels of expertise engaged in reading a Japanese conversation in manga-based interactive E-book that was provided as graphic and textual annotation formats. The findings of this pilot study were as follows. First, the high PK students spent more time observing and investigating the graphic information than the low PK students. Conversely, the low PK students spent more time observing and investigating the textual annotation information than the high PK students. Second, the low PK students showed more NOF of texts that the high PK students and clicked many of textual annotations AOIs than the high PK students. It means that the low PK students spent much time trying to realize the meanings of the annotations what the words and phrases meant. Third, there was a significant difference in score between the pretest and the posttest in the low PK students. That is, the low PK students resulted in better learning efficiency while reading a manga-based interactive E-book in this pilot study. Furthermore, as this was also a pilot study, the sample size might not be enough to find an evident difference in the analysis of eye movement data among students that were with different levels of expertise in Japanese. The future experiment is suggested to conduct a formal study with a bigger population to investigate the effects of manga-based interactive E-book on students’ visual attention and learning performance.