Keywords

1 Introduction

Recent advances in handheld device technologies, in addition to the tremendous expansion of their use, have resulted in a boom in the electronic book market. Online booksellers have reported that their sales of these e-books have now outpaced printed books [1]. However, the resolution of the electronic viewer is still lower than that of the printing press so the character clarity is also inferior to paper. Small screen sizes pose additional constraints for the use of handheld devices as electronic book readers. So it was once thought that the use of handheld devices for reading digital documents would be limited [2]. With the release of popular dedicated devices such as Amazon’s Kindle, as well as the increasing popularity of other handheld devices, the market expanded greatly and many companies began developing e-book viewers.

Although the market size of e-books was one billion JPY in 2002 in Japan, by 2010, the market had grown to 65 billion JPY, with expectations to reach the 200 billion JPY level in 2016 [3]. The 2010 release of the Apple iPad also invigorated the e-book market greatly [4, 5]. Now, new e-book viewing software is being released every day. Reading books on tablets, smartphones, and dedicated readers has become a normal daily activity for many people who love books.

There have been a number of publications on e-books and reading and various experts’ opinions vary on the effects of reading e-books versus print books [6]. There have been a number of studies on “supported eText” which means text with materials to support the meaning of the content, especially for those who have reading difficulties [7]. A general consensus of such research is that multimodal contents such as audio enhancements improve comprehension when they match the contents and may hinder comprehension when not supportive of the content. Since, however, much of this research has focused on reading for comprehension and increasing reading skills, few studies have focused on e-book reading as an emotional experience, nor on how multimodal contents may influence that experience.

This chapter describes near-infrared spectroscopy (NIRS) monitoring of brain activity during e-book reading experiences that are enhanced with audio content versus those that are not, in order to infer differences in emotional experience, and feelings of concentration and devotion. Two popular books in e-book format were downloaded from the Internet and two copies of each were enhanced with sound effects, one copy of each with sound effects thought to be suitable to the content and one copy of each with sound effects thought to be unsuitable. These experimental stimuli are discussed in the following section. Next, the experiments about emotion evaluation of reading by NIRS are conducted using a viewer for the e-books developed based on iPod touch by authors, as described in Sect. 3. Section 4 gives the results of those experiments and discusses what the NIRS monitoring results suggest about how the addition of various sound effects influences the emotions of the e-book readers, such as feelings of concentration or devotion to reading. Finally, Sect. 5 gives the conclusions reached and proposes possible areas of future study.

2 E-Books with Sound Effects

2.1 Original Books

The e-books used in the experiments are “Night Train to the Stars (Ginga Tetsudo no Yoru in Japanese)” by Kenji Miyazawa and “Jack and Beanstalk” translated into Japanese by Masao Kusuyama. In popular opinion, the former is said to have “dark” contents and the latter is said to have “bright” contents. These e-books were selected under the following conditions:

  • The story is well-known in the socio-cultural group of the subject pool.

  • The book is easy to read and comprehend.

  • The book is available in the public domain for educational uses.

2.2 Sound Effects

Sound effects from animated video (DVD) productions of the stories were selected as “suitable” sounds. The reading speed of each subject is different so the timing of sound effect playback was set immediately after the reader has turned a page. Therefore, the sentence where the sound effect was considered to be most appropriate was placed at the top of the page. This method of sound effect implementation has been adopted previously for example in Murakami’s production of the e-book “A Singing Whale (Utau Kujira)” [8, 9].

Experiments were conducted to validate the suitability of the selected sound effects for the book contents. A five-step Likert Scale (1: strongly disagree, ···, 5: strongly agree) were used for the evaluation. Table 1 shows the average of evaluation scores of 10 subjects. The scores 4.67 for “Night Train to the Stars” and 3.08 for “Jack and Beanstalk” were considered high enough to validate their adaptation as suitable sound effects then the adopted sound effects. Various sound effects available for free download from the Internet were used as the “unsuitable” ones. They include various sounds such as noises from daily life, noise of impacts, and various natural sounds. Hundred sounds were collected in total, and these were randomly attached to the e-books in the same manner as the suitable sound effects.

Table 1 Average scores of sound effect suitability for the book contents
Full size table

3 Experiments

3.1 Objective

The objective of the experiments is to use NIRS to investigate the hypothesis that the emotions of e-book readers with the suitable sound effects is stronger than that when reading both e-books with unsuitable sound effects and those and e-books without sound. Emotions in this case refer basically to reported feelings of concentration or devotion when reading, in other words, feelings of an emotional attachment to the story. These feelings are thought to be closely related to the brain activity in the prefrontal cortex region [10].

NIRS is a physiological method to measure various changes associated with the spectrum of infrared light passing through the human body and is employed here to measure changes in the density of oxygenated hemoglobin (oxyHb) associated with brain activity. The basic principle of NIRS brain activity imaging is shown in Fig. 1. NIRS can be used to monitor the status of oxygen density in these states: oxyhemoglobin (oxyHb), deoxyhemoglobin (deoxyHb), and total hemoglobin (totalHb, the sum of oxyHb and deoxyHb). Since the behavior of deoxyHb density is complex and the change in totalHb is relatively small, this study only considers oxyHb density [11].

Fig. 1
figure 1

Light is passed from each emitting optical fiber, through the cranium into the cerebellum, scattering light back into each adjacent receiving optical fiber. Specific changes in the spectrum received indicate changes in the density of oxygenated hemoglobin (oxyHb)

Full size image

3.2 Procedure

Subjects read the e-books using a smartphone application viewer developed by the authors fitting with the NIRS head-mounted apparatus and sitting in a chair. Figure 2 shows the procedure of each experimental session.

Fig. 2
figure 2

Each experiment session starts with setting and starting the NIRS equipment, followed by a brief rest, after which the task begins and, after the task is completed, there is another brief rest before the NIRS equipment is removed

Full size image

First, oxyHb density of the subject is measured by NIRS in a state of rest, with eyes closed, for 5 min. This is called Rest 1.

Next, the subject reads the e-book “Night Train to the Stars” for 30 min or “Jack and Beanstalk” to the end of book. The average of reading time of “Jack and Beanstalk” was found to be about 11 min in preliminary experiments. During this whole time, the oxyHb density of the subject is measured. This measurement is called Task X, where X is the task number shown in Table 2. Finally, the oxyHb density is measured again in the same manner as in Rest 1. This is called Rest 2. The experimental results of Rests 1 and 2 are used for a baseline adjustment of the experimental results of Task X. The subjects participated in either Tasks 1–3 or Tasks 4–6. The order of tasks is random, and the time interval between tasks was more than 1 week.

Table 2 Tasks to be completed for each experiment session
Full size table

3.3 E-Book Viewer

The e-book viewer for the experiments was developed using the iPod touch device, by Apple Inc. The document of the book is arranged in the vertical Japanese form, with up to 21 characters in each column and 8 characters in each row (up to eight columns or lines of text). The e-book viewer imports document and displays it. The page turning of the documents is done by flicking or wiping the display. Each sound effect is played back immediately after the page turn is complete. Figure 3 shows a part of “Night Train to the Stars” displayed on the handheld device.

Fig. 3
figure 3

The reader application screen showing one page of “Night Train to the Stars”

Full size image

4 Experimental Results

4.1 Preprocessing of NIRS Data

The following procedure was used to preprocess the oxyHb density data:

  1. 1.

    A weighted moving average method with a window of 11 data points is applied for smoothing.

  2. 2.

    Baseline adjustment is performed. First, a line connecting the average of oxyHb densities in Rest 1 and Rest 2. Next, the difference between the oxyHb density in Task X and the line is calculated as the adjusted oxyHb density.

  3. 3.

    The most clearly indicative oxyHb density is selected from all channels of NIRS, as evaluated by lack of noise and other probable outlier indications. Below, this adjusted and selected value is called the oxyHb density.

4.2 “Night Train to the Stars”

The oxyHb density at Channel 10 (as numbered in Fig. 4) of Subject A in Tasks 1–3 is shown in Fig. 5 as a typical result in case of reading “Night Train to the Stars”. Table 3 shows the averages of oxyHb for all subjects who did Tasks 1–3.

Fig. 4
figure 4

The locations and position of the channels, where each channel is the path from one emitting optical fiber into one receiving optical fiber

Full size image
Fig. 5
figure 5

oxyHb density changes at Channel 10 of Subject A in Tasks 1–3

Full size image
Table 3 Averages of oxyHb densities for all subjects reading “Night Train to the Stars” for 30 min
Full size table

Figure 5 and Table 3 show that the oxyHb densities in all tasks increased while Subject A was reading “Night Train to the Stars”. In the case of Task 1, in which the sound effect is suitable for the contents of book, the increase in the oxyHb density is largest. The second largest density is for Task 2, where the sound effects were unsuitable. The lowest density is Task 3, in which there were no sound effects. The results shown in Table 3 were compared using a t test, which showed a P ≤ 0.5 significance level between each of the tasks.

4.3 “Jack and Beanstalk”

The oxyHb density at Channel 7 of Subject G in Tasks 4–6 is shown in Fig. 6 as a typical result for subjects reading “Jack and Beanstalk”. In these tasks, all subjects had to read the e-book to the end. So they took different times to complete the task. Subject G took 10 min and 44 s (10′44″), 09′05″ and 09′09″ in Task 4, Task 5, and Task 6, respectively. Figure 6 shows the oxyHb densities by 09′05″ when the time is shortest. Table 4 shows the averages of oxyHb densities at Channel 7 of Subject G while reading the e-book to the end.

Fig. 6
figure 6

oxyHb density changes at Channel 7 of Subject G in Tasks 4–6

Full size image
Table 4 Averages of oxyHb densities for all subjects reading “Jack and Beanstalk” to the end
Full size table

Figure 6 and Table 4 show that the oxyHb densities in all tasks increased while Subject G was reading the e-book. The oxyHb density in Task 4, in which the sound effects are suitable for the contents of the book, is largest as was shown in subjects reading “Night Train to the Stars”. Also, the second largest oxyHb density is Task 5 that the sound effects are unsuitable and the lowest one is Task 6, in which there were sound effects. These results are similar to those in case of “Night Train to the Stars”. The results shown in Table 4 were compared using a t test, which showed a P ≤ 0.5 significance level between each of the tasks.

As shown in Table 4, the averages oxyHb densities of all subjects are largest in Task 4 and second largest in Task 6, except for Subject H. Possible explanations for the oxyHb density in Task 6 to be the second largest, as opposed to the third as in subjects reading “Night Train to the Stars”, include the following:

  • The subjects concentrated less on reading because the reading time is shorter.

  • The bright (positive) contents and the unsuitable sound effects prevented the subjects from concentration in reading.

4.4 Discussion

Previous studies have reported that the prefrontal cortex is activated when a person concentrates on reading or comprehending a task, such as calculation, which requires concentration [12–14]. Since the activation of the prefrontal cortex activity is observed when the subjects read the e-books with the suitable sound effects for the book contents, the results suggest the subjects concentrate on the reading and felt devotion to the story.

Comparing in terms of the reading time, for “long” reading, the suitable sound effects were best and the unsuitable ones were better than no sound effects. On the other hand, in case of “short” reading, no sound effects had better responses than the unsuitable ones. In addition, comparing in terms of the contents of book, “positive” (bright) or “negative” (dark), the suitable sound effect is best for the e-books in both contents and the unsuitable sound effect is better than no sound effects in case of “negative” contents. On the other hand, no sound effect was better than the unsuitable ones in case of “bright” contents.

5 Conclusions

The results of this study give limited support to the hypothesis that audio effects used appropriately in e-books increase feelings of concentration or devotion. Suitable sound effects evoked responses in e-book readers that were consistent with feelings of concentration in reading or devotion to reading efficiently, according to the results of NIRS. The results of these studies suggest that the intensity of such emotional responses is also related to the contents (bright or dark) and reading time (long or short) in comparisons with unsuitable sound effects versus no sound effects.

This study shows methods and an example of studying brain activity during e-book reading using NIRS. The study described here shows how using NIRS allows tasks such as reading in a natural environment without distractions or special environments required in many other imaging systems, such as functional magnetic resonance imaging (fMRI). The results also show some of the methods and challenges in handling NIRS data for evaluation of emotional response. Understanding the role of limited multimedia in e-books allows educators, writers, publishers, and other communicators to plan and publish more effective e-books for education and entertainment. Future studies will continue to elaborate more precisely the neurological and psychophysical influence of sound effects in e-book reading with more accurate NIRS methods and other physiological measurements of brain activity.