1 Introduction

Reading is a long-standing challenge in the eye-tracking field, mainly developed and explored by Reichle et al. (2003). As explained by Burch (2022), this task involves not only considering how text is read but also identifying what and why a particular word or sentence is relevant to a participant.

From the beginning, eye-tracking experiments were conducted for understanding how people process information during a particular task. In that sense, several types of stimuli (image, text, video, etc.) have emerged to explore different aspects of cognition. According to the goal of the task, the stimulus can be dynamic or static. Its complexity depends on the semantic (e.g., an image has visual semantic areas that combine to shape the full picture) and whether it changes over time. Specifically, text is a powerful static stimulus whose internal structure allows exploring the relationship between words and the visual attention patterns through time with minimal effort.

The micro-stories are a particular type of text that are characterized for being short and for having strong semantic content. These properties turn them into an interesting natural text stimulus to evaluate different cognitive processes through the analysis of eye movement. Since the micro-stories have not been previously used as a stimulus, it is not easy to determine which set of characteristics should be considered in their analysis. On the other hand, a large amount of data of many different types is obtained or derived from its reading, several of which have spatial-temporal nature. Therefore, it is essential to analyze together the various characteristics that emerge from the micro-story reading experiments to obtain knowledge about them in order to find a suitable subset that will help in the understating of the cognitive process involved in micro-story reading.

Lately, several visualization techniques have emerged to assist in the analysis of eye-tracking data (see Blascheck et al. (2017) for a complete State-of-the-Art). These have been focused on various applications considering a wide variety of stimuli, among them, the reading of general text (Tang et al. 2012; Špakov et al. 2017; Goldberg and Helfman 2010). Due to their characteristics, the visualization techniques designed for eye-tracking data collected from reading other types of texts are generally not suitable to be applied to micro-stories. Most of the current solutions are designed for analyzing a particular aspect of reading, while the micro-stories require a global approach that integrates all characteristics in a single view for a better comprehension of the cognitive processes involved. Several authors have proposed many characteristics for studying natural reading considering a particular level of analysis. For this work, we survey and employ all these characteristics to achieve a full multi-view examination of micro-stories reading. This motivated us to propose a visualization technique to gain insight into the many and potentially complex relationships that may exist among the numerous characteristics involved in the reading of micro-stories in order to lead to hypotheses and/or new questions.

In this context, we present a novel visualization technique for analyzing eye movement data during micro-stories reading. The proposed technique was designed considering all the characteristics of interest defined for the study of a micro-story. Our contributions can be summarized as follows:

  • We provide a detailed specification of characteristics related to the stimuli and to the eye movements that are useful for the analysis of micro-story reading.

  • We introduce a novel visualization technique for analyzing eye movement data during micro-stories reading. The proposed technique integrates into a single view the stimulus and all the specified reading characteristics. Associated interactions are also provided to facilitate exploration and deeper analysis.

  • We detail two usage scenarios to illustrate how the proposed technique can be used for the analysis of the cognitive process during micro-story reading.

Finally, we provide the online implemented prototype of the proposed visualization technique (see Fig. 1). The implemented version is available at https://eyetrackingvis.github.io/ and allows the analysis of eye movement data during micro-stories reading through a standard web browser supporting all the encoding details and interactions defined in this work.

Fig. 1
figure 1

The proposed visualization technique is composed of two linked views: the Stimulus view (a) and the main Radial chart (b). In this example, a particular case is depicted, corresponding to one micro-story read by one participant. E, F, and G on the bottom show the effect of each implemented interaction and the information that they provide to the user. Bullet points C and D depict how the Stimulus view change depending on if the user selects a specified word or a transition

Full size image

2 Related work

Eye-tracking technologies have been used to promote our understanding of how people read in different contexts. Experiments have been carried out to analyze eye movement behavior during different tasks like reading code snippets (Peterson et al. 2019; Blascheck and Sharif 2019), capturing and analyzing Web reading behavior (Beymer and Russell 2005), online reading (Leckner 2012), reading words and sentences to study the cognitive performance of healthy people and patients with different pathologies (Bax 2013; Biondi et al. 2018), etc. The analysis of the datasets collected from these experiments requires different visualization techniques.

We organize the related work in two sections. First, we describe previous work related to eye-tracking experiments carried out during the fluent reading of different types of texts. Then, we provide an overview of related visualization techniques used to analyze the results of these experiments.

2.1 Eye-tracking in reading

During the past 35 years, researchers analyzed the ocular behavior during fluent reading studying the influence of syntactic, semantic, and morphological properties of the stimulus  (Rayner 1998; Kliegl 2007; Parker and Slattery 2019). The analysis of ocular movements is very useful in neuroscience, and particularly, in the evaluation of the cognitive process and behavior during reading (Raney et al. 2014; Biondi et al. 2018). There are many measures to define and compare the reading process for one and multiple participants. According to Holmqvist et al. (2011), these measures can be classified into two well-established groups: movement and position. Movement measures allow to identify what Areas of Interest (AOIs) were visited (and the path sequence followed), while position measures are relevant to identify where participants look at, how long, and how far from the center of the AOI. Pupil diameter is also an important measure as it allows to evaluate the amount of effort that a participant spent to process a sentence or a word (Just and Carpenter 1993). The meaningful combination of these measures allows identifying the kind of strategies the readers use to process and understand the text. The analysis of different aspects is necessary to understand this phenomenon from multiple points of view.

Finally, it is important to keep in mind that the reading process must be analyzed considering not only the aspects relative to the eye movement but also the characteristics associated with the text under study.  Rayner (2009) and references therein reported that lexical characteristics of the text are relevant to complement the analysis of the reading process.

2.2 Visualization for eye-tracking data

Holmqvist et al. (2011) defined basic visualization techniques to explore eye movement data as a complement of traditional statistical methods. However, most of these techniques are not easy to compare with each other, and it is necessary to apply functions to get relevant patterns (Kumar et al. 2016; Goldberg and Helfman 2010). Since then, several visualization techniques have been proposed to deal with eye-tracking data in a non-traditional way.  Blascheck et al. (2014, 2017) summarized different approaches grouped into point-based and area of interest (AOI)-based methods. These techniques were designed to be applied to a wide variety of stimuli, and most of them do not support a domain-based analysis.

Regarding reading data visualization, Tang et al. (2012) introduced a system to explore reading strategies with basic visualizations to show how participants interacted with the text. This application was a pioneer in the field, and it was adopted by many eye-trackers manufacturers as the default visual explorer of the recorded data. Špakov et al. (2017) presented a technique that combines multiple visualizations in a single view to facilitate the exploration of data collected in experiments where people are learning to read. However, this approach does not support a simple way to evaluate transitions among AOIs. Yang and Wacharamanotham (2018) introduced the Alpscarf as an extension of the traditional scarf plots that allows exploring temporal patterns, but it is only useful when the AOIs can be organized in a hierarchy.

The radial transition graph developed by  Blascheck et al. (2013), and its application to multiples scenarios (Blascheck et al. 2017) is an extension to related previous work to visualize relationships among multiples AOIs in an efficient way. Blascheck and Sharif (2019) describe an application of this method to a reading experiment to visualize common patterns across all participants. However, this approach is limited when the number of AOIs is bigger than ten, it does not allow adding complementary information of the reading process, and it cannot model transitions within an AOI.

Another approach to explore a large amount of eye-tracking data consists of multiple linked views. As Burch et al. (2021) explains, the linked views are useful to inspect the data from several perspectives. These views can be different visual techniques of the same dataset or the same one many times to show different aspects.

In summary, our revision of the current and most-used methods to visualize eye-tracking data recorded during reading exposes that most of the techniques show some particular aspects of the phenomenon. For instance, standard approaches like scanpath (Noton and Stark 1971), attention map (Mackworth and Mackworth 1958) or the transition matrix (Goldberg and Kotval 1999) are useful to get an overview of the data (reading pattern or dwell time), but they are not designed to effectively communicate detailed information.

In this paper, we propose and develop a new visual technique to address the issues described above considering all the important characteristics in micro-stories reading. In our approach, we decide to use a radial visualization instead of a Cartesian alternative. Radial visualizations have recently gained popularity as an interesting and useful alternative to present information due to their esthetic look. However, this aspect is not the only reason to use this approach.  Burch and Weiskopf (2014) handled this major concern in a well-detailed work, where they address the drawbacks and benefits of radial visualizations and their usefulness in knowledge discovery. Draper et al. (2009) provided a survey of papers using radial representations in different domains and proposed a taxonomy based on seven design patterns. These patterns can be explained using design dimensions to be considered when a new radial visualization is built. In Sect. 4.2, we described how these dimensions are implemented in our method and their advantages over Cartesian visualization.

3 Eye movements in micro-stories reading

In this section, we start describing the micro-stories properties. Then, we outline the ocular movements in reading, and we expand and categorize the most important characteristics during micro-stories reading.

3.1 Micro-stories and their characteristics

The micro-story is a literary genre that has reached its popularity among the public in the 21st century and it is characterized mainly by its brevity, structure and grammatical simplicity, and narrative. Most micro-stories share two main characteristics: brevity and effectiveness. The brevity can be typified by a certain number of words or letters (up to 140 words) (López and Conde 2012) and the effectiveness refers to the ability to play with language and the polysemy of words and to tell a story in very few sentences. Valls (2008) describes the compositional strategy of the micro-story as “a flash of meaning.” In other words, a micro-story aims to tell the greatest, richest, and most complex story conceivable within a certain limit of words. This requires the reader’s close attention for a short period of time in order to understand it.

Many studies mainly focused on word- and sentence-level processing to measure cognitive strategies or readers’ skills (Lou et al. 2017). However, readers often read paragraphs of text, and the cognitive load or the literacy skills involved in this situation are more complex than when reading a single word or sentence, involving a set of abilities to accurately process and understand text information during reading (Zimmerman et al. 2007). Additional skills are necessary during text fragments (passage-level) processing, such as focusing on relevant and important information in a passage, making connections with prior knowledge, and integrating important and relevant pieces of information within a passage (Van der Schoot et al. 2009). Compared to traditional stories or fables, or to fragments of larger works, micro-stories are very attractive because they are not restricted to the reading of a single word or sentence, but allow the reader to integrate different sentences to form a mental representation of the text and this can generally be used to derive measures of story comprehension (Schotter et al. 2014) or reading strategies (Hyönä and Nurminen 2006). Micro-stories have many interpretive possibilities despite their brevity, stimulate creativity in language through the explosiveness of their approaches and the use of various forms of expression, and develop skills such as lateral thinking and cognitive flexibility.

These characteristics make micro-stories an interesting stimulus to evaluate different cognitive processes through eye movement analysis.

3.2 Eye movements in reading

Lately, the process of reading started to be studied in many different reading contexts using the facility provided by eye-trackers. This research focuses on eye movements, which can be grouped into two well-established groups: fixation and saccades. Fixational eye movements are locations where the user focuses their attention during viewing, and they are constituted for small displacements of the eyeballs which ensure that vision does not fade during fixation. On the other hand, saccades are the transitions between two consecutive fixations and they give information about how the user processes the stimulus over space and time. During the reading of a text, saccades can be backward (mostly known as regressions) or forward, and they show how a reader scans the stimulus.

The variability in the fixations and the saccades reflects the cognitive processes associated with understanding the text. In fact, skilled readers tend to exhibit shorter fixations duration, longer saccades, and fewer regressions than poorer readers (Rayner 1998). In this context, eye movements are a strong indicator of the cognitive processes involved in the reading process.

Workload assessment is very useful in determining when a person’s cognitive, perceptual, or motor abilities are exceeded. It cannot be observed directly and must be inferred from observation of behavior or measurement of psychological and physiological processes. Various characteristics associated with eye-tracking movements have been used to assess workload in a variety of tasks, including reading (Zagermann et al. 2016). In general, it is known that pupil dynamic and fixations are sensitive to mental workload (Zhan et al. 2016). Chen et al. (2011) express that the duration of a fixation on a given AOI is related to the level of cognitive processing. They conclude that the variations of cognitive load, fixation duration, and fixation rate are indicators of an increase in attention required as the complexity of the task increases. Several authors have also shown that the pupil dilates when the difficulty of the task and the cognitive effort to solve it are higher (Fernández et al. 2016; van der Wel and van Steenbergen 2018).

In several experiments on text reading, it has been observed that the reduction of the amplitude of the transitions is normally related to an increase in the fixation time in each word (Zagermann et al. 2016; Raney et al. 2014; Hyönä and Kaakinen 2019).

The reading comprehension of a particular story follows a bottom-top approach, where the reader builds a coherent memory representation integrating words into sentences and sentences into a micro-story. This higher cognitive demand of comprehending a text is reflected in eye movements (Hyönä and Kaakinen 2019). Studies of eye-tracking during text reading have shown that readers dwell longer at sentence boundaries, where the readers integrate the information presented in the sentence before they move on to the next sentence (Rayner et al. 2000). The reason for this behavior is the readers want to make sure they understand the sentence they have just read before moving on to the next, producing increased fixation times on the last word of each sentence. Integrative processing at sentence boundaries may also be reflected in regressive fixations to earlier parts of the sentence or, sometimes, to an earlier sentence.

In addition to the complexity of eye movements, it is essential to consider the characteristics of the text being read, i.e., the length of the words, their predictability and frequency of use in the language, the syntactic properties, etc. This complex scenario makes it necessary to have a visualization technique that integrates all the information encoded in the eye movement data in a comprehensible way, to relate this information to different properties of the text being read and to facilitate the analysis of the variables involved and how they relate to each other.

Several authors define different characteristics of ocular behavior in reading that are useful for the study of global text processing, like dwell time, gaze time, word length, word frequency, transition direction, reading pattern, etc.

Table 1 Classification for the most important characteristics of eye-tracking data during micro-stories reading
Full size table

Hyönä et al. (2003) provide a complete classification of characteristics of ocular behavior in reading. These characteristics are specifically defined for text reading and consider that words, sentences, and regions are AOIs. We extend this classification to consider additional characteristics needed to analyze the reading of micro-stories (see Table 1). It is worth mentioning that in this particular work, after the data cleaning step, transitions and saccades are identical. Therefore, from now on, we’ll use the words transition and saccade indistinctly. In addition to those, we add other characteristics that are associated with words (Word Length and Word Frequency), with fixations (Landing Position and Pupil Diameter), with stimuli (Line Break and Sentence Order), with the readers (pupil behavior), and with transitions (Amplitude, Angle, Velocity Average, Velocity Peak, Duration, and Direction). Our proposed compilation is detailed in Table 1.

In Fig. 2, we show an example of a scanpath over a traditional micro-story for illustrating the characteristics mentioned above. In Fig. 2a, it is possible to appreciate the fixations performed by a reader with the respective temporal order, duration, and pupil diameter. For example, fixation 1 has a duration of 100 ms, a pupil diameter of 425 mm, and the landing position is situated between the letters q and u. The word Aquel has a dwell time of 180 ms, a gaze time of 180 ms, and a length of 5 characters. On the other hand, the word hombre has a dwell time of 200 ms, a gaze time of 0 ms, and a length of 6 characters. Finally, the word se is a skipped word. In Fig. 2b, the characteristics of the saccades are shown. The amplitude is defined as angular distance \(D(\mathrm{Pos}_i, \mathrm{Pos}_{i+1})\), the velocity is \(D(\mathrm{Pos}_i, \mathrm{Pos}_{i+1})\Delta t\) and the acceleration as \(\mathrm{Vel}_{i+1}-\mathrm{Vel}_i\Delta t\). For example, the saccade from fixation 1 to fixation 2 is a forward inter-word transition with a duration of 100 ms. On the other hand, the saccade from fixation 2 to fixation 3 is a backward inter-word transition.

Fig. 2
figure 2

A scanpath over a traditional micro-story illustrating (a) the characteristics of the fixations and (b) the characteristics of the saccades

Full size image

4 Visualization technique

We design a visualization technique to assist in eye behavior analysis during the micro-stories reading process. Our visualization technique fully supports the characteristics defined for micro-stories (see Table 1) and integrates them into the visual representation. The overview encodes most of them simultaneously at first glance. Then, additional sentences and word-level characteristics can be integrated on-demand. This comprehensive view facilitates the analysis of the eye movements during micro-stories reading enabling the discovery of reading patterns. In this section, we discuss the technical aspects of the proposed visualization technique.

4.1 Rationale design

To design our technique we followed the design criteria described in Draper et al. (2009), where decisions to build a new radial visualization are based on patterns and dimensions. Design patterns allow organizing the components on the screen, and dimensions are the mapping between a domain concept and its corresponding visual representation. They proposed seven design patterns and distilled a series of design dimensions that can be used to create new visualizations. Among those dimensions, we can find the units of visualization that refer to the graphical components that make up the radial visualization, and the significance of centroid, i.e., the role that the center point of the display canvas plays in the semantics of the visualization.

Our approach integrates different design patterns that are used to code multiple characteristics of the problem. According to the characteristic under study, we employ the particular pattern that is the best option to visualize it. To analyze the dwell time related to a word (or sentence) the star pattern is the best option to compare or detect a particular behavior (see the inner Radial chart in Fig. 3). To encode text hierarchy and the transitions between AOIs, we employ the concentric pattern (see the word annulus, sentence annulus, and the transition spiral in Fig. 3). At the dimension-level, we propose different graphical symbols to encode the characteristics of interest. Finally, the centroid in our technique has three different meanings according to the level of analysis performed by the domain expert. It corresponds to the value zero in the time scale that represents the dwell time in the Radar chart. It also represents the root node of the text hierarchy, and regarding the transitions, it gives a temporal reference. The closer the transitions are to the center, the earlier they occurred.

In the next sections, we describe in detail how the remaining aspects are implemented and their meaning in the context of the problem domain.

4.2 Visual representation

Our approach consists of two linked views: the main Radial chart and the Stimulus view (see Fig. 1). Besides, a filter panel and a button panel are provided. The Stimulus view is located to the left of the main Radial chart, providing a context for the exploratory analysis. A navigation panel is provided at the top of the Stimulus view for browsing among micro-stories and participants.

Fig. 3
figure 3

The main Radial chart, consisting of two concentric annuli (the word and the sentence annulus), two piecewise spirals (the pupil diameter variation spiral and the transition spiral), and an inner Radar chart

Full size image
Fig. 4
figure 4

Individual components that constitute the visual technique: a the word and sentence annuli, b the radar chart, c the transition spiral, and d the pupil diameter variation spiral

Full size image

The main Radial chart consists of two concentric annuli, two piecewise spirals (located inside and outside these circular rings), and an inner Radar chart (see Fig. 3). Both spirals wind around a central point which is the origin of the Radial chart, one moving farther from the outer annulus and the other approaching the inner annulus. In the main Radial chart (see Fig. 3), the micro-story is distributed in the annuli in a clockwise direction. The outer annulus is called word annulus because there, the micro-story is detailed at word-level (see Fig. 4a). The inner annulus is called sentence annulus, as there, the micro-story is detailed at sentence-level. The word annulus with inner radius \(r_\mathrm{in}\) is divided into annular sectors. Each of them corresponds to an AOI, that is, to one word or one word followed by a punctuation mark. To preserve the meaning of the AOI, we display a text label with the associated word (or word + punctuation mark). An annular sector span \(\mathrm{AS}_i\) depends on the radius of the annulus and is proportional to the length \(L_i\) of the i-th AOI. Then, \(\mathrm{AS}_i\) is computed as follows:

$$\begin{aligned} \mathrm{AS}_i = 2\pi r_\mathrm{in} \frac{L_i}{L}, L=\sum _{i=0}^{k} L_i \end{aligned}$$

As may or may not be fixations in an AOI, it is important to be able to differentiate between the two situations at first glance. If there are fixations in a particular AOI, the events that occurred in this area should be analyzed. If there are none, it may only be interesting to have a qualitative idea of the length of the AOI. Therefore, the corresponding annular sectors are mapped with a purple hatched texture. Initially, the AOIs are represented white or textured. The sentence annulus, with radius r, is divided into arcs of a given thickness. Each arc corresponds to a sentence depicting its length. In addition, we can also indicate where the line breaks occur in the text. This is marked with a black diamond placed below-right to the corresponding AOI.

The transitions are encoded with the outer piecewise spiral (see Fig. 4c) in an approach similar to that described by Lorigo et al. (2008). A transition corresponds to a saccade and is represented by an arc of that spiral. When in the reading process, there is a change in the direction of the transitions (forward/backward or vice versa), the spiral increases its radius in a given step. The incoming and outgoing positions for every transition j are defined by

$$ \mathrm{positions}_{\mathrm{pq}_{j}} = (\mathrm{AS}_p/2 \pm \mathrm{off}_p, \mathrm{AS}_q/2 \pm \mathrm{off}_q)$$

where off is the fixation offset relative to the AOI middle. The spiral’s color is associated with the transition type (intra-word, inter-word, and backward). The forward transitions are codified with blue or greenish blue, depending on whether they are inter-word or intra-word transitions. The backward transitions are coded with light gray. Besides that, the decreasing lightness of the transition curves (spiral segments) codifies the reading pattern of the complete micro-story reading trajectory.

The pupil diameter variations are encoded with the inner piecewise spiral (see Fig. 4d) moving away from the annuli. The spiral segments are joined by circles. Every circle represents a fixation location, and its color (calculated according to a logarithmic scale that allows showing small changes in the data) represents the pupil diameter. To reinforce the pupil’s change between two fixations, we propose a smooth color gradient associated with the link connecting the circles.

To preserve the temporal order of transitions and pupil diameter variation flow, we define \(2n + 1\) levels of continuous reading, where n is the number of backward transitions. Each level is defined as a sequence of forward transitions during which the reader does not transition to a previous AOI. This spiral follows the same path pattern as the one corresponding to the transitions, thus allowing its temporal association.

In the inner section of the chart, we display the dwell time for each AOI, word, or sentence, with a Radar chart (see Fig. 4b). The total time spent in each AOI for words is presented as a point in the corresponding radial axis. By connecting these points, we generate a polygon. The length from the center of the Radar chart to this point represents the length of a spoke, and it is proportional to the dwell time for the fixation relative to the maximum value of dwell time computed across all the recorded fixations. This chart aims to show a comparable and distinctive shape representing how a participant reads a micro-story with an emphasis on the time they spend on each word. The total time spent in each AOI (sentence) is overlapped to the individual times spent on each word in the sentence and is presented as a circular sector with its radius representing time. We use a logarithmic radial scale in order to have a comparable representation of the partial and total spent time per sentence. The color scales were chosen in such a way that the extremes can be distinguished due to the variation in lightness. The color scheme is color-blind safe (verified by the color blindness simulator Coblis (Flück 2020)).

Next to the main chart, the micro-story text is displayed in the Stimulus view in the same way that is shown during the experiment, allowing the analyst to explore the visualization in the context of the stimulus. Finally, the filter panel located at the bottom left of the screen provides a set of sliders to filter transitions according to their properties. In this way, the user can select which transitions are important for their analysis selecting a range of values for them. The button panel allows the visualization of the sentence time in the Radial chart and the word frequency in both the Stimulus view and the Radial chart view. The sentence time is computed as the average of the times of its words, while the word frequency is encoded with a gray color scale with four categories: “High,” “Often,” “Medium,” and “Low.” These categories were calculated from a histogram of the normalized frequencies of all the words used in the micro-stories.

4.3 Interactions

Our technique supports a rich set of interactions based on AOIs (words or sentences) and transitions. According to  Yi et al. (2007), we provide three different categories of interactions: explore, abstract/elaborate, and filter. Next, we explain how these interactions were chosen and implemented in our technique.

Fig. 5
figure 5

Interactions over a transition (a) and over an AOI (b). In (A), the words associated with the selected transition are highlighted (eso and señor). In (B), the target AOI (the word ser) is highlighted in dark gray and the AOIs of the related words are highlighted in a lighter gray

Full size image

4.3.1 Interactions over an AOI

The micro-stories can exhibit different characteristics at different levels. At word-level, interactions are focused on the word under analysis, considered the target word, and its relationship to other words (see Fig. 5b). This relationship considers words that have at least one transition in common with the target word. To interact at word-level, the target word should be double-clicked. In this case, the AOIs of the target word and the related words are highlighted with a red border, the border of the target word being wider to identify it. Besides, each polygon vertex of the sub-Radar chart is updated considering the sum of times in the involved fixations. That is, if a word has several associated transitions, the position of the vertex on the radial axis corresponding to this word considers the sum of the times of each fixation associated with the multiple transitions. Therefore, the total fixation time is highlighted for that word and the previous and subsequent words in which transitions occurred allowing the appreciation of the variations in time during transitions. A transition-based analysis is useful to find out how the text is explored, while a word-based analysis is relevant to understand what words are important for the participant during reading. In addition, more information about the word is provided, such as word frequency and gaze time (see Fig. 1e).

At sentence-level, interactions are focused on the sentence under analysis. To select the sentence in focus, the corresponding sentence annulus should be double-clicked (see Fig. 1c). The sub-Radar chart shows the sum of times in the involved fixations in the sentence. Its transparency makes it possible to appreciate, in context, the times of the fixations by word.

4.3.2 Interactions over a transition

When clicking on a particular transition, the words associated with it are highlighted with a red border (see Fig. 5a). The remaining words and transitions are softened to avoid the analyst’s attention but are not erased to provide context. The orange sub-Radar chart shown in the center highlights the dwell times at each word related to the fixations involved in the transition. These are shown in the context of the total dwell times in each word (gray polygon). As the views are linked, this interaction is also reflected in the Stimulus view (see Fig. 5a). In addition, more information regarding the transition such as amplitude, duration, mean and peak velocity, angle, and direction can be provided by demand (see Fig. 1d).

Our approach also supports filter transitions through slide bars, situated in the filter panel. The user can filter transitions according to the characteristics mentioned above.

4.4 Encoding the characteristics

The micro-story is presented in the Stimulus view, exactly as it is shown to the reader during the reading session, and in the main Radial chart (see Sect. 4.2). Thus, all stimulus-related characteristics are encoded in these two linked views, namely the Line Break, the Word Order in Sentence and the Sentence Order in Micro-Story (see Fig. 1).

On the main Radial chart, we show how the reader moves his eyes during the reading process, encoding both the order in which he/she reads and the time it takes him/her to do it. The outer spiral represents the Reading Temporal Order. Its lightness decreases as the reading progresses. At the same time, it is also possible to follow the variation of the pupil (Pupil Behavior) in the inner spiral that evolves with the same forward temporal pattern as the outer spiral and accompanies the reading sequence, varying in color according to pupil variation. The Reading Pattern can consider both the total time spent on a word and on a sentence. Both are displayed in the inner Radar chart. The polygon shaped by joining the points of the fixation times on each word constitutes a reading pattern of the micro-story. Each reader will generate a different one (see Fig. 7).

The word characteristics are also visualized in the main Radial chart. These are mostly concentrated in the word annulus, where the words and their lengths are represented (see Sect. 4.2). Skipped Words (words that are never fixed) are mapped with a hatched texture. Double-clicking on the word allows seeing, on the Radar chart, its Dwell Time on the radial axis centered in the word (see Fig. 5b). It is also possible to compare whether that time increased or decreased with respect to the words previously and subsequently read. When a word is selected, those words are highlighted. The Word Frequency, both absolute and normalized, can be obtained hovering over the word (see Fig. 1e). Additionally, an interface button allows showing the corresponding Dwell Time for sentences, superimposed to the Dwell Time of the words in it. Because it is translucent, it also possible to see the dwell times of the words. Another button allows the encoding of the word frequency coloring each word and sentence segment in the Word and Sentence Annulus, respectively. In this case, the words in the Stimulus view are also colored to maintain consistency between the views.

When a fixation occurs, it is necessary to know where in the word the eyes were initially fixed (Landing Position), the duration of this fixation, and the pupil diameter at this point. These landing positions are visualized where the outer spiral is interrupted. At the corresponding point of the inner spiral, the Pupil Diameter is represented by a colored circle. The Gaze Time, calculated on the basis of the given Fixation Duration in the word, can be obtained hovering over the word (see Fig. 1e). Finally, transitions are visualized as segments in the outer spiral. Both the Direction and Transition Type are identified by the color of the arc segment that corresponds to the transition. In the case of the Transition Type, it is either intra-word or inter-word, corresponding to greenish-blue and blue, respectively. The transition Direction relative to a fixation can be forward or backward and are represented blue or gray, respectively. The additional transition characteristics (see Table 1) are obtaining hovering on the transition arc (see Fig. 1d).

It should be noted that all the qualitative characteristics are simultaneously visualized in the main Radial chart. Other quantitative characteristics such as word length and pupil diameter are displayed qualitatively. The latter, as well as other quantitative characteristics, is shown numerically on demand. The results illustrate how the developed visualization technique allows analyzing eye-tracking data during micro-stories reading considering most of its characteristics simultaneously and in the context of the stimulus. These results underline the potential of the technique to perform the visual analysis of eye movements in micro-stories reading enabling an integrated and correlated analysis of the main characteristics in the micro-stories reading process.

4.5 Comparison

In Sect. 2.2, we discussed the limitations and utility of the state-of-the-art methods to visualize eye-tracking data. To better understand how our approach is useful in this context, it is precise to compare it with the state-of-the-art methods. We contrast our approach with the two most recent methods that we consider are most closely related to our proposal, the Radial Transition Graph (RTG) (Blascheck et al. 2017) and the AlpScarf (AlpS) (Yang and Wacharamanotham 2018).

At first, we consider analyzing if the RTG and the AlpS support the characteristics detailed in Table 1. However, since our technique was specially designed for textual stimuli and with the above-mentioned characteristics in mind, we believe that a taxative comparison would not be fair. Then, we compare the effectiveness of the involved methods considering the basics characteristics of the text (Rayner et al. 2006), the basics characteristics of the eye-tracking measures (Holmqvist et al. 2011), and the employed layout (Burch et al. 2008; Kim and Draper 2014).

Our approach encodes the basic text characteristics (word length, word frequency, and word position) and the basic eye-tracking measures (dwell time, fixations, and pupil diameter variation) as explained in Sects. 4.2 and 4.4. The original position of the words is not preserved in the RTG method and not explicitly represented in AlpS; however, in the latter, it is possible to infer it from the color bar labels. Neither the word length (or AOIs’ size) nor the word frequency (or AOIs’ frequency) are encoded in either of the two methods. Regarding the eye-tracking measures, the dwell time is represented in the RTG as the size of the ring segments, but neither the individual fixation duration, nor the pupil diameter variation, nor the temporal order of the transitions are represented in that technique. On the other hand, although in AlpS, the fixation duration is encoded as the width of the rectangles, this method does not allow representing neither the dwell time nor the pupil diameter variation. Besides, AlpS represents the order of transitions by making use of visual components named mountains, valleys, and creeks.

In addition, it is important to analyze whether the methods focus on the spatial or temporal characteristics of the data. Spatial methods only focus on how the participant’s attention is distributed (like RTG), whereas temporal methods depict the order in which AOIs were visited (like AlpS). Our method follows an spatio-temporal approach, integrating both aspects, and having the potential to encode the complete behavior of the participant in one view. Regarding the employed layouts, AlpS follows a Cartesian approach, that, in the particular case of a reading task, does not scale when the number of words (sentences) is large due to a crisscross pattern. The micro-stories can have multiple AOIs and many transitions can occur between them, which can lead to occlusion and misleading of data. Our approach and RTG employ a radial approach that has been studied as an alternative to avoid these problems and to help the analysis.

Finally, our approach and RTG and AlpS present scalability problems. In RTG and AlpS methods, the limitation is given by the coloring strategies that identify each AOI with a unique color (RTG) or a group of AOIs with a variation of hue or luminance (AlpS), inducing multiples and different AOIs to have the same visual representation. This is a problem when the number of AOIs is bigger than a dozen, which is a small number in micro-stories. As we detail in Sect. 5.4, our approach also presents scalability issues, but we deal with this problem by integrating the Stimulus view.

5 Usage scenarios

During the exploration of the data under study, experts commonly formulate several questions and hypotheses about them. Visual exploration is an iterative process where users gain knowledge about the data and build new questions using their own expertise and the acquired information. This feedback loop is possible thanks to the interactions offered by the tool and how the users can process or modify the data to discover useful insights. Taking this into account, we collaborated with an expert group in eye-tracking and neurosciences to perform unpaid tasks with our tool. Due to these experts having basic expertise in visualization, we introduce them to the essentials of our technique and other issues of their interest. The experts defined a set of relevant questions at the initial step of analysis according to their previous experience:

Question 1::

The micro-story requires the reader’s utmost attention. Is it possible to appreciate this cognitive overload from eye movements and pupil behavior?

Question 2.:

Is it possible to visually detect the moments when the reader integrates the text information into memory?

These questions are a first step in the analysis and they must be used as preliminary findings that could be later reformulated to cover another aspect of the study or to detect a particular case. In order to illustrate how the designed visualization technique and its interactions can help in the exploratory analysis of eye movements during micro-story reading, we define two usage scenarios based on the formulated questions.

As follows, we provide details about the micro-stories and the readers involved in the usage scenarios. Then, we examine how the analysis of the data obtained from the reading records can be approached to answer the questions posed. We emphasize that answering the posed questions involves analyzing, as a whole, several of the characteristics listed in Table 1.

5.1 Micro-stories corpus and readers

To set up the scenarios we used a set of 15 micro-stories in Spanish grouped by length. To be consistent with our definition of micro-story, the average length of the text is 54 (SD = 20) words and the mean word length is 5 (SD = 3) letters. Four participants (Spanish native speakers) were part of our recording session. We selected young persons between 25 and 35 years old and college education belonging to our university. None of the participants showed signs of vision problems or received payment for their participation.

Fig. 6
figure 6

An analysis session where the experts identify that the most relevant part of the text is in the middle of the micro-story. The reader makes regressions in sectors A, B, and C. In addition, in sector C, the pupil diameter is dilated and the dwell time remains big for a set of consecutive words (D), indicating that this sector might be the most relevant to understand the micro-story

Full size image

5.2 Usage scenario 1

Question 1:

The micro-story requires the reader’s utmost attention. Is it possible to appreciate this cognitive overload from eye movements and pupil behavior?

As explained in Sect. 3.2, various characteristics associated with eye-tracking movements have been used to assess workload in reading, like pupil dynamic and fixations. Figure 6 shows a particular analysis session where a participant processes the text during a micro-story reading with visual emphasis on pupil behavior (inner ring), dwell time (Radar chart), and saccades (outer ring). The goal of the analysts is to identify the most relevant parts of the micro-story and their effect on reading. At the beginning of the session, the experts analyze the variation in pupil diameter and quickly realize that the middle part of the text seems more relevant to understand than the rest of the story. As the participant progresses through the text, the inner ring depicts how the pupil diameter increases considerably in the central part of the text and then decreases again before reaching the end of the micro-story. Besides, the experts realize that there are 3 sectors of the text where the participant makes regressions while reading (see Fig. 6a, b, and c). In sectors A and B, the reader performs only one regression, but in sector C, the number of regressions increases. Then, the experts focus on sector C, and after exploring the different characteristics encoded in the visualization, it becomes clear to them that for a given set of consecutive words, the dwell time (total time that a participant fixates on a word) remains high (see Fig. 6d), and the fixations rate decreases. To explore the relationship between the amplitude of transitions and fixation time, the experts use the filter panel by selecting only those transitions with small amplitude, verifying that in fact, transitions involving words with longer fixations have smaller amplitudes.

Fig. 7
figure 7

An analysis session where a particular reader processes a micro-story composed of two sentences. The reader only performs one regression at the end of the first sentence (a, sector 1) through the entire reading. By selecting the last word of each sentence, the analysts notice that the reader dwells longer in the last word of each sentence (b and c)

Full size image

The experts were able to appreciate the visual pairing between pupil behavior and transitions between words, revealing that the central part of the text requires more attention for better comprehension of the micro-story by the participant, leading to a large number of fixations and saccades to integrate critical information. This increase is related to changes in pupil diameter and shows how both the oculomotor and memory systems work in harmony to grasp the meaning of the text.

5.3 Usage scenario 2

Question 2:

Is it possible to visually detect the moments when the reader integrates the text information into memory

The cognitive demand of comprehending a text is reflected in eye movements (see Sect. 3.2). Specifically, a longer fixation time on the last word of a sentence indicates the time needed to understand a sentence before moving on to the next one. This process can also involve regressive fixations at the end of the sentences.

Figure 7 shows how a particular reader processes a micro-story composed of two sentences. To answer the posed question, the analyst begins by exploring the Radar chart to verify if the reader spends more time at the end of the sentences, presumably integrating the information into memory. By selecting the last word of each sentence, either from the Stimulus view or directly in the main Radial chart, the dwell times of the selected word and related words are highlighted in the Radar chart, revealing that indeed the reader spends more time on the last word of each sentence than on the rest of the respective sentence (see Fig. 7b and c). Moreover, the reader only performs one regression in the entire reading (see Fig. 7a) at the end of the first sentence. Analysts suppose that by stepping back, the reader provides him/herself with another opportunity to visually sample a text region he/she might find difficult to understand.

Figure 8 shows another analysis session, in which a reader processes another micro-story also composed of two sentences. In this case, the reader also dwells longer at sentence boundaries (see Fig. 8a), but performs two regressions that caught the analysts’ attention. In both regressions, the reader goes back to distant words in the text, and neither regression is at sentence boundaries. The analysts thought that perhaps the reader needed to go back because she/he came across an unknown or infrequent word. The analyst then activated the word frequency encoding and realized that in fact, the words involved in the regressions are low-frequency words, which explains the reason for the regression. In addition, thanks to the correlated Stimulus view, the analysts discovered that this reader exhibits interesting behavior. By selecting the regressions, the analysts realized that in both regressions the target word is placed above the source word on the immediate top line of the text in the original stimulus.

Fig. 8
figure 8

An analysis session where a particular reader processes a micro-story composed of two sentences. The reader dwells longer at the end of the sentences (a.1 and a.2) and performs two regressions in the whole reading (b and c). Thanks to the correlated Stimulus view, the analysts found out that in both regressions, the target word is placed above the source word on the immediate top line of the text in the original stimulus

Full size image

5.4 Limitations

Visual and perceptual scalability is an important aspect to consider. If the participant performs many backward transitions, the space required to draw the transitions grows by a factor of two. This situation results in a large gap between the word annulus and the transition arc in a certain part of the text. While the main Radial chart is more suitable than a linear representation presents a similar limitation when the number of words increases. This problem has been partially solved by reducing the font size. Although it may reach the limit of readability in the Radial chart, in the Stimulus view, the text can still be read as clearly as it is presented to the readers. In this way, the user can analyze the Radial chart and always know, regardless of the size of the micro-story, to which word an annulus sector in the Radial chart relates.

When the participant makes many fixations on a particular region of the text, the circles representing each fixation may overlap. This overplotting can be considered as a visual indicator of those areas where the participant put more attention. However, we consider that specific interactions may be necessary to deal with the overplotting, and we plan to work on this as future work. An interesting approach may be the generation of clusters for those dense regions of fixations and provide the corresponding interactions to manipulate them.

In terms of computing performance, the technique has a complexity of O(n+m) where n is the number of words of a micro-story, and m is the number of transitions performed by the participant. In that sense, when the size of the micro-story reaches up to the limit of words established, the number of transitions grows as a consequence; then, the time to generate the plot is slower.

6 Conclusions and future work

We proposed a new visualization technique for analyzing eye movement data during micro-stories reading. This technique is aimed at a very broad type of users of eye-tracking technology, both scientific and professional. We also provide a detailed classification of the most relevant characteristics of eye-tracking data to be taken into account during micro-story reading (see Table 1) and integrate all of them into the proposed visualization technique, providing the required interactions to facilitate exploration and analysis.

We believe that much can be learned from the application of the eye-tracking method to study reading in general, and especially in micro-story reading. To this end, we believe it is of vital importance to have tools such as the one presented in this paper, which shows the different characteristics of eye-tracking data to be considered in this setting and can highlight the possible interrelationships that may occur between them.

Finally, we expect to apply this technique to domain-specific problems related to the use of micro-story reading. We find it particularly interesting, for example, to study whether each micro-story has a particular reading pattern when read by an average reader. Besides, we plan to extend our visualization technique by focusing on the comparison of multiple readers and micro-stories.