1 Introduction

Over the past two decades, multimedia collections have experienced a rapid growth in terms of importance and volume. An increasing number of scientific fields started to add scientific imagery in their studies in order to advance the state of the art. Such a growth of interest created a need for sophisticated multimedia analytics systems enabling large-scale, interactive, and insightful analyses [30, 55].

As argued in [29], since we live in space and time, in all visualizations the time is a relevant dimension. Numerous applications in different sectors, such as in the industrial, medical, insurance, and financial sectors, store and use temporal (or time-referenced) data, i.e., data with an associated time period. For instance, financial application store the history of stock market data; insurance applications store the validity periods of insurance policies; reservation systems store periods of booked rooms; and medical information systems store patient treatments as periods. Database systems are capable of supporting the storage and management of period data [8, 9] through, for instance, range types in PostgreSQL or the period specification in the SQL:2011 standard [32].

In literature, time-based visualizations have a long and successful history [53]. However, the majority of work is focused on data with associated time points, such as time series data [49]. The importance of an appropriate visualization and comparison of period data has not been sufficiently studied. In fact, the visualization and comparison of time periods received scant attention compared to its storage, representation, and management. Moreover, the use of time as a primary dimension for multimedia analysis purposes is surprisingly little considered [53].

In this paper, we focus on the visual analysis of multimedia collections (images) with associated period data. We present a novel multimedia application that is based on three pillars:

  • the Time∘diff visualization approach [18];

  • an interactive data exploration approach inspired by the Visual Information Seeking Mantra (VISM); and

  • a multi-viewpoint visualization approach.

The Time∘diff visualization approach presented in [18] provides an intuitive way to visually compare data with associated time periods, by visualizing in a concise way the amount of data, the changes in the data, and the number of changes in the data. In this work, we extend this visualization technique to support interactive data exploration based on VISM [48]. VISM allows a user to start from an overview of the data and to reach specific details by zooming in and filtering the data in a step by step fashion. Finally, we adopt a multi-viewpoint visualization approach to be able to provide different viewpoints of the data in a single glance. By combining these three pillars, the proposed application enables decision makers to easily summarize, analyze and compare multimedia data with associated time periods.

To summarize, the main contributions of this paper are as follows:

  • We propose an application which, by combining different content forms (e.g., text, images, graphics) of data, is able to provide a first summary and a subsequent analysis of the data in an interactive way.

  • We implemented an interactive Time∘diff approach for visualizing the amount, the changes and the number of changes of period data.

  • We present a case study of our application using real-world data from eye-tracking experiments. The case study shows that our application is able to concisely handle long histories and to analyze them in an interactive way.

The remainder of this paper is organized as follows. In Section 2, we review previous visualization approaches and techniques that are related to our work. In Section 3, we provide a motivating example and the problem definition. In Section 4, we introduce the Time∘diff approach, which forms the basis for the proposed solution for the visualization of the time periods of the data. Section 5 presents our multimedia analytics application for summarizing and analyzing data from eye-tracking experiments. In Section 6, we discuss an application example on a real-world dataset. Section 7 presents a qualitative evaluation of the feedback from experts using our application. Finally, Section 8 presents the concluding remarks and outlines possible future studies.

2 Related work

In this section, we review previous visualization approaches, techniques and applications related to our work. First of all, this paper builds on the work presented in [18], where a visualization approach allowing to visually compare data with time periods has been proposed. The Time∘diff approach is able to visualize the amount of data, the changes in the data, and the amount of changes in the data at the same time in a very concise way. We extend this work in several directions. First, we make the visualization approach interactive by applying VISM. Second, we add a multi-viewpoint visualization that is able to show different dimensions of the data in the time period under study. Third, we implemented a prototype and evaluated it using experts from the eye tracking domain.

Research work on visualizing and comparing period data can be found in different research areas, such as temporal databases and visual analytics.

2.1 Temporal databases

In the area of temporal databases, most of the past research works focused on the storage, representation, and management of temporal data with period timestamps. Work on the visualization of temporal data focused either on data with associated time points, such as time series data, or on the visualization of queries and patterns in the data. The visualization of period data itself received scant attention. Defining a proper visual representation for period data needs a review and an integration of the work coming from the different areas mentioned above. In the context of temporal data, visual representations of queries [13] and patterns [14] have become more and more used in combination with traditional SQL [40], but none of these works address the visualization of the data itself. Time was also incorporated in the relational model by extending the Entity-Relationship model [25] as well as extending SQL [27, 32]. For these works, the visualization is related to the schema of the data for modeling queries rather than visualizing the content of the data. In [3], the importance of choosing and parameterizing a visualization method with respect to a given time characteristic is highlighted. As for traditional data, visual representations and metaphors were introduced for clustering the results of temporal queries and patterns [5, 26, 27, 42], rather than for visualizing the initial data and output data. PlanningLines [2] is a work that focuses on visualizing period data. In contrast to our work, it does not focus on how to visualize the periods, but on how to visualize uncertainties within periods. Work on point data or time series data is more vast, and different approaches have been introduced, such as Spirals [52] and TimeWheel [51]. There exists also work that combines temporal and spatial data STV (Spatio-Temporal Visualization), such as the system by Chen et al. [11] that integrates three visualization techniques: a GIS view, a timeline view and a periodic pattern view. Despite a prevalence of work on point or time series data, it is worth mentioning that data with time points, such as log data, may also be associated with periods by compressing consecutive points with equal or similar values into periods [6, 45]. Finally, the VISOR tool [37] based on optimal k segments [38] helps users to explore data and their summary structures by visualizing the relationships between the size k of a data summary and the induced error measures.

2.2 Visual analytics

In the area of visual analytics, a greater number of publications on visualizing and comparing time-referenced data are available in comparison with the works found in temporal databases. However, the majority of work is focused on data with associated time points, such as time series data. A survey of different techniques for the visualization of time-series data can be found in [49]. Different techniques, e.g., Fisheye TimeSlider, and visualization patterns, e.g., ProcessLines, presented in [35] could be interesting for future work on our research, in particular towards the implementation of an interactive interface. Although period data can be derived from time-series data, techniques for visualizing time-series data are not the focus of our paper.

Some work is available on specific visualization of period data. In Lifelines [42], a timeline visualization for patient histories in a 2D layout was presented. LifeLines provides a compact hierarchical timeline visualization for personal histories but does not support finding patterns among multiple entities. The timeline visualization can be a possible approach to investigate in our project. AuctionExplorer [47] is a suite of tools for collecting, processing, and interactively exploring auction attributes, which can help investors to comprehend the financial market and make a decision, but it cannot give direct visualization of multiple attributes of an entity. These techniques are limited to navigation methods along timelines. Moreover, our project is not aimed to investigate interactive interfaces of period data. Although other authors often use multiple techniques to visualize different attributes of period data, as in CareVis [1], our project aims to investigate specific techniques, such as timelines. The comparison of period data received scant attention in literature as specific technique. Indeed, most of the time comparison techniques are embedded in data visualization systems. Timeline visualizations have been used to show more complex relationships among period data. In [43], stacked overview combined with detail timelines have been presented to orient the analyst to a specific problem. The visualization consists of a series of individual timelines, where each subsequent timeline is a focused region of the previous timeline. In [28], multiple timelines have been proposed that could be stacked and linked to show relationships among events. However, these works focused on showing relationships among period data, while in our project the focus is to show possible differences or similarities among period data.

2.3 Multimedia analytics

Visual Analytics, defined in [31] as “combining automated analysis techniques with interactive visualizations for an effective understanding, reasoning and decision making on the basis of very large and complex datasets” has been recently combined with multimedia analysis. This combination has been defined as multimedia analytics [12, 44]. In [55], a list of pioneer systems in multimedia analytics is provided. It is worth making reference to all the following works, which share some common features with our application. MediaTable [44] allows users to quickly view and categorize large multimedia collections, by combining automatic content-analysis techniques with an interface optimized for visually categorizing large sets of results. sViSIT [39] is a tool for interactive exploring surveillance video data, allowing a simple browsing through the video content by interactive querying and retrieving data. Newdle (News Wordles) [54] explores tagged online news collections based on wordles, its major visual metaphor. In literature, time-based visualizations have a long and successful history [53]. However, the importance of an appropriate visualization and comparison of period data has not been sufficiently taken into account. The majority of works only represent discrete data points or single data types along a single timeline [4]. Moreover, the use of time as a primary dimension for multimedia analysis purposes is surprisingly little considered. Continuum [4], an example of the few works on that line, is a timeline visualization tool for representing faceted temporal data. It makes it possible to represent and explore hierarchical relationships in temporal data, also by expressing relationships between events across periods. Moreover, in [41] a tool called CyberForensic TimeLab (CFTL) has been presented. CTFL is able to scan a disk image for timestamps and then visualizes them on a graphical timeline. This allows the examiner to see what evidence of different types occurs within the same time frame. Although these last works are focused on the temporal aspect of the data, they show a lack in the comparison of data intervals extracted from the data. To the best of our knowledge, our application is one of the first attempts able to summarize the comparison of data intervals and to exploit such an overview as the starting point of an interactive data exploration.

2.4 Visual exploration of data

In information visualization, Shneiderman’s formulation of the visual information-seeking mantra ’Overview first, zoom and filter, then details on demand’ [48], is a well-known visualization paradigm. It is able to summarize different visual design guidelines, also providing a general framework for designing information visualization applications in a wide range of research fields [7, 10, 15,16,17, 23, 34].

In such a top-down approach, the ’Overview’ stage is meant to obtain a global view of a situation featuring a domain of interest. Shneiderman argues that by using it, a user can gain a general idea of the entire collection of data and, at the same time, such an overview can help users control the contents of the detail view. As for the subsequent ’Zoom and Filter’ stage, the goal is to focus attention on a reduced portion of the whole space, by employing tools which allow users to control the zoom focus and the zoom factor. In the final ’Details on Demand’ stage, detailed information about a particular object or group of objects is obtained. According to Shneiderman, once the entire collection has been trimmed to a minimal subset, users could easily browse details about either a selected group or a single item, e.g., by clicking on the selected area and getting details in a pop-up window.

3 Motivation and problem definition

In this section, we first introduce a case study about an eye tracking application, which is used as motivation and running example throughout the paper, followed by a description of the problem setting.

3.1 Running example

As a running example we consider an “Industrial Usability of Eye Tracking for Manufacturing and Design in SMEs” (EYE-TRACK) project, which aims at assessing the potential of bio-metric systems and eye-tracking devices as a support tool for the design, evaluation and manufacturing of industrial products. More specifically, the study from which eye-tracking data is extracted is concerned with people’s behavior towards more or less sustainable products, which are displayed as static images. The study is fully described in Maccioni et al. [36]; here, many outputs from eye-tracking specific software are manipulated, but areas of interest (AOI) are not dealt with. Therefore, the mentioned study is based on phenomena regarding the observation of products as a whole, without considering the influence of details, parts and portrayed pieces of information, which can result beneficial for more thorough analyses in the future.

For this purpose, 43 participants were asked to observe 40 images of different products, such as the one shown in Fig. 1a. For this experiment, product pictures were assigned different areas of interest, featuring pieces of information. On average a product was given 10 AOIs. The AOIs of our example product from Fig. 1a are highlighted in Fig. 1b. For instance, the AOI highlighted in blue represents the logo of the product, and the AOI indicated in red represents a description. Then, with the help of a Tobii Pro X2-60Footnote 1 remote eye-tracker a total of 5M data entries with a sample rate of 16ms were collected. Each data entry records when a participant was focusing on a given AOI.

Fig. 1
figure 1

Example product (with anatomized brand name) and the relevant areas of interest (AOIs)

Full size image

A first step was to merge consecutive samples during which a user was observing the same AOI over consecutive data entries. This process reduced the data size from 5M to 53k, since users were observing an AOI for several consecutive units of the sample rate before moving on another one, in areas that were not featured by any AOI or out of the picture perimeter. Of course, the latter two conditions were irrelevant for the experiment.

3.2 Challenges and current limitations

Figure 2 graphically illustrates on a horizontal time line an excerpt of the data after it has been transformed into periods. For a better visibility, we use small integers instead of milliseconds for the time periods. Each tuple has a unique identifier (Id) and records the time period from TimeStart to TimeEnd during which a participant (User) has observed a given AOI of a product. The time periods are grouped by AOI (logo, description, and picture), and users are shown in different colors, i.e., blue for user U1, green for user U2, and red for user U3. The tuple identifiers (Id) are given at the beginning of each period. For instance, the tuple with Id= 1 records that user U1 observed the AOI Logo of the product from time 0 to time 6. The tuple with Id= 2 records that user U1 observed the AOI description of the same product from time 6 to 7.

Fig. 2
figure 2

Running example shown on a horizontal time line

Full size image

From such a visualization, domain experts can identify possible regularities among the time periods (as referred to different products and participants), and therefore possible correlations. For instance, we can see from tuples 1, 5, and 8 that all three users focus on the logo first and then on the description (tuples 2, 6, and 9). However, this type of visualization is useful only for very small datasets (11 rows in our example) and does not scale to tens or even hundreds of tuples as in our case. Temporal aggregation [24] provides a way to summarize the amount of data historically. More specifically, temporal aggregation counts the number of tuples in the data at each time point and merges identical results over consecutive time points into time periods.

Figure 3 shows a graphical representation of temporal aggregation applied to our running example in Fig. 2 grouped by Media and AOI. The count is illustrated on the vertical axis. The aggregation produces a compact overview of when and how many users observed a specific AOI of a given product. For instance, the first tuple records that 3 users (Count) observed the AOI logo of the product from time 0 to 3, and the second tuple records that later on only two users where observing the same AOI from time 3 to time 4. This representation gives a time-varying overview on the number of users that observed the same AOI at the same time during an experiment.

Fig. 3
figure 3

Visualization of temporal count aggregation for our running example AOIs

Full size image

However, the graphical representation in Fig. 3 has several shortcomings. First, it fails to show all changes and the number of changes in the data. For instance, Fig. 3b shows that from time 7 to time 11 the number of users is 1, but actually 2 different users focused on the AOI Description, one from time 7 to time 9 and the other from time 9 to time 11. This situation is clearly highlighted in Fig. 2 (tuples 6 and 11) but not made evident by the visualization. These changes are hidden and not visible at all in the graphical representation, since the total number of users in the time interval 7 to 11 is the same. Second, it is difficult to use for a comparison of different information elements (AOIs in our case), since it requires multiple graphical representations as otherwise line segments of different AOIs would interfere with each other. For instance, we need three different diagrams to represent data of Fig. 2. More specifically, Fig. 3a represents data in rows from 1 to 5, Fig. 3b represents data in rows from 6 to 11 and Fig. 3c represents data in rows from 12 to 14.

Other critical issues are related to the analysis of the multimedia collection. As a matter of fact, the above described visualization approach provides the user with a static summary of the data over a specific time interval, thus limiting the possibility of analysis. To this respect, it is worth pointing out that data analyses supported by commercial software applications target the development of illustrations, such as static and dynamic heatmaps, which poorly support quantitative or graphical comparisons among products or participants with respect to AOIs gazed at. Since interactivity is crucial for any analytic interface [55], the analysis of the data could be facilitated by tuning the extremes of the time interval or the time granularity. In the first case, only a subset of tuples in Fig. 3 could be used from visualization. In the second case, for instance, the granularity could be modified, e.g., from milliseconds to seconds.

Moreover, the above visualization approach shows only a single temporal aspects of the data. Different viewpoints on the data may help improve the analysis.

Table 1 shows that for product A0.JPG the number of observations involving the AOIs logo and description is equal to 36.4% but the display time of the AOI logo is very close to 50% of the total. This aspect is even more evident considering the AOI picture. Indeed, the number of observations of the AOI picture is slightly lower than of AOIs logo and description, to an extent of approximately 8.7%. Nevertheless if the observation time is taken into account, the AOI picture has been observed for a shorter time than logo and description, in terms of 27.4% and 18.3% less, respectively.

Table 1 Further data aggregation
Full size table

In summary, the critical issues for the visualization and analysis of period data, based on the above analyzed related work and practical evidence, are the following:

  • Visualization of: historical amount of data, changes in the data, and the amount of changes in the data;

  • Scalability for large datasets;

  • Support for different groupings/datasets in a single chart;

  • visualization of period durations;

  • visualization of frequencies;

  • interactive exploration of data.

4 Period visualization with Time ∘diff

The critical issues of comparing and visualizing period data mentioned in the previous section have been tackled in [18], where a novel approach, called Time∘diff, is presented. Time∘diff is based on the visualization of timebars and is designed to preserve the simplicity of reading, allowing a concise summary of the data and better manage scalability. In this section, we provide an overview of Time∘diff, by first showing how it illustrates the amount of data and the changes. Then, how it illustrates the amount of changes, and finally how it illustrates different grouping for comparison.

4.1 Timebars

At the core of Time∘diff are so-called timebars to visualize aggregation results over time periods [24]. The key features are as follows:

  • It uses different color densities over constant intervals to show the historical amount of data, separations to visualize changes, and numbers to visualize the amount of changes in the data.

  • It switches from numbers to line charts to illustrate the number of changes, and is applicable for different data granularity, such as weeks or months.

  • It uses separate timebars for different groupings/datasets with the same color scale.

Figure 4b shows a basic building block of Time∘diff, called timebar, for our running example AOI description from Fig. 4a. It illustrates the amount of data using color densities and changes by separation of periods, i.e., vertical lines. The operation behind this representation is a temporal count aggregation with change preservation [19,20,21]. Conceptually, a temporal count aggregation computes the number of tuples for each time point, i.e., it applies a traditional count aggregation at each time point. In our case it produces the number of users observing the AOI description at the same time. For instance, at time 6 we have three users observing the AOI description (cf. Fig. 2), or at time 8 we have one person observing the AOI description. This conceptual result produced at each time point is then merged into periods; change preservation ensures that periods in the result preserve the changes from the input. For instance, the period from 7 to 9 is one constant interval (with a count of 1 for each time point) because only the user with id= 6 was observing the AOI. The period from 9 to 11 is a different constant interval (with a count of 1) because a different user (with id= 11) was observing the AOI. This mechanism of change preservation allows us to display change points in our timebar. To calculate the temporal count aggregation efficiently, we use SQL’s cumulative aggregates or window functions as described in [22], where we omit the additional coalescing step to preserve changes.

Fig. 4
figure 4

Timebar bar versus temporal count aggregation

Full size image

A timebar is a visualization of temporal aggregation with change preservation. In Fig. 4b we indicate the constant intervals, i.e., periods for which no change happens in the input. The key idea is to exploit the non overlapping property of constant intervals allowing us to “squeeze” all periods into a single timebar. Change points are the separation of constant intervals at which some changes happened, i.e., either a user stopped observing the AOI, or an new user started observing it. The amount of data is indicated by the color density (values given in the color mapping). In this paper we use blue, but in principle different color scales could be used, such as green-yellow-red like in heatmaps. We can see that this timebar includes all the information from Fig. 3b and, at the same time, it also shows all the changes, such as the change in the interval from time 7 to time 11. More specifically, the last two constant intervals of Fig. 4b represent the same amount of data by using the same color, but they are split by a change point that highlights a change in the data.

We proceed by showing how Time∘diff combines the amount of changes with a timebar. To ensure scalability, there are two different modes, depending on the space for the visualization. Figure 5a shows the first mode with full details. For each change point, a bubble indicates two numbers, corresponding to the number of periods that started and ended, respectively. The numbers in the figure correspond to the number of users that started and stopped observing the AOI description, respectively, from our running example. The change cardinalities from the data can be calculated in a similar fashion as the amount of data using SQL’s cumulative aggregates or window functions, inspired by [22, 56], with the endpoints of the periods as the input.

Fig. 5
figure 5

Components of Time∘diff

Full size image

For larger numbers of changes where the numbers would not be visible anymore, we adopt a more compact representation. This mode is illustrated in Fig. 5b, where instead of numbers, the total amount of changes are illustrated using a line chart on top of the timebar. This mode does not distinguish between changes of periods that started or ended, and does not provide the concrete numbers, but instead illustrates dominant changes at a glance. For instance, at time 7 we have 0 periods that started and 2 that ended, i.e., a total of 2 changes (cf. Fig. 5a). This is visible in the graph by a peak that is highest, since nowhere else the number of changes is higher than 2. Similarly, at time 6 we have one change (smaller than at time 7), and at time 9 we have 2 changes.

4.2 Combining multiple timebars

The result of temporal aggregation with change preservation and grouped by AOIs has been already shown in Fig. 3. We now proceed to show how multiple timebars with change point cardinalities can be combined into a single Time∘diff to allow a comparison.

Figure 6 shows the Time∘diff approach for our running example. We can observe that the graphical visualization of Fig. 6 is able to summarize all information, which in Fig. 3 is represented in three different graphs. By using the color density, we can compare the different AOIs according to their number of observations. For instance, we can see that the logo is observed at the beginning of the experiment, while the picture is only observed at the end. With the change points we can compare the evolution of the AOIs and identify clusters with many changes in a specific AOI. For instance, we can see that the logo AOI had generally changes at the beginning, while the picture AOI had changes only at the end of the period. Finally, we can use the bubbles to identify if a specific change was dominant or not in terms of changes. For instance, we can see that the change at the beginning for the logo AOI was a dominant change, since all the users initially focused on that specific AOI.

Fig. 6
figure 6

Timebars with annotations for multiple AOIs

Full size image

5 An application for summarizing and analyzing eye-tracking data

In this section, we first describe the visual approach chosen as a baseline for implementing the proposed application. Then, we provide details on the software implementation, with a specific focus to the interface.

5.1 Application overview

Our goal is to implement an application that, by combining different data content forms (e.g., text, images, graphics), is able to first provide a visual summary of the data, and then to facilitate the subsequent analysis of the data in an interactive way.

In the previous section, we have described the visual metaphor adopted in our application for representing and comparing time periods, namely the Time∘diff approach. This approach provides the user with a static summary of the data over a specific time period only.

Since interactivity is a key factor in multimedia applications [55], for the second step, we relied on the Visual Information Seeking Mantra (VISM) [48] stated by Ben Shneiderman, namely “Overview, Zoom and Filter, Details on Demand”, in order to add interactivity to the Time∘diff visualization. Our application combines temporal data visualization and interaction techniques to represent and analyze data in an innovative manner. In our application we assigned the “Overview, Zoom and Filter, Details on Demand” operations with a specific behavior. Each interaction provides a context-sensitive meaning and invokes a proper functionality among the ones specified in agreement with the VISM. More specifically, users may zoom and filter the data by acting on the time period width or by changing the time granularity. Moreover, users may obtain details on specific data by clicking on the specific part of the Time∘diff visualization obtained as output of the zoom and filter operations. Finally, a multi-viewpoint visualization approach has been adopted, which is able to provide different viewpoints of the data in a single glance, but always aligned with the time period under study. More details about these features and how they were implemented are provided below.

5.2 Prototype interface

We developed our prototype as a web application. The implementation is based on ShinyFootnote 2, an R package that enables to easily build interactive web applications with powerful dashboards directly in R. We decided to use Shiny to implement the project for two main reasons:

  • it provides a highly flexible and powerful tool for web applications and dashboard generation, ensuring the possibility of extending these applications using CSS, widgets and Java script action;

  • it brings all functionalities offered by R, i.e., a large variety of statistical analysis techniques in combination with flexible and extensible tools and graphical techniques.

As shown in Fig. 7, the interface of our prototype is divided into two distinct areas: the control area and the presentation area. The former, detailed in Section 5.2.1, contains all the controllers of the application the user may interact with. The latter, detailed in Section 5.2.2, contains different kinds of visual elements, namely:

  • the Time∘diff timebar and line chart areas, which represent the time-varying summary of the data;

  • the wordcloud area, which visualizes the frequency of observations (time-independent);

  • the pie-chart area, which visualizes the duration of observations (time-independent);

  • the highlighted AOI area, which shows the product with the relevant AOIs;

The wordcloud, pie-chart and highlighted AOI areas are triggered by the interactions of users with the Time∘diff timebar, and the line chart by using the control area. All these five visual elements represent different viewpoints of the data. By using them, the user is able to view, in a single glance, different data characteristics, always aligned with the time period under study.

Fig. 7
figure 7

Interface structure of our prototype

Full size image

5.2.1 Control area

The control area contains five widgets providing different functionalities. Each parameter modification on the widgets has a real-time effect on the presentation area, where all the visual elements are updated according to the chosen parameters.

The first widget is the image selector (see Fig. 8a). As mentioned previously, in our test we used data related to the EYE-TRACK project experiments, in which 40 different images of products were used. In order to analyze data related to each image, the image selector widget is used to switch among images. The user can choose the image to analyze using a drop-down list. Once the image is selected, an event is generated. The event resets all the widgets to the default settings and then displays the data of the selected image. This allows a global exploration of the entire available data sets.

Fig. 8
figure 8

Image and area of interest selector widgets

Full size image

Recall that each image is partitioned into a set of AOIs. The AOI selector widget (cf. Fig. 8b) allows to select the AOIs of the image. Initially, the widget displays information of all AOIs. By using the AOI selector widget, the user can select a specific subset. To improve the usability, it is equipped with “Select all” and “Deselect all” keys. The third widget is the range selector (see Fig. 8c). Using this widget, the user can focus on a subset of the time period he/she intends to explore.

By acting on the two sliders, the time period can be reduced. This action triggers a real-time regeneration of the visual element contained in the presentation area according to the parameters selected by the user. In this way a user easily analyze the data in a selected time period.

The granularity selector widget, shown in Fig. 9a, allows the visualization at different levels of time granularity. Data can have various levels of granularity, in our case we have chosen four different levels, i.e., 1 millisecond, 10 milliseconds, 100 milliseconds, 1000 milliseconds. Each time the user acts on the granularity, the system produces an event that modifies the visual element contained in the presentation area.

Fig. 9
figure 9

Granularity and counter limit selector widgets

Full size image

During the analysis of the data, the user may not be interested in data that contains few observations. For this reason, the Counter limit widget (see Fig. 9b), was introduced. It is implemented through a slider bar that allows the user to hide all entries that have a number of views lower than the selected threshold. Like the other widgets, when a value has been selected, an event is activated and all the visual elements contained in the presentation area are updated in real time.

5.2.2 Presentation area

The presentation area is divided into five different visual elements arranged on three different lines. As shown in Fig. 7, the Time∘diff timebar and graph line occupy the first two rows, while the wordcloud, the pie chart and the highlighted AOI areas share the third row. As previously highlighted, such visual elements provide users with a multi-viewpoint visualization approach. Each element shows a different characteristic of the data and all the elements are always aligned with the time period under study.

In contrast to the work in [18], in our prototype we separated Time∘diff timebars and line graphs, and integrated them into two visual elements. This allows us to have a more concise comparison since the time bars can be shown more compactly and the line graphs can be visualized using a single chart with different colors.

The Time∘diff timebar (see Fig. 10) shows on the X axis the timeline, expressed according to the selected granularity. On the Y axis the different AOIs are displayed. Each timebar identifies the relative AOI, and the areas with the highest color intensity identify the time period with the largest number of users observing the AOI. In Fig. 10, it is worth noting that some areas turn out to be white, meaning that in the corresponding periods no users observed that specific AOI.

Fig. 10
figure 10

Time∘diff Timebar

Full size image

The Time∘diff line graph (see Fig. 11) shows the time expressed according to the selected granularity on the X axis and the number of changes of AOI visualizations on the Y axis. All lines of the graph are distinguished by color and associated with the corresponding AOI.

Fig. 11
figure 11

Time∘diff Line Graph

Full size image

At the bottom of the presentation area there are three visual elements, namely wordcloud, pie chart and highlighted AOI. The main purpose of the word cloud, popularized by [50], is to provide only a general overview of the data, since they do not allow exact measurements or a comparison of the underlying data [46]. The wordcloud labels in our case are the AOI labels related to a specific image (see Fig. 12a). The weight of labels, which is rendered with different sized characters, is the frequency, i.e., how many times (with interruption) users observed a specific AOI. The second visual element is a pie chart (see Fig. 12b) that shows a different point of view on the data compared to the wordcloud. The displayed percentage is given by the fraction of the time duration for which an AOI was observed compared to the other AOIs. The last visual element is the highlighted AOI (see Fig. 12c). This element displays the product image that is being analyzed, and where the most visualized AOI is highlighted by using a yellow contour.

Fig. 12
figure 12

Different viewpoints of data

Full size image

6 An application example

In this section, we show how our application can be used to analyze a real world dataset, by applying the previously discussed VISM “Overview, Zoom and Filter, Details on Demand” on the entire dataset of the “EYE-TRACK project experiment”. Assume that the user decides to analyze the image G0.JPG, which is selected by using the image selector widget (see Fig. 8a). The system updates all the visual elements according to the data related to the selected image, and sets up all the widgets with default parameters, as shown in a screenshot of our prototype (see Fig. 13). In particular, the granularity is set to the lowest value, 1 millisecond in this case, the range time selector is set over the entire time interval and all the AOIs are displayed. This first output of the application represents the overview step of the VISM. As a matter of fact, the user can see different viewpoints of the overall G0.JPG dataset, both in terms of time and number of AOIs.

Fig. 13
figure 13

Application example on image G0

Full size image

More specifically, from the wordcloud it can be instantly noted that the most viewed AOIs are “Product” and “Description 1”. By looking at the highlighted AOI visual element, the user will see which part of the image corresponds to the most viewed AOI “Product”. This information is shown by highlighting the AOI contour. Finally, by looking at the pie chart, it can be noted that the “Product” and “Description 1” AOIs were gazed at during 75% of the total time. All these insights are immediately available and refer to the whole period considered.

At this point, once the overview is clear, the user may apply the Zoom and Filter steps in order to reduce the search space, both in terms of numbers of AOIs and time. He/she may focus, for example, on the AOIs of greatest interest, such as “Product” and “Description 1”, by selecting them using the AOI selector widget (see Fig. 8b). The application, updated in real time according to the selected parameters, shows only two AOIs in the Time∘diff timebar and graph line (see Fig. 14) and updates the three bottom visual elements accordingly. Now, by looking at the Time∘diff visualization, it is possible to focus the attention no more on the overall period but on specific time periods. For example, by observing the color gradation, the user may notice that the AOI Product has been visualized especially in the time period from 0ms up to 2000ms and subsequently from 5000ms to 6000ms, while the AOI Description 1 has had a peak of visualization in the interval 1000ms up to 3000ms.

Fig. 14
figure 14

Control and Time∘diff areas after a Zoom and Filter operation

Full size image

As shown in Fig. 15, the above insights are even more visible if the time granularity is modified from 1 millisecond to 1 second using the granularity selector widget (see Fig. 9a). In particular, the interest in the two AOIs is diminishing after about 8 seconds of observation. Indeed, there is a frequency of views that is basically similar between the two AOIs until the end of the experiment (from second 7 up to second 17). Therefore the user may apply another Zoom and Filter step and reduce the time interval under observation from time 1 to time 11, as shown in Fig. 16.

Fig. 15
figure 15

Control and Time∘diff areas after a time granularity change

Full size image
Fig. 16
figure 16

Granularity seconds with range selection from second 1 to 11

Full size image

It is worth noting two aspects about the use of the application. First of all, by using the Time∘diff visualization, it is possible to explore data in a very intuitive and simple way by focusing on the most interesting aspects. For example, it is immediately clear from Fig. 16 that the attention on the AOI Product decreases strongly from second 2 to second 4 and then undergoes a strong increase from second 4 to second 5.

Secondly, different functions of the application are useful at different times of exploration. Wordcloud, pie chart and highlighted AOI areas are more useful in the early stages of exploration as they are used to return data statistics. In contrast, Time∘diff timebar and graph line are useful during all the stages of exploration, first to provide a visual summary, then to help the exploration up to some specific data. Indeed, in the details on demand step, the user may click on both Time∘diff timebar and graph line in order to ask for some specific data. As shown in Fig. 17, the user may be interested in knowing specific data about what happens after the spike on the blue line at time 5. By clicking on the corresponding time interval, the number of observations (17) in that interval is shown.

Fig. 17
figure 17

Details on demand step

Full size image

7 A qualitative evaluation

Two scholars (experts from now on) from different scientific disciplines with some practice in eye-tracking measurements and partially involved in the EYE-TRACK project, were informed of our application for analyzing and reporting data related to multimedia collections. We analyzed their feedback after using our application in order to provide a qualitative non-structured evaluation of the developed tool. In particular, they were asked to compare the visualization our tool with the visualization of the eye-tracking software they use. As they can easily figure out how outputs of eye-tracking software would look like, no direct comparison of kinds of outputs was deemed necessary.

The first expert is an Assistant Professor who does research in the field of engineering design and is a beginner in the use of eye-tracking devices for experimental scopes. His main scope for the use of eye-tracking devices in research is to establish how people explore products and their representations. In our case study, his focus was on understanding which product features have attracted participants’ attention and with which priority. The outcomes of the temporal visualization tool are of interest for him to evaluate its potential in terms of extracting more knowledge than what is presented in [36] and, more in general, than what is achievable with traditional eye-tracking software.

The expert was exposed to the outputs of the temporal visualization tool (as in Figs. 14–17) for each of the 40 products that are investigated in [36]. He judged the system innovative in terms of providing a different perspective on summaries of eye-tracking data and specifically AOIs. He stated that common representation methods enabled by eye-tracking software provide an overall view of data accumulated over time, e.g., heatmaps, but fail to consider time-related effects when either single participants or more people are scrutinized. In other words, what is achievable with these systems is an overall representation of what has been mainly observed in a time span, but information about the sequence and time-varying nature of observed areas of pictures is not available.

To this respect, he believes that the Line Graph (as in Fig. 11) is the most useful representation, because it is capable of highlighting diffused people behaviors, e.g., gazing at some kinds of AOIs first before switching to areas providing other pieces of information about the illustrated products. For example, in the described case study, the diffused behavior of observing symbols of brands before sustainability-related information could be qualitatively captured. These typical behaviors, once identified, can be better focused on with the Timeline representation (as in Fig. 10) in order to examine how many participants contribute to the determination of the phenomenon. In particular, it is here extrapolated if some participants are gazing at specific AOIs for a long time or if more people are observing (fixation) the same AOIs for shorter times. The latter can be interpreted as attractiveness of the AOI, although accompanied by limited interest or effort in storing the corresponding information. The same expert stressed the possibility to post-process data, and therefore to extrapolate interesting information offered to users that do not master eye-tracking systems fully. Indeed, once the data are acquired, any insightful interpretation can be made with the developed time-based application. Its capability of merging data related to the same stimulus and different participants enables the organization of experiments in which any number of pictures (or other media), experimental sessions and participants can be managed also when using random orders. To this respect, it was reported that specific eye-tracking software poses some constraints, especially when the experiments are organized in multiple sessions. In addition, it was remarked that arrangement of AOI names in the graph facilitates the focusing on the information-related content of the AOIs, e.g., gazing at brands is interpreted as interest, curiosity for or bond with that trademark. In traditional representations, where AOIs and fixations are highlighted on the corresponding pictures and dynamic phenomena are shown, the analyst is likely to focus on observation trajectories (scan paths) rather than the rationale behind search for information about the product. Still with reference to product-related information, the expert claimed that a major development of the application could be the merging of data from different products that feature akin numbers and content-related typologies of AOIs – a sort of semantics-based aggregation. As this could lead to the automatic definition of observation strategies, he individuated the potential of this application to become a substitute for design protocols involving product evaluations, e.g., [33], which, in the reference field, need speeding up with priority. Eventually, he individuated the main limitation in the disregard of different times spent by participants in exploring pictures; for instance, the duration of stimuli was not set a priori in the described EYE-TRACK experiment. This results in a very large number of downward curves when most of the participants have ended their observation. Consequently, the possibility to consider time as a fraction of each participant’s exploration duration could be advantageously taken into account.

The second expert is a Professor in the field of psychology with an established practice in the use of eye-tracking instruments and a relevant record of studies involving biometric and neurophysiological measures. He uses eye-tracking in research to characterize human reactions to different stimuli. As for the described case study, his objectives mirror those of the research group. However, his attention is particularly devoted to interpret cognitive behaviors, therefore attempting to interpret how the observed information might have affected people’s understanding of the presented products. He was exposed with the same outcomes of the temporal visualization tool illustrated to the first expert. He agreed on the utility of the application and focused on the practicality of showing time-related information in a variety of ways. He also endorsed the usefulness, for certain applications, in showing AOIs as text and outside of the reference picture, although some spark of criticism emerged to this respect. Indeed, the already mentioned scan paths have relevance in psychology and cognitive sciences and should not be overlooked. Consequently, he individuated the possibility to classify AOIs based both on their content and on their position within the picture. Other shortcomings were likewise discussed. For example, the application fails to consider the different size of AOIs and, therefore, the different probabilities of gazing at different AOIs is disregarded. Finally, he added some remarks from a practical viewpoint. As the data are imported from the outputs of eye-tracking software, it is required to define AOIs before the conduction of experiments. A major development could consist in the possibility to create AOIs directly through the developed application. This would make it possible to introduce previously ignored criteria for the subdivision of the picture in AOIs, e.g., based on dominant color instead of information content.

In summary, the experts confirmed that our prototype application helps them to summarize and analyze the multimedia data related to their project. In particular, because it exploits the time dimension as the main driver for the summary and analysis. It can be used as a valuable building block for future development to cover domain related visualizations.

8 Conclusion

In this paper, we presented a novel multimedia analytics application for summarizing and analyzing data from eye-tracking experiments. Our prototype application presents the following features. First of all, the application implements the Time∘diff visualization paradigm, which enables the visual comparison of multimedia data with time periods. It is able to visualize the amount of data, the changes in the data, and the amount of changes in the data at the same time in a very concise way. Second, the application is based on an interactive data exploration approach inspired by the VISM, which allows the user to start from an overview of the data and reach specific details by zooming and filtering the data, step by step. Finally, the application proposes a multi-viewpoint visualization approach of the data, which is able to provide the user with different viewpoints of the data in a single glance. Our application contributes in helping decision makers to easily summarize and analyze information containing time-based data, by exploiting the following characteristics: (i) visualization of the historical amount of data, (ii) visualization of changes in the data, (iii) visualization of the amount of changes in the data, (iv) scalability for large datasets, (v) support for different groupings/datasets in a single chart, (vi) multi-viewpoint exploration of data, and (vii) interactive exploration of data. The usefulness of these features has been highlighted by expert from different scientific areas, who used eye-tracking systems in their research.

In the future, we are planning to enrich the application in order to allow the simultaneous analysis of multiple images with the relative AOIs. This feature would allow to study and analyze the behavior of users on AOI with the same semantic meaning but contained in different images. Furthermore, a usability experiment with a greater number of experts/users is foreseen in order to collect their feedback and allow an interface improvement and a possible addition of new functionalities.