Keywords

1 Introduction

Bibliographic information retrieval systems, such library catalogs, have a long history of being described as difficult to use, time-consuming, producing long results lists users need to shift through to find those most relevant to their needs. With the emergence of the IFLA Functional Requirements models (FRBR) [1] several studies examined the potential improvements in system design, display of results, and hierarchical browsing of work-based information, including exploring relationships among variations of a work and relationships between works represented in the bibliographic data [2,3,4] but most importantly the potential to enhance users’ experience searching and browsing bibliographic data. A few prototypes were developed and tested using FRBR-based systems [5] but very few examined FRBR-based implementations of visualizing bibliographic information [6].

This paper presents a continuation of a study (2012-Slovenia) that examined user performance and preferences between a typical bibliographic information system and four hierarchical visualizations of work-family based bibliographic information.

2 Study Background

The study reported here (2016-United States), utilizes the same prototype designs first introduced in a similar, 2012 study [6], using a similar methodology. Five different prototype designs (Fig. 1) were implemented and tested. Four of them are using different hierarchical visualization techniques to create displays of the same information using the work-family concept: Circlepack, Radial tree, Intended tree, and Sunburst.

Fig. 1.
figure 1

Four hierarchical visualizations and one baseline prototype displays

Full size image

The fifth design implements a typical bibliographic information system with faceted navigation, using edition-based displays of information. This design, Baseline, serves as a typical interface users currently experience, for comparisons to the visualization interfaces.

The bibliographic data included in this study are FRBR-based work family data sets of three works, varying in the degree of a work’s expression and manifestation complexity. For example, one work has only two expressions in English (Destiny of the Republic) and another has multiple expressions, including translations in several languages, other variations, and related works, contributing to the workset’s complexity (Don Quixote), with the third being a children’s fictional work, offering a variety of expressions (Histoire de Babar, le petit éléphant). Only one common work family data set, Don Quixote, was used in both the 2012 and 2016 studies in order to provide comparative data. One of the goals for repeating the study using one common work, was to test the feasibility of designing FRBR-based visualization displays that appeals to users in different cultural environments. To expand the original study, the 2016 study used two new work family sets, one non-fiction with limited complexity and one fictional children’s work in order to test if work complexity has an effect on user’s interactions and perceptions of FRBR-based visualizations of bibliographic data.

The data collection took place in April 2016. The 79 participants were undergraduate students of a large university in the United States, representing different areas of academic study. Each participant interacted with four interfaces, the Baseline and three visualizations. The same tasks for each work (total 10 tasks for each) were randomly assigned to each system interaction. Tasks ranged from simple finding and identifying to more exploratory and understanding questions. Tasks were grouped as work-set related (labeled in graphs as versions), other related works (related), and author-related (author). Sequencing of the interface, work, and task group for each test was designed to avoid order bias. In total, there were 275 valid tests, distributed among interfaces as indicated in Table 1.

Table 1. Number of tests performed per visualization interface
Full size table

Participants were asked to assign a difficulty score for each task they completed (1 = very easy, 5 = very difficult) and rank the interface designs at the completion of the study from their most favorite to their least favorite. In addition to these user-reported measures, researchers recorded time-on-task, completion success, and navigation success for each task.

3 Findings

The study looked at several measures and factors. This paper is limited to summary findings on a number of performance and perception measures. Preliminary comparison findings between the two studies were reported in 2016 [7].

3.1 Performance

Three performance measures are reported here: time it took for each participant to complete each task, how successful each participant was to complete each task, and how successful each participant was in navigating each system, whether through result lists and facets for the Baseline system or the hierarchical navigation features of the visualization systems.

Time on Task.

Overall, participants needed considerably more time to complete tasks in the Baseline system than any of the four visualization systems (Fig. 2). On average, participants needed the least amount of time when using the Indented tree.

Fig. 2.
figure 2

Average (mean) time on task

Full size image

When comparing the time on task by task group, participants needed more time for work-related tasks than related works or author-based tasks. Again, we see that Baseline time on task being highest for all three task groups among all systems (Fig. 2). Indented tree had the lowest time on task for work-related tasks and related works tasks, and Sunburst for author-related tasks.

Task Success.

Participants were least successful in completing their tasks when using the Baseline system. They were most successful when using Indented tree with a close second when using Sunburst (Fig. 3).

Fig. 3.
figure 3

Average (mean) task success for each design

Full size image

Relationships to other works is one of the areas where work-based bibliographic information system design offers improvements over the traditional edition-based bibliographic information displays.

Navigation Success.

Each design offers navigation features to aid users in their tasks. For example, Baseline offers faceted navigation and visualizations offer different hierarchical navigation features for expanding or collapsing displays.

When examining how participants used these navigation features to complete their tasks, we see that they were least successful using the Radial tree followed by the Baseline system (Fig. 4).

Fig. 4.
figure 4

Overall average (mean) navigation success

Full size image

3.2 Perception

Participant perceptions on ease of task and their system preference ranked from the most favorite to their least favorite are reported in this section.

Ease of Task.

At the completion of each task, participants were asked to rate how easy or difficult the individual task was, with “1” being “very easy” and “5” being “very difficult.” Overall, participants rated the tasks more difficult when using the Baseline system and easiest when using the Indented tree (Fig. 5). When comparing task groups, author-related tasks using Sunbursts were rated considerably easier than using any other system, in addition to being rated the easiest among all task groups across systems.

Fig. 5.
figure 5

Average task difficulty

Full size image

System Raking.

At the completion of their tests, participants were asked to rank the systems they interacted with from their most favorite to their least favorite, based on their experiences using each system to complete their tasks. Looking at the interfaces ranked as their most favorite (ranked #1), participants favored Circlepack the most with a close second the Indented tree and Radial the least. According to Fig. 6, when combining the #1 and #2 rankings for each system, Indented tree is ranked considerably higher (67.31%) than other systems, with Baseline ranked lowest (44.93%).

Fig. 6.
figure 6

Participant system ranking

Full size image

4 Discussion and Conclusion

The study examined if there are any differences on performance and perceptions between the Baseline prototype and the four visualizations systems. Overall, the study findings show that the Baseline prototype performed least favorably among all systems on a number of measures, including longest time-on-task, least successful task completion, highest task difficulty, and second to last on navigation success. This later finding is interesting, given the fact that the majority of current systems offer faceted navigation. Further examination of the factors for the low performance on navigation success is warranted.

The study shows significant difference between the Baseline system and the visualizations on the time it took participants to complete their tasks and successful completion of tasks with visualizations performing higher overall. Participants ranked three of the four visualizations (Indented, Sunburst, and Circlepack) higher than the Baseline system based on their experiences. Also, when it comes to more complex tasks of exploring relationships, the Intended tree and Sunburst outperformed the other systems.

The results suggest that, generally, hierarchical organization and visual display of bibliographic information enables users to better navigate bibliographic work families and complete their tasks successfully. Although it was their first interactions with FRBR-based systems, participants expressed their preference for these visualization systems over the Baseline system. These findings are similar to the 2012 studies, although a comparison between the 2012 and 2016 studies show that the was a significant difference in the time it took to complete the tasks, with the US group average time of task completion being longer.

A detailed analysis between the repeated work family in the two studies and the newly added work families in the 2016 study will examine whether some of the differences between the prototype design scores derived due to the different environments or the different work sets. In addition, further analysis is needed to explore the differences in performance and perceptions based on work complexity and task complexity. Findings from these experimental studies of prototype systems will inform better visualization systems design for bibliographic information and improve user experience.