1 Introduction

Word search is a common user operation in a wide range of document readers and text editors for various kinds of files, such as documents, emails, and source code. Typically, the system starts searching when the user provides the keyword. Since the word may appear in multiple locations in the document, the user may jump between occurrences of the word to locate the target context. Manually checking the occurrence of each word is highly inefficient. Hence, organizing and visualizing search results to facilitate users’ searching have become an important issue.

Fig. 1
figure 1

a−d Windows for word search in various conventional software systems. e Our interface with the proposed structured list view integrates document structure and full sentence snippets into search windows to enhance word search

Full size image

The simplest way to show the results of a word search is to highlight them in the document directly and pinpoints the word occurrence locations (see Fig. 1a). The occurrences of keywords are distinguished by simply modifying low-level visual features or attaching visual markups. Strobelt et al. (2015). However, the locations must be checked sequentially to find the target since users cannot get an overview of the results.

A significant number of techniques for visualizing textual search results are based on list (Hearst and Pedersen 1996; Dumais et al. 2001; Hoeber and Yang 2006a, b; Gomez-Nieto et al. 2013). Listing the search results in a separate panel can help minimize the number of eye movements while shifting through the results in the document. Figure 1b shows the Heading view which presents a table of contents and highlights section titles that contain the search words, where the users can click on the title and reach the respective search result in the document. Egan et al. (1989) show that such interfaces enable users to find information relevant to the search topics faster and more accurately, notwithstanding users cannot directly access the search results.

Similarly, the Page view shown in Fig. 1d visualizes search results as a page list, where each page containing the search words is shown as a thumbnail, and the search words are highlighted in the main document view. Although the Page view in macOS Preview provides additional information in the search results (i.e., page numbers, number of word occurrences on each page and a summary of the first matching result on the page), the user still needs to check through all the word occurrences in the main document view to locate the target context.

Rather than ordering search results based on headings or page numbers, other interfaces directly show the search results as a list of proximal text around the search word for each word occurrence in the document; referred to as the List view (see Fig. 1c). Since users can see the partial local context for each word occurrence in a list, they can filter out irrelevant results as earlier as possible before dive into the main document, which saves a lot of effort.

However, doing so requires a selection of suitable partial contexts for all word occurrences, where existing methods simply take a few words before and after the search word. Such contexts are known as search snippets (Tombros and Sanderson 1998). Several snippet construction strategies have been introduced. As shown in Fig. 1c, Microsoft Word puts the search word at the center and takes around 50 characters before and after the search word while stopping earlier if it reaches the beginning/end of a paragraph.

Various visual interfaces have been developed, combining different organization methods and snippet construction strategies, see Fig. 1. While these interfaces are used on a day-to-day basis by a massive amount of users, we found that there are only very few empirical studies on the design and presentation of such search results. In this work, our goal is to fill this gap.

While there is little work on word search in documents, there are several studies related to web search, a similar problem (Dumais et al. 2001; Käki 2005; Clarke et al. 2007; Hearst 2009; Feild et al. 2013), explore the effectiveness of different visualization designs for examining web search results. Results from this line of work show that: (i) augmenting the search result list with a hierarchical category structure of web pages produces better performance (Dumais et al. 2001; Hearst 2009), and (ii) presenting the entire sentence as the search result usually increases the effectiveness of the search; however, long sentences are not generally preferred (Aula 2004; Clarke et al. 2007; Feild et al. 2013) by users. Motivated by these findings, we discuss the advantages and disadvantages of existing text visualization design for word search interfaces and develop two design goals for improving their efficiency:

  1. 1.

    Minimizing unnecessary user actions on the exploration of search results; and

  2. 2.

    Optimizing the amount of on-screen search information that the users can see.

In this paper, we present two new text visualization techniques for these goals. First, we integrate the Heading view (see Fig. 1b) and List view (see Fig. 1c) strategies while minimizing irrelevant and redundant information presented in these two views. Specifically, we show the result list as the leaf nodes of the hierarchy of headings and remove the section titles if they are irrelevant to the search term; this is referred to as structured List. We hypothesize that the user can learn if a search result is relevant in the search window without going through every occurrence in a section. Duplicate search results in the List view, which are generated by more than one search term in the matching sentence, are filtered out. Second, we propose to use search snippets that cover the full sentence of each matched result, as suggested by web search practitioners (Aula 2004; Clarke et al. 2007; Feild et al. 2013). We assume that such full sentences will help users determine if they contain the relevant target information, although they occupy more space than others.

We developed three new interfaces to test these ideas by combining the proposed two techniques and configuring them with existing ones. Figure 1e shows a screenshot of the interface generated by combining these two techniques. We conjectured that these two new techniques might help people effectively locate search targets. Together with a few interfaces of existing software tools, we evaluated seven interfaces through a 35-participant user study in terms of accuracy, time, and eye tracker data.

Our results show that our structured List view can significantly improve efficiency and accuracy; while the full sentence snippet is helpful, the improvement is marginal. Most participants also preferred our new interface since a structured List provides more context to the document structure and facilitates structure-based navigation.

2 Related work

Existing related work can be divided into text visualization, search interfaces, and snippets.

2.1 Text visualization

The last decade has seen a growing interest in text visualization and visual text analysis in areas ranging from digital humanities to large email databases. Visual document analysis brings the human into the loop by providing visual interfaces that help users to explore documents. Most search engines employ text-based visualization of search results, whereas advanced search interfaces may use graphical text visualizations at the same time. In this paper, we pay more attention to text-based visualization, which provides a more accurate visual representation.

The most straightforward text visualization technique is text highlighting which emphasizes certain types of words by augmenting visual features of the text or attaching visual markups to them. This technique plays an indispensable role in close reading (Jänicke et al. 2015). Alexander et al. (2014) encodes the importance of words by changing their background transparency of them. Rectangular blocks are used to visualize the spoken length of syllables and poetic feet in the Myopia Poetry Visualization tool (Chaturvedi et al. 2012). The Baseline view in Fig. 1a adjusts the background color to orange to distinguish the search words in the document, while the location user focused on is colored with a gray background. Nevertheless, in the word search, the user needs to check the occurrences of each word in turn to find the target location, which makes the search relatively inefficient when the document content is very long.

Given a collection of texts, the most common way to visualize them is to list them directly in a panel. Most of the interfaces in Fig. 1 are based on list. As the most widely used visualization method in web search and word search, it requires no substantial additional effort from users to become familiar with it. Although users have to scroll the list when too many results are displayed, it still shows good performance compared with the text highlight technique by providing an overview of the search results, since users can skip some unnecessary results as earlier as possible without dive into the main document view. However, simply listing snippets provides limited information. By incorporating a table of contents into the list and placing the list on the side of the main document view, our interface can help users to locate the target faster and more accurately.

More sophisticated visualization techniques also exist. Word clouds, also known as tag clouds, spatially arrange the words on the canvas (Wang et al. 2017, 2019). While they do provide some improvements in summarizing descriptive information, word clouds slow users down when they need to retrieve specific entries (Kuo et al. 2007). On account of this limitation, we do not adopt this technique.

2.2 Search interfaces

Over the past decades, research on visual interfaces for web search received extensive attention and significant progress in various research fields (Hearst 2009; Wilson et al. 2010). Wilson et al. Wilson et al. (2010) presented a taxonomy of advances in search interfaces, for which the integration of document classification with search results is the closest to our work. The benefits of augmenting the search results with classification labels have been demonstrated in several works (Chen et al. 1999; Drori and Alon 2003; Dumais et al. 2001). However, the classification of general web search results is nontrivial due to their unstructured characteristic, so some automatic and semi-automatic classification methods (Stoica and Hearst 2004; Efron et al. 2004) have been proposed.

Unlike web search, which finds target information among an unstructured set of documents, word search in a document is inherently supported by the document structure, which is the table of contents. However, only a few studies have evaluated the effectiveness of different interfaces for organizing the word search results. Egan et al. (1989, 1991) developed and compared two interfaces, SuperBook and PixLook, which are similar to the ones shown in Fig. 1a and b. SuperBook organizes the search results in terms of the table of contents, whereas PixLook visualizes the search result as a list. Study results show that SuperBook improves the performance and accuracy by \(25\%\) for the search task, while PixLook performs better for other tasks such as displaying the target pages since it allows users to access the search results directly. In this paper, we propose a new interface that combines the table of contents and result list and compares it with various interfaces shown in Fig. 1.

The thumbnail view is another commonly used interface technique that gives an overview of relevant document pages through a separate window. Clicking on a thumbnail brings us immediately to the respective page in the main document view with the highlighted search words. Microsoft Word and Mac Preview provide such user interfaces for the word search task. Cockburn et al. (2006), Gutwin et al. (2017) ran a field experiment of thumbnail views for document navigation. Their results show that a spatially stable thumbnail view enables faster page search than the other techniques. As far as we know, this idea has not yet been evaluated for the word search task.

When the search returns many results, limited screen space becomes an issue. Egan et al. (1989) introduced a fisheye view for the table of contents so that varying levels of detail in chapters and sections can be expanded. Similarly, Paek et al. (2004) proposed WaveLens, which combines a fisheye lens with the search result list to show more descriptive text in the search results. Although the participants performed faster and more accurately on the search tasks with WaveLens than on a static result list, it has focal disorientation and focuses targeting issues (Gutwin 2002). Due to these drawbacks, we do not adopt fisheye views in our interfaces.

2.3 Search snippets

A variety of web search engines were developed to help users find the target document from a collection of documents on the website. Given the search words, a web search engine returns a list of search results, each with a summary of the result page, referred to as a snippet. Generally, a search snippet is a query-biased summary (Tombros and Sanderson 1998) derived from a document, which shows context such as sentence fragments around the search words as a preview. Users can see the relevant local context on the strength of this preview format and decide whether to visit the page (Clarke et al. 2007; Feild et al. 2013).

Existing snippet technologies can be roughly divided into three categories: text-based, image-based and plot-based. The most widely used is text-based snippets, which directly summarize the results in the form of texts, providing a more precise abstraction than pictures. It has the most extended history and continues to play a significant role in today’s search interface. Many interfaces provide textual summaries of individual documents in a list and then leverage a variety of technologies to augment the list (Kuo et al. 2007; Clarkson et al. 2009): Tile Bars (Hearst 1995) and Hot map (Hoeber and Yang 2006c) displayed the results in a list and placed an item before each, while each item is a summary visualization of the document entry. Image-based snippets bring visual information into concern which replace textual snippets by thumbnails (Woodruff et al. 2001; Lam and Baudisch 2005; Cockburn et al. 2006). Page view shown in Fig. 1d makes use of this technique. They are great for supporting the re-finding of previously seen pages (Teevan et al. 2009), whereas the performance in finding new documents is arguable and takes up a lot of screen space. By means of depicting search results through graphic plots, various plot-based snippet techniques were developed (Nguyen and Zhang 2006; Dörk et al. 2008, 2012). Typically, plot-based snippets are used in conjunction with text-based techniques since plot-based techniques often fail to capture the local context of search words in the document. Without word context, users need to visit the possible target documents to determine if they are what the users want, which will severely affect the accuracy of the word search.

The construction of readable text-based snippets involves a trade-off between showing contiguous sentences to aid the result comprehension and consuming less space to allow more sentence fragments to be shown on screen. Some studies (Aula 2004; Rose et al. 2007; Kaisser et al. 2008) suggested that showing full sentences is better than sentence fragments, but long sentences are not preferred. In general, web search users prefer short snippets that indicate the relevant page information.

In the case of word search in a document, the snippets only need to show the matching results rather than summarizing the local document context. Almost all existing document processing tools construct snippets by showing a fixed number of proximal words around the search word. This strategy might result in sentence fragments, which are hard to read. Motivated by web search studies (Aula 2004; Rose et al. 2007; Kaisser et al. 2008), we suggest showing full sentences of the matched results and comparing this technique with strategies being used in existing software.

3 Our approach

Word search in a document involves a manual process of finding the user-targeted document location among the search results, or word occurrences, reported by the software. This procedure can be characterized as a “query-result-evaluation” cycle (Wilson et al. 2010), where the user formulates a query term, examines the results, and selects a particular result for further evaluation at each iteration until the target location is found.

In the same spirit as the web search interface design (Hearst 2009; Wilson et al. 2010), we attempt to improve the efficiency of this manual process. More specifically, our goal is to minimize the time spent by the user in the query-result-evaluation procedure, e.g., by avoiding unnecessary user actions such as viewing irrelevant and redundant information on the screen.

3.1 Characteristics of existing search interfaces

Before describing our design goals, we first briefly discuss the characteristics of the existing search interfaces. Table 1 presents a summary of four existing interfaces that are predominately being used today: the Baseline, Heading view, Page view, and List view; see Fig. 1a–d, respectively.

In the table, the rows correspond to the four interfaces, while the columns correspond to the interface characteristics.

Notably, the Baseline interface requires the user to check word occurrences one by one to find the target document location, while the heading, page, and list views provide a separate window for users to filter the word occurrences.

Table 1 Characteristics of existing search interfaces: baseline, heading view, page view, and list view
Full size table

Heading view

By highlighting the headings of the sections and subsections that contain the query term, the Heading view (see Fig. 1b) enables users to navigate over the sections in the document with a table of contents. However, it requires the user to review the word occurrences in each highlighted section, like the Baseline. Moreover, it may present irreverent section titles that do not contain the query term, thus requiring users to skip them by additional scrolling; see the example shown in Fig. 2a.

Page view

The Page view provides a thumbnail view with additional information for each page that contains the query term (see Fig. 1d). This approach has a similar problem as the Heading view since it requires users to go through the word occurrences on each page carefully.

List view

The List view is similar to the common web search interfaces, where the results are arranged as a list of search snippets. The snippets provide partial local context, thus enabling the users to more easily determine if a result is related to the target document location. However, existing snippet construction strategies might produce sentence fragments with poor comprehension; see Fig. 1c for an example. Moreover, if a sentence contains the query term multiple times, the sentence will appear multiple times in the search window; see Fig. 2b, where we search for “sun”, and the same sentence appears three times.

Fig. 2
figure 2

a The Heading view may show many irrelevant section titles in-between the highlighted titles. b The List view may show duplicate snippet for the same sentence, if the sentence contains the query term multiple times

Full size image

3.2 Design goals

The “query-result-evaluation” cycle (Wilson et al. 2010) requires the users to spend a considerable amount of time examining the search results, where we can see from the existing software interfaces that the redundant search results, the irrelevant headings in the search window, and the irrelevant words included in the constructed snippets (etc.) would all contribute to slow down the user in the query-result-evaluation cycle. This is because the user would require more time to scan through the information in the search window and scroll through the search information when trying to find particular search results for further evaluation.

To improve the user exploration efficiency, we suggest the following two design goals for search interfaces:

  • Reducing the cost of interaction (e.g., scrolling distance) in exploring the search results; and

  • Providing context with more complete semantic information, while improving the utilization of screen space.

The two goals aim to minimize the user exploration time from different perspectives, one on user interactions and the other on user viewing and evaluation, while strategies to improve user efficiency might contribute to both goals.

To meet the second goal, we proposed two techniques that can provide more semantic information by showing results in the context. At the same time, we made some additional optimizations to reduce the use of screen space, such as only showing a sentence once if it includes many search terms. In the user study in Sect. 4.2, with three measures: scan path length, number of fixations, and mouse movement distance, we demonstrate that our design achieves the first goal of minimizing unnecessary user actions. The study results also show that achieving the two goals improves completion time and accuracy of user search.

3.3 Interface design

In this work, the following two techniques are proposed for the goals.

Structured list

As shown in Table 1, the List view allows users to filter out irrelevant matched results more efficiently by examining the partial document context in the snippets. However, the view does not contain any document structure to help users locate where the snippets are in the document.

Hence, our first technique is to integrate the Heading view and the List view by attaching the list items as leaf nodes in the section heading hierarchy and showing the section headings with the corresponding snippets in a group; see the search window on the left side of Fig. 1e.

We refer to this technique as the structured list, or sList short. By doing so, we should be able to help users to filter out unwanted results early and reduce unnecessary user actions since the user can also see the section titles for the search results. Concerning the second design goal, section titles that do not contain the query term are suppressed in the view to optimize the screen space utilization. Furthermore, if a sentence contains the query term multiple times, we show only one snippet for the sentence and highlight all word occurrences (i.e., the query term) in the snippet.

Full sentence snippets

Second, motivated by the web search interface design, we use full sentences as the snippets for the search results and refer to these snippets as the full sentence snippets, or fs for short. In this way, the user can see a more complete local context to determine if a search result is relevant to the target document location that the user wants to find. Although it might result in a few long sentences and consume more screen space, we presume that slightly more scrolling in the search window is generally faster than checking every search result in the main document view.

Indeed, both techniques aim to show search results with more context information. The sList view presents the snippets in groups with associated document sections, while the full sentence snippets provide more complete context, allowing better comprehension.

Compared with the Heading view and List view, each technique takes more space in the search window but might expedite the search tasks and improve the user accuracy; see our experimental results in Sect. 4.2. Figure 1e presents a screenshot of our interface with the two techniques.

4 User study

In this section, we describe our user study to evaluate the effectiveness of different visual interfaces for word search.

Table 2 Search interfaces employed in our study
Full size table

4.1 Study design

We prepared a set of search tasks for two types of documents: Paper and Novel on seven interfaces using different compositions of searching techniques. We conducted a within-subject study in which all participants were exposed to all conditions. While we do not treat the task as an independent variable in our study design, we added necessary variations to avoid learning effects. To examine how our techniques, sList and fs, impact the word search effectiveness. We raised the following three questions before conducting the study:

  • Q1 (Result Organization Strategy): Which way to organize the search results is the most effective and preferred by users?

  • Q2 (Snippet Construction Strategy): Which kind of snippets is the most effective and preferred by users?

  • Q3 (Organization vs. Snippet): Which of our proposed techniques is more important for effective word search: result organization or snippet construction?

4.1.1 Participants

We recruited 35 participants (20 males and 15 females) aged from 22 to 34 (median 25). All participants have normal or corrected-to-normal vision. They are from different majors, including computer sciences, biology, math, and art. They all passed the College English Test (CET) and were used to reading electronic documents. Among which, 28 participants have at least ten years of experience using word search in Microsoft Word, Adobe PDF, or Mac Preview.

4.1.2 Apparatus

We used a desktop computer with an Intel i7-8700K 3.7 GHz CPU, 16 GB RAM, and two 27’ LCD displays with a resolution of 1920\(\times \)1080, one for displaying the tasks while the other for providing the document and search interface to do these tasks. Meanwhile, it was also outfitted with the Tobii X2-60 eye tracker, calibrated for each participant.

4.1.3 Search interfaces

To investigate the effectiveness of our techniques and compare them with the existing interfaces, we use seven interfaces with different compositions of search techniques as listed in Table 2.

Fig. 3
figure 3

The three interfaces with our proposed technique(s): a List+fs; b sList+ms; and c sList+fs

Full size image
Fig. 4
figure 4

List views with different snippet construction strategies: a List+ms; b List+fs; and c List+as

Full size image

Among them, the Heading and Page views refer to the ones shown in Fig. 1b and d, respectively, while List+ms and List+as refer to the List view with snippets constructed by Microsoft Word and Adobe PDF. Thus we directly used the theme in our study to test these four interfaces.

The last three interfaces (see Fig. 3) listed in Table 2 involve our techniques: (i) List view with the full sentence snippets (List+fs), (ii) sList with the same snippets as Microsoft Word (sList+ms), and (iii) a combination of the structured list and the full sentence snippets (sList+fs).

Note that List+as constructs the snippets by putting the query term as the first or the second word and selecting around ten more words after the query term. Figure 4 compares the snippets provided in the three list-based views: List+ms, List+as, and List+fs.

4.1.4 Tasks

Since the full sentence snippets might take up more screen space than others, we collect two types of documents (Paper and Novel) to investigate how this factor influences the search efficiency. Table 3 summarizes the differences between them, where we can see that the Novel document generally has shorter sentences and more paragraphs than the Paper document, while the TOC depth of Paper is more profound than that of Novel.

We prepared 14 tasks for each document (two distinct tasks for each of the seven interfaces). Each task includes (i) a question that requires the participant to answer by searching through the given document, and (ii) a keyword in the answer for searching; see Fig. 5 for an example task. Particularly, to evaluate the effectiveness of the various interfaces, the number of word occurrences for each task varies from 9 to 36, so the user can not quickly locate the answer without going through the items in the search window. Furthermore, the question in each task can be answered with the exact sentence from the documents so that we can calculate the accuracy by comparing the participants’ answers and those standard answers.

Table 3 The average sentence length (number of words), the average paragraph length (number of words), the number of paragraphs, and the table of contents (TOC) depth of the two documents: one Novel and one Paper used in our evaluation
Full size table
Fig. 5
figure 5

An example task in our study, the participant needs to search the document for the answer with the given keyword

Full size image

4.1.5 Procedure

At the beginning of the study, we briefly introduced the study goals and procedure, then calibrated the eye tracker for each participant. We have the following procedure in the study:

  1. 1.

    Conduct the training session

  2. 2.

    Perform part 1 of the study with the novel

  3. 3.

    A ten-minute break

  4. 4.

    Perform part 2 of the study with the paper

  5. 5.

    A questionnaire and a short interview.

The orders within (2−4) were counterbalanced to avoid systematic bias through potential learning effects.

During the training session, a third document, an eight-page research paper, is used to instruct the participants. After explaining how to use the interfaces, each participant was asked to complete several search tasks using each of them.

In both parts of the study with different types of documents, the tasks are randomly assigned to each interface. We also counterbalanced the order of interfaces and document types in the study to reduce the learning effect. For each interface, the participant was given two distinct tasks. For each task, the participant was asked to seek the answer with the given keyword. The participants recorded their answers by copying and pasting them to another word document, which clearly stated the question and the interface ID. Besides, we recorded each task’s completion time (in seconds, the clock stopped when the participants started to write their answers). We also monitored the eye movements and fixations on the study interfaces using an eye tracker and recorded the mouse actions for each task, such as the distance of the mouse movement.

After completing all tasks, the participants were asked to complete a questionnaire to provide summary feedback on these interfaces, including a final ranking of the interfaces, an explanation for the ranks, and suggested improvements.

4.2 Study results

In this section, we present our study results by analyzing subjective and objective measures, including search time, accuracy, and several measures of user interaction. Specifically, we examine whether our proposed techniques can improve accuracy, reduce search time, reduce user interaction and improve user satisfaction.

Fig. 6
figure 6

a The average answer correctness. b Mean values and deviation as 95% CIs of the completion time for each condition

Full size image

4.2.1 Accuracy

We first counted the questions that were given up by participants and found that participants gave up less than 0.5 of 14 questions per condition on average. For the left questions, we evaluated the accuracy and got the answer correctness of each interface as shown in Fig. 6a. We can see that the correctness of three variants (sList+fs, sList+ms, and List+fs) of our techniques is higher than the others, but there is no significant difference (\(F(6,7)=1.33\), \(p=0.35\)). On the other hand, we found the accuracy on the Novel is significantly higher than the one on the Paper, verified by the t-test: \(t(6)=4.54\), \(p<0.01\). This is reasonable because the Novel is easier to understand due to its shorter sentences.

4.2.2 Completion time

Figure 6b shows the effect sizes and confidence intervals for each condition. The result shows that the interface sList+fs and sList+ms perform significantly better than the other methods, especially on the Novel document, followed by three list views which perform similarly in both documents. The Heading and Page views are the worst on the Paper and Novel, respectively.

The first analysis is to analyze how different factors influence the completion time of tasks. We performed a 4 (Heading vs. Page vs. List vs. sList) \(\times \) 4 (No snippet vs. ms vs. as vs. fs) \(\times \) 2 (Novel vs. Paper) ANOVA. Note that the Heading and Page views are regarded as interfaces without snippets. The result shows that the organization factor has a significant effect \(F(3,477)= 9.97\), \(p<0.001\), whereas the other two factors do not have a significant effect. To determine which organization method is significantly different from each other, we further conducted a Tukey post hoc test and found that sList leads to less time than the other (all \(p \le 0.01\)), whereas the other organization methods (heading, page and list) has no significant difference with each other.

To learn which one between structured List and full sentence snippets takes a more critical role in the completion time, we performed two t-tests, first comparing sList+fs to List+fs and then comparing sList+fs to sList+ms. The first test shows that sList+fs is significant faster than List+fs, \(t(69)=3.24\), \(p=0.001\), while the second test shows that full sentence snippets yield slight performance improvement, \(t(69)=1.08\), \(p=0.28\). These tests further demonstrate that full sentence snippets do not significantly affect performance.

4.2.3 User interaction

We computed three measures of user interaction for each task:

  • Scan path length: Total distance traveled by the eye when examining search results (in pixels).

  • Number of fixations: Total number of fixations on the document, where fixations are measured by Tobii Studio software.

  • Mouse movement distance: Total distance traveled by the mouse when examining search results (in pixels).

The values for these features for each system are shown in Fig. 7. In combination, these measures estimate the number of users’ efforts to find information. They show that the structured list significantly reduces all three interaction features (all \(F(3,477)>2.92\), all \(p<0.01\)), and the Tukey post hoc tests show that the Page and sList significantly differ in mouse movement distance (\(p\le 0.05\)) and the other variables (\(p\le 0.01\)). In addition, sList and List also have significant differences in the number of fixations (\(p\le 0.05\)). These results further confirm that our structured list view yields significant performance gains.

To learn if the full sentence snippets can reduce user interaction, we performed a t-test between sList + fs and sList+ms on each of the three measures. The results show no significant effect for the full sentence snippets (all \(t(69)\ge 0.88\) and all \(p>0.17\)), although sList+fs performs slightly better than sList+ms.

Finally, we found that the document type (Novel vs. Paper) significantly affects on scan path length and the number of fixations (both \(F(1,477)>8.7\), \(p<0.001\)), while no effect on mouse movement distance exists. We assume that the Novel has a larger number of short paragraphs, causing participants to sample many other sentences/paragraphs prior to finding the target information.

Fig. 7
figure 7

Three measures of participant interaction with each interface: a scan path length; b number of fixations; and c mouse movement distance. These measures are studied on different interfaces and documents, while the mean values and deviation as 95% CIs are shown here

Full size image

4.2.4 Subjective preferences

As noted earlier, each participant was asked to provide a final ranking of the seven interface systems they had used in terms of their overall preferences. The ranking was done in a progressive way that incrementally added the rank of the interface after they had used it. The final relative ranking (1 = best, 7 = worst) is summarized in Fig. 8.

A Friedman test reveals that participants preferred the sList+fs interface to the others (\(\chi ^2(6) = 117.4\), \({\it p} < 0.01\)), where sList+fs led to higher preferences than all interfaces other than sList+ms (Dunn’s post hoc tests: all Z\(\ge \) 3.14, p \(\le \) 0.05). Over half of the participants in the study (57.0%, N=20) preferred sList+fs and over 77% preferred the sList view. Beyond sList+fs, the variants sList+ms and List+fs are ranked as the second and third, where the results of the post hoc testing between each of them and the other four interfaces are all Z\(\ge \) 3.14, p \(\le \) 0.05, indicating a clear preference for the proposed two techniques.

In our open feedback about sList+fs, almost all our users said, “it is easy for me to know which part I am exploring and helps me search results according to the document structure.” One user said, “after giving the question, I can guess which part of the document has the answer and then select some sections in the search window to find answers."

Regarding the full sentence snippets, we introduced this feature to participants during the training but finally found that most users did not use this feature well. One participant mentioned, “after finding a few related keywords from the snippet, I will click the snippet to read it in the main document view.” On the other hand, one participant who uses this feature said, “I can understand the meaning of the search result and even directly find the answer from the search window without going through the main document view.” From this perspective, we believe that if users are used to this feature, the full sentence snippets might have an effect on the search performance. However, some participants also raised the concern that the search results are distributed in multiple sections or subsections, and the document structure will take a lot of screen space in the search result panel. Therefore, we have optimized the presentation of the document structure by showing only lower-level structures.

Fig. 8
figure 8

a The distribution of the number of participants who ranked the interface; b Average relative rankings of each interface

Full size image

4.2.5 Summary

The main findings from this study are that participants:

  1. 1.

    preferred and were more efficient with structured list than other search result organization strategies (Q1).

  2. 2.

    felt that the readable full sentence snippets gave them a complete meaning of the search result but did not help them to be more efficient (Q2).

  3. 3.

    preferred structured list over the full sentence snippet and liked to perform structure-based document navigation (Q3).

5 Discussion and conclusion

Efficient word search is essential for day-to-day office work, which is needed by a massive number of users worldwide. To this end, we were surprised to find very few empirical studies on the design of word search interfaces. In this work, we thus set out to revisit the word search problem and evaluate various interfaces and corresponding text visualization techniques. By analyzing the advantages and disadvantages of existing interfaces, we propose to show search results in context with two techniques: a structured List view presents the snippets in the context of their associated document sections and full sentence snippets to provide a more complete local context for each matched result.

We conducted a user study to assess how well these two techniques work compared to state-of-the-art interfaces. Our findings show that the search process can be more efficient with the structured list view, and 77% of the participants also prefer it. Of the two structured list variants (sList+ms and sList+fs), almost three times as many participants preferred the one generated with the full sentence snippets technique, indicating that the full sentence snippets were also preferred by the participants.

Our findings also show that there is no significant difference between the search snippet construction strategies, although the full sentence snippets seem to be more preferred by the participants. From the post interviews, we found that a few participants still keep the habit of reading snippets in conventional software tools that read the search results in the document rather than perusing the snippet in the search window. Hence, we assume that the advantage of the full sentence snippets technique might not have been thoroughly studied and exploited during our study. Once the users get used to our interface, its benefits might become more apparent. To explore this, we plan to perform a longitudinal study as future work.

Interestingly, the list view in Adobe PDF is ranked the last, whereas its accuracy and performance are not the worst. The post-interview shows that most users assume that its snippets do not provide sufficient local context since the words before the search word are ignored. On the other hand, Egan et al. (1991) showed that the Heading view works better than the List view for word search, which is inconsistent with our observations. We found that their study setting differs from ours, where the keyword is not provided. In fact, our setting is similar to their citation task, where the participants were asked to answer a question with a given keyword. In this setting, the Heading view performs worse than the List view, which is in line with our results for the Paper (see Fig. 6b).

6 Limitations

Currently, only two eye-tracking measures were used: the scan path distance and the number of fixations. In the future, we plan to explore advanced filters to identify the scroll distances in the search window and the main document window, respectively. This way, we can investigate if full sentence snippets might reduce the word search in the document. Second, the folding and unfolding functions of the structured list views were not used during the study. For a document with long headings, they might help to improve the performance further. Third, only controlling the location of search results in the middle of the list may not be enough. Participants might find it easier if the results appear early in the document. We will further study how the search results’ location affects the efficiency in future work. Fourth, currently, only one novel and one paper are studied in this work. More documents could be used to avoid learning effects further. Last, our current evaluation is limited to a single search term. We plan to study search tasks with multiple query terms in the future.