Keywords

1 Introduction

Digital/web search has become the primary tool for most of the information seeking tasks. Most search activity can be classified as basic look-up, searches that are known-item or fact retrieval tasks. Yet, long term search activities, such as exploring new domains of knowledge, are often more complex. Information seeking in such long term exploratory search often involves evaluation, comparison and synthesis of results and iterative querying. However, as expressed by Marchionini such open ended information needs are not well addressed in today’s search engines that are oriented towards high precision rather than maximizing the number of possibly relevant objects [1]. Related to this observation is the questioning of the dominant interaction model for search tools, “query-response paradigm”, and the reevaluation of guidelines for the design of systems to support exploratory search [2].

Typical query-response pair illustrates some common interactive qualities of search interfaces. Consider this sequence of events that occur during a basic look-up task:

A user types a query to a predefined search box in the graphical user interface and submits her query. The system responds by returning a set of results that are displayed as a list, sorted by their relevance as computed by the search engine.

The sequence reveals many aspects of interaction that include but are not limited to (a) the input modality (typing), (b) the initiation mechanism (explicitly initiated by user), and (c) the timing (concurrent), (d) presentation (textual) and (e) layout (list) of search results. In HCI various prototypes and design frameworks emerged that took a different approach to one or more of the various aspects listed above. This paper aims to identify future interactions with a particular focus on information visualization and various input modalities (such as EEG, EDR, fEMG, eye gaze and pupillometry which are currently positioned as peripheral in respect to more established and precise input techniques). Academic search, in which open ended and evolving searches is common, is used as a case.

Input modalities such as gaze and other physiological signals expand the resources available to the system for inferring the state of user [3], increasing the capability of the system for collaboration [4]. In search scenarios such input allows an increased ability of the computer in assigning user responses to information items. The increased awareness of user response also implies possible changes regarding how the user performs queries and how the system presents information in future search interfaces.

2 Querying

A potential outcome of human computer symbiosis in exploratory search scenarios is the possibility of performing search without having to articulate precise queries. In this setting, the system carries out the necessary work of formulating the query by possibly relying on user’s spatial references (pointing), the sequential organization of user actions and query metadata.

Pointing. Pointing, as a spatial referencing technique, allows directing attention to objects by using contextual information and is widely used in graphical interfaces through cursor and touch input. Pointing is also a possible method for facilitating the easy construction of queries as opposed to using typed queries. In most search interfaces pointing is used for selecting specific items, filters or links. Apart from these, several visual interfaces allow pointing to objects such as keywords or document surrogates for open ended querying. IntentRadar [5] features radially organized keywords as a representation of the user query and enables dragging the keywords for directing the search. Apolo [6] enables pointing to surrogates of scientific publications for query, with the system populating the visual interface with similar publications in response. In addition to pointing single items, pointing to multiple items or regions is possible by using interactive visualization and query relaxation techniques [7]. In common, these examples allow the user to form queries by using items that are readily available in the interface and assign the task of formulating the query to the system. On the other hand, possible forms of pointing are not limited to mouse or touch input. Spatial references can take many forms such as different hand postures, gaze and orientation of head, body and arms. Future interfaces can build on these different forms of indications for diversifying the possible modes of explicit inputs and for tracking user’s attention.

Sequential organization. Interaction during information seeking involves consecutive user actions while interacting with the user interface. The sequence of these actions constitutes part of what is called the search context and have been used to infer user preferences and to expand or disambiguate queries. Traditionally, most implicit mining of user data involved actions that are common to graphical user interfaces such as scrolling, selecting, navigating, saving, deleting [8], with few examples using alternative inputs such as gaze [9]. Incorporation of peripheral physiological input, allows more detailed information related to user state and interest level during the interaction, without having to rely on user’s explicit feedback. At the same time, the sequential organization does not always imply relation. Thus, one possible research area would be to identify gaps and adjacency pairs during interaction, or to provide the user the necessary means to designate search sequences.

Query metadata. In addition to sequential organization, the queries can be contextualized by using inputs that are concurrent to the primary input (Fig. 1). These can include both the metadata of the primary input (e.g. the pace of typing when entering a query), or peripheral input that accompany the primary input (e.g. accompanying eye gaze or physiological signals while the user is typing a query). These metadata can be used to clarify user’s query, similar to the role of intonation, pitch and amplitude for interpreting participants’ utterances in daily conversation [10]. An HCI example of using input metadata is registering pressure information while typing on a small keyboard to explicitly control the level of input uncertainty [11].

Fig. 1.
figure 1

Querying actions like typing can be contextualized by using immediate or concurrent input. The figure shows a possible interaction scenario, in which the query box follows user’s gaze input. In this case, the space and the nearby elements constitute part of the context of the typed query.

Full size image

3 Display of Information

Another outcome of human computer symbiosis is adapting the display of information to the user state by relying on detailed information about user’s attention level and response. The system utilizes this information to determine the timing and visual representation of information items.

Timing. In typical search interfaces the display of the results is concurrent: all the results are displayed at once. In contrast, future systems can use time order expressively to highlight and relate certain items within the retrieved result set by the order of their introduction. Gaze input in these cases can be used to detect user’s attention levels, in turn affecting the pace of display.

Besides using timing in response to a query, a radical alternative is to eliminate queries and proactively retrieve information based on user’s changing context. In HCI various design frameworks exist such as non-command [12], mixed-initiative [13] and attentive user interfaces [14] that promote information retrieval without users having to explicitly query, but rather rely on software agents [15]. Such proactive display of information, however, potentially causes the problem of distracting the user from the task at hand. As Allen [16] has noted, as opposed to fixed initiative systems in which the initiation of interaction is well-defined, in mixed initiative systems the agents should decide on the appropriate time of starting the interaction. User actions, as well as gaze and other physiological signals in this case can potentially provide necessary triggers for the timing of information display.

Visual representation. Search interfaces usually represent retrieved set of results in surrogates, namely the representation of original documents. Surrogates can also indicate various user actions performed by the user. Traditionally these indications relied on the explicit input of the user. Typical examples are the change of color for visited Web pages in a search result list and the display of actions such as forwarding, replying and reading for emails. EEG and other physiological inputs provide the opportunity to visually distinguish information items without any explicit input (Fig. 2), by tracking user attention (whether the surrogate is gazed at by the user) and physiological response (how the interaction with the document affected the cognitive load and arousal level of the user). Interaction history can also enable tailoring the layout of surrogates in the visual space by taking the user interaction history into account. A possible example is changing the position of a document in a graph after user interaction, by registering what the user specifically engaged within the document. In this scenario possible clues for the change include sequential organization of search as well as various physiological inputs recorded while the user is engaged with the different parts of the document.

Fig. 2.
figure 2

The document surrogates can be visualized to indicate different attention levels and responses a document goes through. The figure shows illustrative visualizations of a document surrogate which (1) is not yet seen, (2) is seen and (3) has been opened and the document it represents has been read. Different color saturations can show different levels of attention and arousal.

Full size image

4 Discussion: Roles of Visualization

This section identifies possible roles for the visualization of items during search regarding the querying and display of information examined above.

As a resource for pointing. Visual display of information provides set of items that the user can point to for explicit querying. In addition to querying, visualization enables contextualizing the search through sequential organization or query metadata by pointing to the items prior to or during the query. These pointing actions can be used to indicate not only the items themselves but the specific user interaction history with the item (e.g. what the user has specifically found relevant within the item). In this case the registration of user’s previous response allows referring to the past context of interaction.

Orient the user. Search results from various stages of a search session can be visualized to orient the user by juxtaposing newly retrieved information items with familiar ones. Visual representation of the previously interacted items, in this case, indicates how familiar the user is with an item and her past implicit response during interaction.

Prioritize events and information items. Visualization enables prioritizing information items and events by displaying them in different visual areas and with different visual features. Together with timing, visualization enables notifying the user of possible actions that are initiated by the system such as adding, removing or highlighting items.

Represent system estimation of user intention. The systems estimation of user state or intention can be conveyed through visual representation, for the user to make sense of system actions and repair any miscommunications.

5 Conclusion and Future Work

The paper identified potential design directions regarding querying and display of information in future search scenarios by focusing on new input modalities and information visualization. These design directions also described how human computer symbiosis can be furthered with the increasing role of computers for formulating queries and adapting the display to the user. The review of the above interactive aspects is by no means exhaustive but is intended to spark a discussion for future design. Future work will focus on prototyping these different dimensions for realistic search scenarios.