1 Introduction and background

Mobile information appliances (i.e., smartphones) have become an omnipresent means of sending and receiving information and conducting online transactions. Because mobile interactions are often conducted on the go, they are sometimes affected by the presence of “situationally induced impairments and disabilities (SIID)” [28] or, simply, situational impairments. Consider a user, for example, who wants to read a message on their smartphone outdoors as it begins to rain. Because water droplets are both masking content presented via the mobile GUI and affecting the ability to interact with the touchscreen, for the duration of this ambient event or until the user can find shelter for their device, transaction completion is delayed, canceled, or made otherwise less than optimal. As mobile devices continue to offer more advanced functionality, performing simultaneous tasks (e.g., browsing or sending an SMS while ambulatory) becomes more attractive to fit in with users’ busy lifestyles. However, as a result, situational impairments may be faced by larger numbers of users than previously experienced.

While the onset of a situational impairment may simply result in annoyance or transaction failure, encountering one in certain contexts could actually place the user and/or others in danger. Consider, for example, distracted driving (a type of inattention where attention is diverted from the driving task to focus on some other activity), which can be classified as visual (requires one to look away from roadway to visually obtain information), manual (requires one to take hands off the steering wheel and manipulate a device), or cognitive (mental workload associated with a task involves thinking about something other than driving) [17]. The task of attempting to type an SMS or email message while driving would be an illustration of all three types of distraction. Even hands-free interaction (i.e., using Bluetooth to speak or input text) can still be considered a cognitive type of distracted driving [11]. Not being able to read a message may be annoying. Not being able to conduct business can affect one’s bottom line. However, considering that in 2015, an estimated 391,000 injuries in the USA involved distracted drivers [16], the need for creating a remedy for situational impairments is clear, as these may have contributed to the number of injuries.

Adding to the issue of attempting interactions in the presence of situational impairments is a special category termed “Severely Constraining Situational Impairments (SCSI),” which was found to exist within the situational impairment problem space. These severe versions of situational impairments occur when a workaround is not available or easily obtained, or where a technological solution is found that leads to the introduction of a new situational impairment [27]. Consider a user whose device vibrates in their coat pocket on a bus filled to capacity with passengers, while holding a portfolio in the one hand and a handrail with the other. The vibration is the result of an SMS that requires timely attention. The user cannot access or respond to the message with their primary modality (hands/touch) due to the hands not being available, as well as the crowded nature of the environment. They deploy an existing technological solution (viewing the text on their smartwatch) but cannot send a reply using an alternative modality (voice) due to the ambient noise present. By the time they can arrive at a state where they are able to complete the transaction, the value of the information becomes meaningless.

Prior studies have recognized that mobile interaction represents a paradigm shift requiring new interaction rules that may not map well from those established for the stable desktop environment [32]. However, research to date has tended to focus on situational impairments as a general phenomenon or addressing a specific cognitive or physical transaction issue. In addition, the preponderance of research to date has focused on quantitative evaluation, measuring effects, and/or attempting to improve transaction performance. Little research to date has attempted to examine the problem space from a qualitative perspective.

While Saulynas et al. [27] were able to thematically classify situational impairment events thematically and to define severely constraining events as existing beyond simple classification, the steps that users might be attempting in order to complete the interaction (i.e., workarounds) were not explored. In addition, further examination was needed to understand user motivations when choosing to delay or forgo interaction completion even when no physical barriers were present. If, for example, a user was driving and chose not to interact with their device, was that due to (1) concern for safety, (2) concern of legal implications (e.g., police might issue a citation), or (3) acceptance of the norms of their society/culture (i.e., just not the right thing to do)?

The authors of this paper, therefore, wish to add to the understanding of the issues affecting users in this problem space at a phenomenological level of depth and focus that has not been much studied by previous research. The severity and added complexity of “Severely Constraining Situational Impairments” suggest that it may be important when designing for mobile interaction to examine whether the user approaches their onset differently from that of a regular situational impairment. In addition, this current study extends the research from Saulynas et al. [27] by addressing the limitations of that study noted above, as well as further clarifying the differences between impairing and severely impairing events. Specifically, this research will attempt to answer the following research questions:

  1. RQ1.

    What are the motivations for mobile device users either attempting or postponing/abandoning a mobile transaction during the onset of a situational impairment?

  2. RQ2.

    What type of workarounds do mobile device users attempt when encountering a situational impairment, and are they different in the presence of a severely constraining situational impairment?

  3. RQ3.

    Can mobile technology design better account for actions attempted and the transactional needs of mobile device users while on the go during the onset of a severely constraining situational impairment?

This paper describes a study that consisted initially of a series of semi-structured interviews with mobile device users. Information was gathered concerning tasks performed using their device, methods for addressing difficulties during a mobile transaction attempt, and motivations for choosing to forgo a mobile transaction even if no physical barrier presents. The data on transaction events and motivation collected during the interviews, along with the corpus of situational impairment events from the study by Saulynas et al. [27], resulted in a set of rich situational impairment scenarios. These scenarios were then used during a series of participatory design workshops, where teams of users and design experts engaged in brainstorming exercises, resulting in a set of user-led recommendations for the accounting of situational impairments of varying complexity in common mobile interaction scenarios. The results revealed that users are highly motivated to complete mobile transactions in a timely manner, even if it may place them (or others) in danger. Also revealed was a desire from users that technology becomes more context-aware to reduce the complexity and cognitive load that situational impairments produce during mobile transaction attempts.

The studying and addressing of situational impairments can: (1) benefit all users, universally, as we all experience limits to our abilities, (2) promote design solutions for what people can do in a given situation, as opposed to what they cannot, and (3) have real, life or death consequences as people’s abilities are significantly diminished by their presence [33]. By obtaining a better understanding of the problem space, this study aimed to produce implications for mobile interaction design that could help users to perform tasks safely and effectively and/or gain critical information when in the presence of ambient events that are so severe and constraining that no reasonable solution attempt offers any value.

2 Related work

The examination of situational impairments usually begins with an acknowledgment of the limited cognitive resources available to humans and the recognition that, in the mobile transaction space, part of those resources must be dedicated to account for context. Context can be viewed as the interaction of (1) the user, their activities, and the social environment, (2) the environment or location, physical conditions, and infrastructure, and (3) the available applications and I/O channels [28]. Four challenges have been noted for humans when attempting interaction while on the go: (1) cognitive load (limited attention resources); (2) physical constraints (non-mobile activities may place constraints on physical resources); (3) terrain (external environment affects how a user will interact); and (4) other people (movement activities often involve a social element) [14]. Lin et al. [13] is an example of a study that demonstrated the effect of allocating resources to external context. In attempting to input data onto a PDA with a stylus while walking through an obstacle course, participant walking speeds were reduced and error rates increased versus walking on a treadmill.

Understanding that context can affect interaction performance, researchers have approached the situational impairment phenomena from a human factors perspective, focusing on identifying areas during a task attempt where performance may be diminished and designing or suggesting the technology that attempts to mitigate the effects. Examples include the study by Goel et al. [5] and that of Kane et al. [9], who developed adaptive text entry techniques and mobile prototypes designed to improve mobile device interaction performance while walking. Barnard et al. [1] revealed that more than just the act of walking can result in sub-optimal mobile interaction. By varying three contextual factors (task type, motion, and lighting level), the researchers’ study demonstrated that varying lighting conditions as well as motion affected mobile task performance. Lee et al. [12] tested the effect of varying two smartphone form factors (width and bottom bezel) on touch behaviors during one-handed interaction. The results showed that task performance, subjective workload, and electromyography all deteriorated with increasing width level and that difficulty increased as the bottom bezel level decreased. To address a similar mobile interaction scenario described in the Introduction section of this paper (attempting a mobile transaction in the rain), “RainCheck” was a prototype developed to account for a wet touchscreen interaction surface by filtering out potential touchpoints caused by water [31]. Other specific contexts/tasks that have been examined recently include navigation while running [29], obtaining real-time gestural performance on-device without computationally heavy and memory-hungry additional hardware [7], and developing guidelines for gestures used in conjunction with smart rings [4].

Other studies have sought to identify specific environmental/exogenous factors that might bring on the onset of a situational impairment and analyze/measure their effects. For example, Sarsenbayeva et al. [23] investigated how acute cold exposure might affect fine-motor movements as well as user vigilance during mobile interaction. In a further study, Sarsenbayeva et al. [26] investigated the effects of ambient sound (e.g., construction sounds, exogenous human conversation) on transaction completion while engaged in a separate (unrelated) activity. More recently, stress (an internal as opposed to exogenous factor) was identified by the researchers to reduce both completion time and accuracy during mobile interaction tasks [24].

The fact that situational impairments are describing a temporary contextual inability to perform has led some researchers to examine possible commonalities in accessibility and assistive technology research. Nicolau [19], by focusing on walking conditions and tremor disorder and the situational conditions that may bring about similar issues, set forth the possibility of building a relationship between the assistive technology and situational impairment domains that could contribute toward a more inclusive and universal design approach. In order to improve both the functional and affective information and communication technology experience of certain impaired user groups (e.g., hearing-impaired, intellectually disabled, or older adults), Neerincx et al. [18] developed a methodology for speech technology to compensate for certain usage difficulties for identification of context-dependent user needs in multimodal user interfaces. Quinde et al. [22] discuss the advantages as well as the challenges of the use of existing context-aware technology to support asthma management. Jarl and Lundqvist [8] argue that an artificial separation of assistive technology and mainstream technology might represent a barrier toward a universalistic view and then offer a concept model called the person–environment–tool (PET). With this model, activity and participation are described as a function of factors without making distinctions between people of different ability levels, between environmental modifications intended for people of different ability levels, or between different function-enhancing technologies.

A qualitative analysis of a problem space can sometimes add a level of richness to the understanding of that problem space as elements may be revealed that might not have been considered or hypothesized. While much of the research to date has focused on quantitative analysis and measurement of the phenomena, some recent research has begun to look at situational impairments with a more qualitative eye. Examples include Sarsenbayeva et al. [25], who reviewed “established situational impairments” and their impact on mobile device interaction. The authors suggested methods for their detection as well as design guidelines. To better understand the effects specifically of situational visual impairments (which can include ambient light, moving surroundings, position of device, device accessories, or content design), Tigwell et al. [30] revealed that situational visual impairments are frequently and broadly experienced by the mobile users that participated in their study. Saulynas et al. [27] showed that situational impairment events could be classified into one of five generalized categories: (1) technical, (2) ambient/environmental, (3) workspace/location, (4) complexity, and (5) social/cultural. In addition, the authors identified a special category of situational impairment that was termed “Severely Constraining Situational Impairments (SCSI),” which were distinguishable based on several characteristics. A solution or workaround, for example, may not be found in sufficient time for the transaction to have any value to the user.

Situational impairments have been studied as single events to be addressed through re-evaluation of how users might interact with mobile appliances. Little research to date, however, has looked into the possibility of situational impairments comprising complex or compound events. The present study built on previous research geared toward the identification, classification, and design implications that situational impairments of all levels of complexity present in the mobile transaction space.

3 Methodology

The authors wished to gain a deep understanding of both the user and the context that might influence the ability to interact in the variable and complex mobile problem space. Therefore, a participatory design approach was adapted from a method developed to support individuals with visual disabilities [10]. Participatory design emphasizes that user interaction cannot be seen independently of other conditions [2] and emphasizes iteration for generating ideas and solution creation through interactive evaluation by the intended users [10]. It also allows users with domain knowledge to learn from those with technical knowledge and vice versa. The study followed a five-step approach that is outlined in Fig. 1 and described in detail in Sects. 3.1–3.5. Ethical approval was gained for the study through the Institutional Review Board (Protocol: 17-055: IRB Chair: Dr. Jeffrey D. Elliot, Stevenson University). All participants signed a consent form for their participation.

Fig. 1
figure 1

Overview of approach taken

Full size image

3.1 Structured interviews

Prior to the assembling of the participatory workshops, an understanding of common mobile device tasks, particularly those tasks that are most affected by the presence of a situational impairment, was needed. Step 1 was designed to obtain this information through a set of structured interviews with a heterogeneous sampling of smartphone users. The authors desired a sampling of the ways that users react when encountering a situational impairment. Questions probed for common mobile device information activities. Additional questions shed light upon internal decision-making processes leading to the execution of task steps in the presence of a situational impairment. Data collected at this stage were supplemented with the corpus of over 350 situational impairment events collected from the study by Saulynas et al. [27], and inter-rater reliability tests were performed on the motivation data that were collected.

3.2 Non-severe and severely impairing scenarios

Utilizing the data from Step 1, Step 2 developed three rich and representative scenarios showing common, meaningful, and identifiable interaction contexts involving typical mobile users encountering the onset of a situational impairment. Each scenario consisted of a “non-severe” situational impairment as well as an enhanced/severely constraining version to allow for the examination of the unique characteristics of a severely constraining event as defined by Saulynas et al. [27].

These scenarios formed the basis of discussion for the participatory design workshops assembled in Steps 3–5. A detailed description of the workshops that were held appears in the subsections below. In general, in order to represent a broad swath of mobile device users, each stage consisted of a heterogeneous sampling of four or five participants. The number of participants for each workshop was kept at five or below to mitigate any concerns of equitable participation from each member and was consistent with the methodology of the similar study by Kuber et al. [10]. Each workshop team was comprised of at least one “digital native” (high school/college age at the turn of the century), one “digital immigrant” (born prior to digital natives) [21], and one domain expert (defined by the authors as someone who is an interface designer with experience of developing for situational impairments and/or other disabilities). Using a heterogeneous population allowed non-experienced users to contribute ideas due precisely to their lack of knowledge about marketable technology [20]. The intent of the research was the exploring of solutions that could apply to a broad spectrum of users while maintaining verisimilitude in regard to present and perhaps near-future technology. The interplay of the two worldviews represented by the user groups combined with the practical and academic knowledge of the domain experts served to produce an effective and useful set of holistic design solutions. A detailed breakdown of each group’s demographics appears in the Results section of this paper.

3.3 First participatory design workshop

The first group met multiple times to offer perspectives to support users when facing situational impairments. At the first meeting, the role of mobile transaction modalities was discussed and ideas were brainstormed, through divergent thought exercises, toward the collective attainment of potential solutions to each of the scenarios. Solutions were developed for both the non-severe and severely constraining impairment versions of each scenario in order to help determine whether severely constraining events might warrant different design considerations from those of their less severe siblings. The researcher led the discussion, allowing design suggestions and idea reflection relating to the addressing of the phenomena and/or providing stronger notifications to identify their presence. Each subsequent meeting served to refine ideas, eventually converging on one or two of the best solutions to each scenario. This iterative process enabled strengthening of ideas, as well as the opportunity to suggest new ones.

3.4 Second participatory design workshop

The first team produced design suggestions as well as possible input/output modalities that might be utilized for effective transaction completion. In order to evaluate the validity of the first team’s solution, and/or to offer further design refinements, a second heterogeneous group was convened. As with the first group, this team consisted of a mixture of digital immigrants, digital natives, and domain experts. It was important that ideas be iterated sufficiently until arriving at data saturation. However, once that moment is obtained from one group, this does not necessarily mean that further refinement or even newer ideas cannot be achieved. Therefore, similar to Kuber et al. [10], the authors asked this new team not only to review the results from Step 3; they were also charged with drilling down into the solutions to obtain very specific design ideas for the various input/output modalities that the solution sets call for (e.g., If a sound is required as output feedback: How loud? How long? Speech or non-speech based?).

3.5 Combined team workshop + prototypes

A final participatory design team, consisting of members from both previous teams and maintaining a similar demographic mixture, was assembled. Combining participants from both teams aided in the process of further refinement of the design suggestions and promoted the strengthening of ideas toward a set of consensus solutions. The team was shown both the current solutions and low-level prototypes representing samples of the input/output modalities for evaluation (e.g., If a sound is required as output feedback: Are the levels of intensity/duration appropriate? Does this match your expectations?). The preset stimuli were presented in these preliminary prototype designs so that design recommendations could be made with specific design characteristics.

4 Results

4.1 Structured interviews reveal motivations

Interviews with 20 participants (7 female, 16 digital natives) conducted in Step 1 revealed a deeper understanding of user motivations for workaround and postponement/abandonment. Ages ranged from 19 to 66 (mean 28.6, st.dev. 15).

Oft-cited events from the events corpus in Saulynas et al. [27], where a situational impairment occurred because the user consciously chose not to attempt interaction due to the presence of one of three legally or socially unacceptable contexts (driving, on public transport, or at a public performance/meeting/lecture), were presented. Participants were then asked whether they ever wanted to interact with their smartphone using their hands while in each of these three unacceptable contexts but chose not to do so. Specifically, for each of these contexts, they were asked: (1) Whether they ever chose not to engage in the interaction; (2) If “yes,” could they list the reasons for their transaction forbearance; (3) Have they ever done it anyway; and (4) If “yes,” could they list the reasons why they “overrode” the forbearance reasons listed in (2) above. The results were confirming and, at times, worrisome.

For (1), all 20 participants indicated that they chose to forgo or abandon the transactions for most or all of the contexts. As suggested by the sub-themes for the social/cultural issues defined in Saulynas et al. [27], participant responses for (2) reflected concern for (a) socially acceptable behavior, (b) safety, or (c) fear of reprisal from an authority. A subset of examples appears in Table 1. Axial coding was performed, and two coders tested the results for inter-rater reliability. None were determined to reside outside of the three sub-themes for social/cultural situational impairments defined in Saulynas et al. [27]. Each context was analyzed separately for inter-rater reliability, with all calculating to a Cohen’s Kappa score above 0.6, indicating good agreement among reviewers. Because the responses helped reveal user motivation, this helped to confirm the initial findings of the study by Saulynas et al. [27] (that social/cultural situational impairments can be classified by one of three sub-themes) as well as to substantiate RQ1 (What are the motivations for mobile device users either attempting or postponing/abandoning a mobile transaction during the onset of a situational impairment?).

Table 1 Sample responses to social/cultural concerns
Full size table

4.2 Scenarios

The interview responses, along with the situational impairment corpus created and presented in the work by Saulynas et al. [27], were used to construct three rich situational impairment scenarios (see Table 2) and used by the participatory design teams in Steps 3–5. Each scenario had two versions: a non-severe version and an “enhanced” severely constraining version. These scenarios were chosen and ultimately used because they were common stories that the interview/corpus data were telling. In order to derive rich and meaningful suggestions, the participants needed to empathize better with the needs of others facing these kinds of issues, some of which they may have experienced themselves. It was important, therefore, that typical/common situational impairment scenarios were offered for their consideration and analysis. While the number of scenarios was restricted to three for logistical purposes (to allow enough time for meaningful discussion of each during each session), on aggregate, they represented the characteristics of both situational impairment events and more severely constraining events from Saulynas et al. [27], which were the primary focus of the research.

Table 2 Three impairment scenarios (each version)
Full size table

4.2.1 Scenario 01—driving: rationale

This scenario represents the common environmental context of operating a personal conveyance. It examines the severely constraining characteristic of multiple transaction events attempting to occupy the same transaction space as described in Saulynas et al. [27]. It also displays three of the four challenges for humans attempting interaction while on the go as described in Marshall and Tennent [14] (i.e., cognitive load, physical constraints, and terrain).

4.2.2 Scenario 02—at the movies: rationale

This scenario is representative of any situation in which a user is in a shared public space during an event where engaging with a mobile device would be considered socially or culturally inappropriate as described in Saulynas et al. [27]. In addition, even though the ambient condition is a stationary one, it displays the fourth challenge for humans attempting interaction while on the go as described in Marshall and Tennent [14] (i.e., other people).

4.2.3 Scenario 03—making dinner: rationale

This final scenario reflects situations where the primary resources needed to complete a transaction are unavailable. The severely constraining enhancement includes the characteristic of “transactional half-life,” as described in Saulynas et al. [27], where for the transaction to have any value, it must be completed in a timely manner. Finally, like Scenario 02, even though the ambient condition is somewhat stationary, it displays three of the four challenges for humans attempting interaction while on the go as described in Marshall and Tennent [14] (i.e., cognitive load, physical constraints, and terrain).

4.3 Design teams overview/demographics

4.3.1 Introduction

Three different design teams were formed. The second team consisted of a completely different set of participants from that of the first team, whereas the third and final team consisted of participants from the first two teams. The first team met for two separate sessions. The initial meeting produced a broad and divergent set of possible solutions to each scenario. Session 02 resulted in a convergence on one solution that offered the greatest potential for overcoming the transaction barriers represented by each scenario. Team 02 was then convened to examine each scenario as well as the solution set from Team 01 to offer any modifications to the existing solutions or to suggest alternative ideas. Finally, the final team with members from each of the previous groups met to offer a final review of the ideas put forth, to reconcile any differences between the solution sets, and to test some basic prototypes that represented some of the offered design solutions.

4.3.2 Workshop goals and demographics

The goal of this series of participatory design workshops was ultimately to discover whether and/or how designs varied when considering the addressing of impairing versus severely impairing events, as viewed from the lens of a mobile user and supplemented by experts. The participatory design method facilitates the suggesting of ideas that could then be strengthened from group to group.

A total of nine individuals participated across the three workshops (four digital immigrants, three digital natives, and two domain experts). The first session in Workshop #1 consisted of five participants (two digital natives, two digital immigrants, and one domain expert; mean age, 38.8), whereas the second session was conducted with four. (One digital immigrant was unable to attend.) Workshop #2 was convened for a single session and consisted of four participants (two digital immigrants, one digital native, and one domain expert; mean age, 42.5). The team for the final workshop convened for a single session and consisted of the domain experts from each of the previous sessions, as well as a digital native (Workshop #2) and two digital immigrants (Workshop #1). The detailed demographic breakdown of each workshop appears in Tables 3, 4, and 5

Table 3 Workshop #1 demographics (two sessions)
Full size table
Table 4 Workshop #2 demographics
Full size table
Table 5 Workshop #3 demographics (combined group)
Full size table

4.4 Flow of ideas from session to session

4.4.1 Workshop #1: sessions summary

In the initial session, each participant brainstormed the ways (modalities) in which one can interact with a smartphone, writing ideas down separately to promote free flow of thought without being biased by the ideas/opinions of others in the group. Each scenario was presented separately (the non-severe version being presented first). The group worked on design solutions, first separately, then by comparing their individual ideas and working toward a consensus for viable solutions. By the end of the session, the group produced a list of ideas for each version of the three scenarios. This list was to form the starting point for Session 02. Whereas the goal of Session 01 was ideation and the elicitation of many ideas, the Session 02 focus was on the refinement and/or revision of the ideas from Session 01. The group was asked or encouraged to consider the best solution to each scenario, even if that solution involved a non-technological workaround or transaction forbearance. As with Session 01, the researcher facilitated discussion by allowing for open idea reflection and design suggestions, assuring that discussions stayed on point and that the solution ideas were based on the equitable collective consensus of the group.

What started to emerge from the solution set is a clear distinction between the participants’ solutions for the two versions of the scenarios (i.e., the situational impairment and severe “enhancement” shown in Table 2). In particular, it was apparent that the non-severe impairment solutions involved either utilizing an existing technological solution (or with minor enhancements) or simple transaction forbearance. Further details are described below by scenario.

4.4.1.1 Driving scenario

For the non-severe version of the scenario, the group systematically eliminated solutions from the previous session where a flaw was noted, and concluded that the best course of action would be automatically connecting the smartphone via USB or Bluetooth connection to the console prior to departure, followed by controlling the GPS app with voice or minimal touch. For the enhanced version, the group moved to a solution where calls should go to voicemail while using navigation with (1) an enhanced reminder/notification banner that will allow the user to override if conditions are safe and (2) certain contacts receiving a voicemail with additional information (i.e., estimated arrival time, current location) based upon the situation, context, or environment where the user is heading. Depending on the phone location, the modality could be touch or voice.

4.4.1.2 Movie scenario

For the non-severe version of the scenario (without the confounding factors represented in the enhanced version), the group, after a brief discussion, concluded that the workaround of simply postponing the transaction reflected the most reasonable solution. For the enhanced version, the group directly took to the idea of custom vibrations for those on a “VIP priority” list of contacts, similar to the solution to the driving scenario. Vibrations could be delivered to the phone or perhaps a secondary device such as a smartwatch, which would be accessible even in the presence of ambient sounds. The discussion included suggestions of the types of vibrations that may be used (e.g., varying in intensity based on level of importance) or even a contextually appropriate sequence of sensations (e.g., pulses that simulate a heartbeat from a contact related to an impending birth).

The group seemed to be settling on a solution where, upon recognizing the alert as important, they would then excuse themselves from the theater to complete the transaction. The researcher at this point suggested that this would still not solve the social/cultural issue of having to disturb other theater patrons. The group seemed adamant that, given the circumstance, it would be culturally appropriate to bother people. The researcher then redirected by suggesting that they might be giving up too easily and should perhaps brainstorm a little bit more to see whether a solution can be derived where one does not have to settle for the second best. This then led to a discussion of contextual awareness (e.g., incorporating a geo-fence) in addition to the incorporation of a VIP priority list to provide the smartphone with the contextual information needed to make an appropriate decision as to how best to handle the output. The group saw the advantages of incorporating technology like a geo-fence as allowing the user’s mobile appliance to know (1) that they are in a movie theater complex, (2) which theater in that complex they are in, (3) what movie is being shown in that theater, (4) the length of the movie, and (5) how much of the movie has transpired. This information could then be incorporated to provide essential information that could be used to send the appropriate contextual response.

4.4.1.3 Cooking scenario

There was only one non-severe cooking scenario solution that resulted from the first session, so after briefly considering it and the merits of perhaps other simple possibilities, the group stayed with their original solution of using voice as an alternative input modality to record the idea using a voice-activated personal assistant (VAPA) such as Siri or Amazon Echo. For the enhanced version of the scenario, the group also moved toward the use of a secondary device using a voice interface. Ideas included using a VAPA, a smartwatch, or perhaps some other form of future “Internet of Things” type of kitchen appliance. There was some pushback on this thread as issues of ambient noise might be present in the kitchen during the cooking process. An idea offered to resolve this was the incorporation of a “wake-word” that could be yelled to shut down a noisy “smart” appliance temporarily. Other ideas included writing the content “in flour” and capturing the scribble, which, although messy, would offer one solution to help support the task. To solve for the solution of limited short-term memory, similar to that of the driving scenario, the group considered the inclusion of an automatic reminder that would be enough to jog one’s memory as to what the idea was. Ultimately, a consensus was reached on the use of a secondary device that works in conjunction with other connected devices (e.g., speaking into a Bluetooth headset or utilizing smart kitchen appliances). Having all these elements work in conjunction could support recording the note and could assist by pausing noise or cooking processes.

4.4.2 Workshop #2: session summary

Team 02 began with similar orientation exercises as were done with Team 01. Each scenario was then presented along with the solutions from the first group, and they were asked to evaluate the solutions and validate or modify them. For each scenario, the group compared their individual ideas and worked toward a consensus for viable solutions for each version of the scenario. The researcher facilitated discussion in a similar fashion to that of the Team 01 sessions. As was true from Team 01, at the conclusion of the session the solutions/modifications offered continued to show a clear distinction between solutions for the non-severe versus enhanced scenario versions. The second group offered some modifications to the first group’s solution set as well as specific qualities for the modalities used in the solution. Details are discussed in the following subsections.

4.4.2.1 Driving scenario

For the non-severe version of the scenario, this group (after discussion) agreed in principle with the Team 01 solution but added that, rather than simply connecting, the user is notified and is asked whether they want to connect the device. Also, for safety reasons, it was suggested that the system needs to recognize that the car is in parked gear, to eliminate the possibility of this being attempted while driving or at a stoplight. For the enhanced version of the scenario, the group moved to a modification of the Team 01 solution that involved the contextual message being sent/terminating a phone call but not being initiated automatically. Instead, the group came up with a creative idea whereby the user would employ the steering wheel as a binary, single-touch input device to cancel the call and initiate the voicemail message. The rationale for their idea was based on the logic that the technology could realize that GPS was on but also that a phone call was being initiated. Because both conditions were true, the vehicle could recognize the squeeze as canceling the new input. They also noted that the wheel should immediately vibrate to confirm that the call cancelation took place. The haptic feedback should be intense enough to be felt while driving and quick enough for the brain to know something was done, but not so long as to add to cognitive load.

4.4.2.2 Movie scenario

There was a general agreement with the Team 01 solution of ignoring the call for the non-severe version of the scenario. For the enhanced version, Team 02 agreed that contextual awareness from the technology available to the user was important. They added a novel suggestion for the use of a secondary device to receive/reply to the message that utilized the cultural affordance associated with the use of a wristwatch. The rationale stemmed from the fact that it is common, even during a public performance like a movie, play, or lecture, to look at one’s watch without it representing much of a distraction (if at all). Therefore, using haptic feedback that represented “importance” to signal that an important message is attempting delivery would alert the user. They could then see the message (or the phone number that is calling) and, with minimal touch or using a flick gesture, initiate the contextually aware response with minimal to no social disruption. To reduce the cognitive load of having to discern between multiple haptic sensations, the group stressed that there should only be two sensations present: (1) a vibration representing a “normal alert” and (2) a vibration representing “importance.”

When the researcher prompted for how “importance” should be represented in a haptic response, the consensus solution was that many rapid staccato pulses would be appropriate, to gain the user’s attention. As with the driving scenario, the intensity of the pulse needed to be sufficient to be perceived, which led to some pushback regarding the residual noise that a haptic pulse presented through a vibration motor would generate if too intense and how that may disturb others. One of the domain experts and the digital native both noted, however, that haptic engines could be made so that they do not vibrate the chassis of the watch. Finally, there was a suggestion that, like a phone call, which terminates after a set number of rings, the haptic sensation should also have a terminating rule. When prompted for ideas, the group concluded that the vibration should last for a minute, stop, take up again after 10 min, and repeat that cycle up to four times prior to terminating.

4.4.2.3 Cooking scenario

For the non-severe version, Team 02 agreed that Team 01’s solution matched user needs. For the enhanced version, the discussion was predominated with the use of some type of “stylus” that would allow input onto a smart surface. In addition to this being a viable alternative if voice was not available, Team 02 noted that sometimes ideas are not always represented well with words (i.e., a piece of music or a sketch). It was suggested that perhaps using one’s hands, by “writing in the air,” could be useful to input content. This idea was abandoned after consensus was reached that the lack of feedback to the “marks” being made would require too much of the available cognitive resources to make this tenable given the cognitive overload already existing within this scenario. The group settled on the use of ordinary utensils (i.e., whatever is currently being used in the cooking process or readily available) as a stylus for recording the note on a smart surface.

4.4.3 Combined group session summary and end solutions

The goal of this final session was to have representation from both previous teams and to present both sets of solutions to the combined group. Where the individual teams’ solutions differed, the combined team was charged with attempting to reconcile the differences to arrive at one unified consensus solution for each scenario. In addition, the researcher conducted a primary usability test on the effectiveness and usefulness of some of the modalities suggested in the solutions. For the usability test, a crude prototype of the haptic sensations was reproduced using the free Contact Vibrate app [3], where cues could be quickly created and parameters of touch easily modified. For each of the driving and movie scenarios, a haptic sensation, based on the specifications from the design sessions, was created. For the cooking scenario, a common cooking utensil (spatula) was used to allow the participants to simulate the attempt of using the utensil as an ad hoc stylus during the cooking process. There was very good interaction between the two subgroups as the combined team attempted to reach a reconciliation point between the two solution sets. The researcher would at times redirect when a point was made against one group’s idea by asking whether that group might offer a rebuttal.

4.4.3.1 End solution: driving scenario

For the non-severe version of the scenario, the combined team reached a consensus by agreeing that the solution suggested by Team 01 should be accepted as amended by the modifications suggested by Team 02. Assuming that the technology being used is “smart enough” to recognize that the park gear is active in the car, then the current utility of being able to activate GPS via voice interaction would represent the best solution needed to create an ad hoc navigation session. For the enhanced version, the concept of having a VIP list (Team 01) was reconciled as long as the user can control how the list is utilized (Team 02).

There was some debate, however, as to how to reconcile the feedback for when a call is canceled in the context of this scenario as well as revisiting the use of the steering wheel as the input device. This led to the re-examination of older ideas (i.e., having a prominent “Cancel” button on the wheel). After discussion, it was agreed that using the entire wheel was the best choice. The rationale was that, despite best practices, drivers can at any given point in time utilize any part of the wheel and all drivers have their own preference as to their preferred positioning. Only with the entire wheel being the input device could the capturing of an input during a period of high and cognitive overload be universally achieved. The group also came to a somewhat serendipitous realization that there is no need to present any additional feedback to confirm that the input was received. Since the scenario was one of a mobile task overriding another mobile task (i.e., phone call interrupting navigation), when the phone call went away and the GPS directions returned, that alone would provide sufficient feedback that the call had been canceled. Since this fact was incorporated into the end solution, the usability test on the haptic feedback was canceled.

4.4.3.2 End solution: movie scenario

For the non-severe version of the scenario, the group reached a consensus on the original Team 01 solution of simply ignoring the call. For the enhanced version, the Team 01 solution as amended by Team 02 was accepted. Each user then was asked to test the prototype to assess whether it represented the concept of “importance.” The rapid staccato pulse that was specified during the Team 02 session was tested, and all participants were satisfied that it accurately represented the concept of “importance.” In addition, it would be distinct and adequately discernible even when engaged within a movie.

4.4.3.3 End solution: cooking scenario

For the non-severe version of the scenario, all were reconciled that using voice command to activate a note application through a phone or a smart speaker represented the best solution, especially as it is a currently available option on these technologies. For the enhanced version, the central point for reconciliation revolved around the cognitive load and time sensitivity for both tasks, which of course was the central theme of the scenario. Participants from both subgroups began to question whether they would have time to enter a complete thought, sketch, etc., quickly enough as to not mess up the cooking process (by either forgetting a step or delaying execution of a step). The solution eventually became to create a quick audible placeholder for the ideas. If voice was not feasible, then using the ad hoc appliance would be implemented to create a quick placeholder note. The participants came to the realization that the key to solving this problem would be to get by that “critical juncture in the cooking process which…most likely may be a few seconds to a minute.” Once that passed, the user would then be freer to pause the cooking task and complete the note so that it could be delivered in two or more stages. Participants simulated movements that would be performed when cooking and then attempted to write a quick note using the ad hoc stylus in their hand. All agreed that this represented the best option, particularly in this scenario as the input could be achieved quickly, whereas washing hands in order to interact with a traditional electronic input device would take too long.

5 Discussion

Through coding and testing of the interview responses in Stage 1, revealing user action motivations and some of the workarounds deployed during the onset of a situational impairment event, the study was able to substantiate both RQ1 (What are the motivations for mobile device users either attempting or postponing/abandoning a mobile transaction during the onset of a situational impairment?) and RQ2 (What type of workarounds do mobile device users attempt when encountering a situational impairment, and are they different in the presence of a Severely Constraining Situational Impairment?). The design suggestions generated from the participatory design workshops in Stage 2, showing a distinct difference in the way that participants wish for technology to provide solutions to a non-severe versus an enhanced situational impairment event, also helped to substantiate RQ2 as well as RQ3 (Can mobile technology design better account for actions attempted and the transactional needs of mobile device users while on the go during the onset of a Severely Constraining Situational Impairment?).

In reviewing the proposed solutions to the three scenarios from the latter portion of this study, as well as the results from the structured interviews, three distinct implications for design emerged. Each will be discussed in detail below (Sects. 5.1–5.3).

5.1 All mobile users will, at times, feel the need to complete transactions “at all costs”

As data from the structured interviews show, mobile device users (whether digital immigrant or digital native) are willing to complete mobile I/O transactions at all costs, even in scenarios where they may be placed in harm’s way. All participants indicated an understanding of certain contexts where interacting with a mobile appliance can be a distraction, a socially/culturally unacceptable behavior, or dangerous. This is consistent with recent research such as that by Moser et al. [15], who showed that attitudes toward social/cultural norms could play a factor in the willingness to forgo mobile interaction. Inappropriate interaction in these contexts could result in unofficial cultural censure (other people thinking the behavior annoying/rude) or official censure (getting a ticket or having a teacher/boss issue demerits of some sort), or the interaction could put themselves or others in harm’s way. However, even though 100% of the participants acknowledged the existence and value/purpose of the thou shalt not rules, those same humans, without exception, indicated that they willingly at times ignore these rules just to complete a mobile transaction.

5.2 Cognitive load is a significant factor distinguishing a severe from a non-severe situational impairment

In Saulynas et al. [27], of the five situation impairment event themes that were developed, the most common event reported (31.2% of the total) was that of complexity issues (e.g., cognitive overload). The enhanced impairment solution sets from the workshops illustrate the need to address the diverting of cognitive (or cognitive + physical) resources from another task (e.g., driving or walking) in order to complete a mobile transaction. Marshall and Tennent [14] state that along with changing environmental conditions, cognitive load represents a compounding, or at least aggravating, factor to task completion that is not present in examining transactions with the desktop paradigm. Sending an email using a desktop computer, for example, requires no added cognitive load or physical task to be accounted for other than the keyboard/GUI interaction needed to complete the task. In addition, because the interaction is taking place in a private or semiprivate space, no environmental situational awareness is required. Finally, because typing and clicking are relatively silent modalities, there is at best only a minimal level of social/cultural consideration that must be accounted for. Of course, cognitive disruptions may occur even in a stable environment (e.g., a phone ringing, a co-worker attempting conversation, or someone eating a meal nearby that has a distracting olfactory presence). However, in addition to the same exogenous disruptions that can occur at one’s desk, the mobile user attempting the same task must also account for the variable nature of all of the exogenous variables that exist outside of the task as well as the interface required in completing the task.

Research to understand and account for situational impairments in the mobile transaction space (such as was highlighted in Sect. 2) has produced artifacts that are already providing an improved mobile user experience. One can now, for example, complete the steps for creating/sending an email message entirely with a voice interface effectively accounting for contexts where the primary physical resources are occupied (e.g., hands are making messy food or driving an automobile). However, where current solutions seem to fall short of providing the optimal/safest user experience is recognition of the true potential complexity of the mobile transaction space. Severely constraining events, such as those represented in the three scenarios used in this study, are examples of situational impairment events that, while still single events, are affected by multiple ambient factors where the current technological remedies, such as offering an alternative I/O modality, do not represent an effective solution to support the user experience.

5.3 Importance of context awareness in designing for severely constraining situational impairments (SCSI)

What became clear, especially for the enhanced scenarios, was the mobile user’s desire for better technological context awareness. Solutions offered for the non-severe scenario versions ranged from forbearance (movie scenario) to utilizing existing solutions (cooking scenario) to utilizing existing solutions with minor enhancements (driving scenario). For the enhanced scenarios, however, participants consistently felt that the mobile device should act as a personal assistant. According to this study’s workshop teams, technology should (1) recognize the context of the situation, (2) assess the best course of action given the environmental context, and (3) execute the steps necessary to complete the best course of action with minimal to no in situ input from the user.

In the driving scenario, the user must devote most of their active resources to a combination of paying attention to the task of driving and to the driving directions. When the added task of a phone call presents, the solution offered from the workshops is, in effect saying that we need technology to handle some of this load for us. Their solution represents a context awareness need that off-loads many of the decisions regarding effective transaction completion. With technology handling the minutiae, the user can continue to focus on the primary two tasks with minimal to no distraction and with an understanding that the third important task is being taken care of.

The cooking scenario enhanced solution reflects a similar mindset. Technology needs to provide a means to shorten the length of active concentration on one task so that focus and value are not lost on another. Even in the movie scenario, where the dominant issue is the social/cultural minefield that mobile device users traverse rather than one of cognitive overload, the acknowledgment of limited cognitive resources is present in their solution by recognizing that anything more than binary haptic feedback would be too much of a potential cognitive burden. This is supported, for example, in previous haptic research such as that of Wolf and Kuber [34], who found that users had difficulty distinguishing three-parameter tactile signals presented at sites on the head and being able to identify the interval pattern and stimulation location while visually distracted.

This need to offload information from our working memory is not without analogy in the analog world. When we are temporarily physically or cognitively overloaded, we often turn to a human ally for assistance. A personal valet or office assistant often helps their employer by managing the minutiae of their day, which frees the employer to focus on higher-order tasks and problem solutions. However, there are limitations to human assistance that are not present in technology. As Hollan and Stornetta [6] noted, we can design interaction to be as close to emulating human-to-human interaction as possible, or we can take advantage of the strengths of the computer to help overcome human limitations and enhance/complement human–computer interaction. A technical assistant is not sentient; we do not have to have concern for their feelings, their stress, or their rights. Our only concern is for capacity and capability limitations.

If mobile technology is to continue to be able to support users, it needs to adopt a greater assistive role. The results from the workshops of this study highlight that users need mobile technology to be more context-aware and anticipative in order to begin to solve the problem of “Severely Constraining Situational Impairments” so that users can be not only safe and productive but also satisfied. To promote these user-desired features, future work could look to, as Wobbrock [33] suggests, simplified software and sensor toolkits that can support platform-independent context-aware applications in order to support a greater proliferation of context awareness technology.

6 Critique of approach, limitations, and future work

The five-step process adopted was found to be valuable, as it provided a structured method of gathering data and allowed for interaction and refinement of ideas. The use of participatory-based workshops allowed participants to work together to discuss and reach a consensus of ideas. Including users with different levels of technical knowledge and expertise led to a diversity of suggestions from participants. However, discussing three scenarios (each with two versions) within each session proved challenging within the time period allotted. Including limits on the length of time apportioned to discussing each scenario enabled the researchers to move through the sequence of scenarios.

The scenarios in this study were designed to represent common events based on previous data. There certainly exists in the world more and diverse scenarios where users may wish to interact with their mobile device but cannot due to issues of context. Clearly, many of the solutions that were suggested by the participants of the study were fashioned to address the problem that was specifically offered by each scenario. The workshops produced some design ideas and solution examples, but these ideas and examples were both acutely focused and perhaps somewhat limited. As such, one cannot look to the designed suggestions offered in this study alone to glean generalizable design guidelines. However, it is important to note that one of the goals of this study was to see whether there is any commonality in how a user would like to approach transaction completion in different situations. Now that some commonality has been revealed in these three acute circumstances, future work might continue to examine more diverse situational impairment scenarios (e.g., while walking or exercising).

In this study, users examined situational impairments, whether by sitting for an interview or by being engaged in deriving potential design solutions, in a closed laboratory environment. Situations were examined by allowing participants the opportunity to relax, and to think logically and deliberately, in a non-transactional setting. In the design team sessions, for example, it was common and consistent for the participants to suggest transaction postponement. It seems clear that whether one is a digital immigrant or a digital native, there is general acceptance that postponing or even forgoing the transaction when a reasonable workaround does not present itself is both the acceptable and the desirable thing to do. However, this assumption was not tested in a “live” setting where the weight of the relative values of (A) completing the transaction now versus (B) the acceptance that forbearance might be the correct and safe thing to do would be compared. Still, as the interview results from Step 1 show, 100% of the participants who drove a car indicated that there have been and continue to be instances where indeed (A) > (B). One could certainly argue that there is a cognizant recognition by all mobile phone users that this safety and cultural acceptance principle does not always outweigh the need for timely transaction completion. To test this hypothesis, however, the future study would need to examine real behavior in the wild.

Valuable generalizable information was obtained through iteration, the participatory design process, and the heterogeneous involvement of both digital immigrants and digital natives. All participants were over the age of 18, and none were of retirement age. Future studies might extend the age stratification analysis to include adolescent and older (retired) mobile device users. In addition, workshops that used homogeneous groups defined by age stratification might reveal design solutions that speak more acutely to the needs of the individual groups as opposed to a more universal set of solutions.

This study produced implications for design. However, the implications were somewhat general in nature and limited in regard to specific insights into how mobile designers, researchers, and app developers should consider the design of mobile technology and applications to accommodate the various forms in which situational impairments present. For example, the conclusion that technology needs to be more contextually aware, while it is a legitimate conclusion, is neither really novel enough nor offers enough detail to enable a designer to identify ways to implement this. Future work might continue to explore the implication gleaned from this study and produce a set of actionable guidelines for the addressing of situational impairments of all forms and levels of severity.

7 Conclusion

This study was an initial foray into the examination of how users of mobile technology deal with situational impairments. The research represented in this study examined the increasing complexity of the mobile transaction space and the effect of this complexity on mobile interaction. The presence of “Severely Constraining Situational Impairments” and the different design considerations that were suggested will help to support mobile interface designers in the addressing of this confounding phenomenon. In shedding light on user motivations, this study has also contributed to a richer and deeper understanding of how situational impairment events are affecting users. By going beyond the measurable effects on task completion, this study has offered an insight as to why users chose to perform (or forgo) some actions when a situational impairment presents.