Requirements Engineering in a Mobile Setting: How Travelers with a Cognitive Impairment Ask for and Use Help

Abstract

A requirements-gathering study of getting-lost behavior is described. Two matched groups of subjects, one with and one without acquired cognitive impairments, were asked to navigate a walking route. Two foils were introduced to induce problems in route following. A phone helper was available to assist with problem solving. Both quantitative and qualitative results are reported.

These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

1 Introduction

Our group has been involved with assistive technology for travelers with an acquired cognitive impairment (ACI). We have previously studied the type and mode of assistance necessary for walking trips within the community [1]. However, from those studies came evidence that we were ignoring an important issue: how do travelers with a cognitive impairment deal with getting lost or other obstacles that occur within their travels? This was heightened when we realized that much of this population’s reluctance to “get out” into the community stemmed from anxiety, i.e., worry and concern about not knowing what to do in problem situations [2].

We saw two directions to pursue in our research: (1) we could continue to build better and better prescriptive navigation and way-finding tools that minimized a traveler getting off track. (2) We could begin to explore a new mode of assistance, that of recognizing a travel problem and attempting to remedy it in the field. Clearly we would prefer the former. Who wants to get lost and anxious? However, for those with a cognitive impairment in particular, there are many ways for a trip to go off track, too many to feasibly control or avoid with an assistive device. We felt it was prudent to begin to look at assistance for people who were in a problem state (lost, confused, anxious) during travel. This paper discusses a pilot study to gather information about the way travelers with a cognitive impairment interact with human assistance through phone support. From this, we hope to gather the requirements of a computer-based assistant [3–6].

2 The Challenges of Field Evaluation with Impaired Populations

Goodman et al. [7] have suggested field-based mobile-device evaluation procedures for the non-impaired population. It is instructive to look at the metrics proposed and compare each with what we found appropriate for our studies with participants with a cognitive impairment.

• 1. Time to complete. Although we typically record timing information, we find this measure less than useful in evaluating field devices for our population. There are two problems. First, we have found that someone must be available for help. We have used quiet companions in the past. In this study, we used shadow followers. Given that participants were sometimes “interfered with” by the observer, i.e., given feedback when confused or lost, timing information was deemed less than accurate. Further complicating timing information, it is not unusual for us to employ a “foil”, an actor in the environment that interrupts the participant to break his or her train of thought, e.g., by asking for a cigarette. But more importantly, we are rarely interested in efficiency. Instead, our interest is whether some technology will enable a person to do something that is currently beyond their means. Of course, timing can’t be totally ignored. If a trip takes a person longer than is feasible (e.g., beyond their physical endurance, longer than stores stay open or buses run), then that is an issue. But whether a tool or device is minutes ahead of others is not a priority for us.

• 2. Errors. For us, this is perhaps the key metric in our studies. For each field trial, we spend considerable time developing an evaluation instrument around gradations of errors in navigation. We also video-record each trial to further analyze and validate our written notes. We pilot our evaluation methods to refine them.

• 3. Perceived Workload. Goodman and colleagues note: “Workload is important in a mobile setting as users must monitor their surroundings and navigate, therefore fewer resources can be devoted to an interface. An interface that reduces workload is likely to be more successful in a mobile setting.” We would add physical workload to this concern. For instance, we chose a device-on-arm approach in an earlier navigation study [1] to avoid (a) a participant having to carry something in his or her hand over the routes, or (b) having to pull the device in and out of a bag or pocket constantly. However, we observed, and our participants commented on, the problems with a device strapped to their arm. It grew heavy over time. More related to safety than workload, another goal of the device-on-arm was to allow participants to look away from the device (for instance, to watch for traffic), and only glance at it when a new instruction arrived. We did this by beeping on arrival of new directions before they reached a potentially unsafe intersection. Nevertheless, most participants monitored (looked down at) the device more than we would have liked, commenting that they were worried they would miss a direction or not hear the beep. Chewar and McCrickard [8] reported similar issues of participants being extremely cautious of not missing a choice-point direction. In the study reported here, we have finessed many of these issues by using an always-on audio device.

• 4. Distance and route. Goodman and colleagues note that there are various ways one can track a subject, e.g., pedometer, GPS, observation. We do not ask our participants to travel independently while we monitor. Instead, we define “field labs” where routes and distances are known. In this way, routes can be balanced for number and type of choice point, distance, and complexity. This supports controlled studies. We do not see a way of holding the study we report without controlling variables with a set field lab.

• 5. Percentage preferred walking speed (PPWS). We understand the motivation for this metric. If someone is accustomed to walking at a given pace, a device that slows them down will be unpopular. In our studies, we find that most of our participants do not venture far from their facility; there is no baseline for them navigating in unfamiliar surrounds. More generally, our experience is that our participants are much more anxious about getting lost than about how fast they travel.

• 6. Comfort. Of course, in a general sense, comfort is a universal concern. We have discussed how comfort can intersect with workload in a prior point. What we have found particularly heightened in our studies is social comfort: avoid standing out because you are carrying a geeky looking device. Clearly times change, and what might have looked geeky 5 years ago is now the norm. Our approach is to hold small focus groups prior to device selection for each study to gauge participants’ perceptions. For this study, in particular, participants were fine with the cellphone earbud in focus groups and during the trial.

The remainder of the chapter describes a study that motivates the points above.

3 Study Outline

Participants with a Cognitive Impairment (experimental group) and without a Cognitive Impairment (control group of age, education, and gender matched peers) were asked to follow a set of written walking directions. The directions had two foils: a missing step (a turn) and an incorrect destination description (a bookstore instead of a bowling alley). Subjects were given an “always on” ear bud (with cell-phone in pocket) that connected to a researcher in an office (the phone-helper). The phone-helper did have knowledge of the general area of travel, but did not have information about the location of the subject. Hence, the subject was required to describe his or her location over the phone, as well as any other information about their problem that might be useful. The phone-helper followed a general script for helping a subject problem solve. The phone-helper never initiated a conversation, but only spoke when directly addressed by the traveler. Two other researchers (field observers) shadowed the traveler, and kept field notes.

All participants were given transport by research staff to the trial site. The route covered a seven block area of downtown Springfield, OR. Figure 7.1 provides an overview of the route and the eight-step written directions provided to participants. There were four steps along the route where challenges were anticipated: initial orientation (Step 1), a missing step (Step 4), a step with a hidden street sign (Step 7), and the incorrect destination (Step 8). A series of pilot evaluations with uninjured adults ensured clear wording of instructions, especially for steps not designed to present navigational challenges (i.e. steps 2, 3, 5, and 6).

Fig. 7.1

Route map and instructions

4 Research Questions

We will focus on the following questions in this paper.

• RQ1. Are there differences in how participants with ACI solve navigational challenges compared with age, gender, and education-matched non-injured control participants?

• RQ2. Do individuals with ACI demonstrate greater delay and less planning compared to age, gender, and education-matched non-injured peers when asking for assistance due to a missing step with written navigational instructions?

• RQ3. Are there differences in preference for re-orientation by telephone between participants with or without brain injury?

• RQ4. Are there differences in how individuals with ACI or non-injured peers describe their current location to an unfamiliar phone assistant at a remote location?

• RQ5. Are there differences in the quality of potential solutions to on-route navigational challenges described by individuals with ACI compared to non-injured peers? (Part of the phone-helper’s script is to ask the traveler to describe what they think a potential solution is to their problem.)

5 Mechanics

5.1 Participants

Two groups of participants completed this study: 18 adults with ACI and 18 matched control participants. Table 7.1 provides further information on our subject pool. The ACI group was drawn from several assisted-living facilities in the area that support those with a cognitive impairment. All members of the ACI group were free to come-and-go independently from their facilities. Each facility had a bus-stop nearby that would support community travel. Nevertheless, a prior study showed that while there were no policy or procedural barriers to community travel, few residents of these facilities took on independent trips, either by bus or by foot [2].

Table 7.1 Participant profiles

5.2 Scoring

Observational data at each of eight choice points along the route was gathered, and scored on a 6-point scale:

• 5 = correct & independent

• 4 = self-corrected an initial error

• 3 = asked for verification of information

• 2 = asked for assistance

• 1 = required intervention (more than 2 blocks off-track)

• 0 = Unable/Quit

In addition, we scored “directness” at each of eight choice points (1 = direct, 0 = hesitate). We are most interested in how participants did at 4 of these choice points:

• Step 1: Initial orientation (“Face 8th Street”)

• Step 4: Missing Instruction (omitted “Turn Right on 9th Street”)

• Step 7: Obscure Sign (difficult to see “Main Street” sign)

• Step 8: Wrong Information (told to stop at a bookstore, not a bowling alley)

Additional qualitative data were gathered from four sources: field-observers (2) field notes; phone transcription recordings, and the post-trial interview. These data allowed the researchers to describe and investigate possible group differences in how participants described their current location, generated potential navigational solutions, and opinions about use of a cell phone for pedestrian navigational assistance.

5.3 Fidelity of Adherence to Phone Script

Each time participants asked for help using their earbud, the phone-helper asked them a series of scripted questions, and tried to re-orient them with correct information over the phone. Phone conversations were audio recorded and transcribed after each trial. A researcher checked each transcription to ensure that the phone-helper did not deviate from the script when attempting to gather data and re-orient the participant. Fidelity was calculated for a sampling of 21/36 (well above the suggested 25% for single-subject experiments) trials by marking on a checklist if each instruction was delivered correctly and at the correct time, across three different phone helpers and for both groups. Overall, the phone helpers (three different helpers) followed the script with 95.56 % fidelity (Range: 88–100%) across the sampling.

5.4 Data Analysis

Quantitative data were entered into SPSS 16.0. Mixed Model analyses were employed, due to repeated measures for each participant (i.e. 8 choice points per participant). The Mixed Models analysis controls for repeated measures within-participant when investigating between-group differences [9]. To investigate the effects of group and type of direction on wayfinding ability, we ran a Mixed Model analysis for Accuracy and Directness. Chi-Square analyses investigated group differences in Wayfinding Strategies [10]. Significance tests explored relations between all quantitative variables. Effect size measures estimated the practical significance of any statistically significant finding using Cohen’s d [11]. Two researchers analyzed qualitative data for themes [12]. The researchers independently analyzed the data, extracted themes as categories, compared results, and reached consensus.

6 Results

Descriptive statistics for the two dependent variables included in the Mixed Model analyses revealed non-normal distributions, especially for control group performance (see Table 7.2). However, we conducted no transformations because these non-normal distributions represent hypothesized, naturally occurring phenomena (i.e. the control group performed near ceiling levels with less variance). The overall mixed model was significant, (Wald Z = 11.66, p = .000).

Table 7.2 Non-normal distributions of data for mixed model analysis

In the remainder of the section, we will return to each research question, and provide study results for each.

RQ1. Are there differences in how participants with ACI (BI in Table 7.3) solve navigational challenges compared with age, gender, and education-matched non-injured control participants (Con in Table 7.3)

Table 7.3 Non-normal distributions of data for mixed model analysis

As seen in the table above, there were differences in how participants with ACI solved navigational challenges compared to matched controls. This is a small effect, accounting for only 6% of the variance. Participants with ACI were less independent overall, requiring self-correction, verification, assistance, intervention, or quitting the trial early due to difficulties. While differences between groups were not significantly different at each individual choice point (likely due to reduced power from relatively few choice points and small n), there are a few general trends worth mentioning:

• One participant with ACI was unable to be re-oriented over the phone while another participant with ACI quit the study early due to frustration; all control participants were able to complete the route.

• The field-observer had to intervene (when a participant was >2 blocks off-track) only one time for a control participant, but had to intervene five times for participants with ACI.

Specific to the Missing Direction (Step 4): Participants with ACI were more likely to request assistance (10/18) than control participants, who tended to either ask for verification as they solved it themselves (6/18) or request assistance (6/18).

Specific to the Hidden Street Sign (Step 7): Participants with ACI required assistance (6/18) while the majority (17/18) of control participants independently solved the challenge.

Specific to the Wrong Destination (Step 8): Both groups asked for assistance to solve this problem, but two ACI participants were unable to complete this last step accurately.

RQ2. Do individuals with acquired cognitive impairments (ACI) demonstrate greater delay and less planning compared to age, gender, and education-matched non-injured peers when asking for assistance due to a missing step with written navigational instructions?

Each time a participant asked for assistant for the missing step (step 4), we coded this as a 2 if they asked before reaching the intersection of 9th & B, as a 1 if they asked at the intersection, and a 0 if they asked after an unsuccessful search. So, higher scores indicate greater planning.

Table 7.4 shows that control participants demonstrated greater planning and were able to anticipate errors when a step was missing from their set of written directions (M = 1.83), while participants with ACI generally did not anticipate the error or waited to ask for assistance after an unsuccessful attempt to solve the problem (M = 0.82). This is a large effect.

Table 7.4 Dealing with missing step

RQ3. Are there differences in preference for re-orientation by telephone between participants with or without brain injury?

Table 7.5 provides a summary of quantitative data related to cell phone use. For ‘Ease of Use’ and ‘Helpfulness,’ participants were asked to rate these on a four point scale (1 = not at all; 4 = extremely). Higher scores represent greater ease of use and helpfulness. For ‘Use a Phone Again,’ this was scored as 0 = no or 1 = yes. Higher scores represent higher endorsement.

Table 7.5 Satisfaction data with phone connection to phone helper

Control participants unanimously endorsed the cell phone as easy to use and helpful for accessing navigational assistance. All control participants also indicated they would like to use a similar system in the future if they required navigational assistance. All participants with ABI reported that the assistance they received via the cell phone was helpful to reduce anxiety and re-orient them.

When participants with ACI were off-track or starting from a corner not expected by the phone helper, then they encountered greater difficulty, and the phone helper had to orient the participant to a landmark before giving a left/right direction. A few ACI participants had difficulty with left/right and needed landmark directions, which were harder for the unfamiliar phone helper to give. Our phone helper created a photo map of the streets and tried to use houses, bus stops, or stores to orient participants. It sometimes turned into more of a “trial and error” exercise to get the person re-oriented (“If you got to C, then you went the wrong way. Turn around and go two blocks to get to A.”)

RQ4. Are there differences in how individuals with ACI or non-injured peers describe their current location to an unfamiliar phone assistant at a remote location?

When participants asked for assistance over the cell phone, the phone helper asked them to describe their current location. By analyzing the phone transcripts, there were clear qualitative differences in how participants described their location. The descriptions by participants with ACI were more vague or inaccurate.

For the control group, 97% (29/30) of control descriptions provided a clear description of their location, which included two streets of the intersection at minimum, and often included additional modifiers, such as:

• “I’m at 8th & B, I just turned Right onto B.”

• “I’m on B, between 8th & 9th.”

• “I’m on Main, in front of Alpine Service Imports that services Volvos… I just passed 11th St… now I’m at 12th.”

• “Right in front of the Sutton Hotel, on Main St, about 1/2 way between 11th and 12th.”

• “1100 Main St, in front of Springfield Spas & Tanning.”

A small set (3% or 1/30) of control descriptions were vague, e.g., “I’m on Main St” (phone helper assumed correct part of Main and did not ask for clarification).

For the ACI Group, 68% (32/47) of ACI descriptions provided a clear description of their location, which often included two streets at the intersection, sometimes with a modifier, such as:

• “I’m at 8th and B, behind the Post Office.”

• “I’m right in front of the Post Office, at the corner of 8th & A Streets.”

• “I’m at the Brandt Finance sign, at the corner of 8th & A.”

• “I’m at 8th & Main, by a Legit Misfit place and a Subway.”

• “There’s no Street sign. The Street is just South of A, at the intersection of 8th, and it’s the first One-Way sign, with traffic going East to West.”

• “I’m at 10th & Main, or the equivalent of 10th & Main. The last street I passed was 9th, so this must be 10th. There’s a sign going toward Springfield High School, but no sign identifying the street.”

• “I’m on Main St, right by ‘Hidden Treasures,’ at the corner of Main and… right past the ‘Alpine Service Imports.’ I’m right in the middle of a block… now I’m at the corner of 12th & Main”.

The ACI Group had a moderate number (23% or 11/47) of vague descriptions. These generally only gave one street or specified a landmark only, such as:

• “I’m on 10th St.”

• “On Main St.”

• “I’m by the ‘Grocery Outlets’ and ‘Autocraft’.”

• “I’m in front of a bowling alley, I guess.”

The ACI Group had 9% (4/47) of descriptions that were inaccurate.

RQ5. Are there differences in the quality of potential solutions to on-route navigational challenges described by individuals with ACI compared to non-injured peers?

The phone helper asked participants to provide potential solutions to navigational challenges before providing participants with the correct solution. By analyzing the phone transcripts, there were clear qualitative differences between potential solutions. Participants with ACI gave more vague, inaccurate, or non-solutions. For instance, at Step 4 (Missing Step), 100% (15/15) of solutions generated by control participants were “reasonable”. In contrast, 56% (9/16) of solutions generated by participants with ACI were “reasonable” with remaining solutions being vague/inaccurate (31%, 5/16) or non-solutions (13%, 2/16).

Looking at Step 8 (Wrong Destination), 100% (29/29) of solutions generated by control participants were “reasonable”. In contrast, 68% (13/19) of solutions generated by participants with ACI were “reasonable” and similar to controls, but 11% (2/19) were vague and 11% (2/19) were non-solutions. We believe it is worthwhile to look in a bit more detail at the potential solutions given at the two foils, step 4 and step 8.

STEP FOUR (Missing Step)

We will first look at the solutions offered by the Control Group (15 generated), and then those generated by the ACI group. The correct solution was to “turn Right on 9th and walk towards A St”.

• “Walk towards A and then go Right on A St.”

• “I guess I’d have to turn Right onto 9th, go South, to get back up to A St.”

• “Take a Right here, because I know A St is South of me.”

• “I think somebody gave me the wrong directions. They meant ‘Turn Right onto 9th’”.

• Two other solutions use a strategy of gathering more information:

• “Walk on B for a while and see if it did run into A St.”

• “I was just gonna ask, just now I was thinking, and I saw someone on the street, but they told me to talk on the phone if I was lost.” [failed attempt to ask person on street for directions].

We can now compare how the ACI Group did on the same problem (16 generated solutions). Five solutions were similar to the control group and correct. The remaining 11 solutions were unique to the ACI group. We view four to be reasonable solutions, with two examples being “go back to the starting point and start over” and “I think the streets are on a grid so I should be able to use that.” Another six solutions we marked as vague, inaccurate or not useful in the current context. These include “look for the bookstore”, “go/turn [wrong place/direction]”, “go back to a better part of town where people know me better or I know the place better, if at all.” The remaining solution was to abandon the entire enterprise: “Look for a bus stop & go back to Eugene”.

STEP EIGHT (Wrong Destination)

Solutions offered by the Control Group (29 generated), and those generated by the ACI group (19 generated) had similarities. In the list below, the notation a/b denotes the number of subjects in the control group (a) that offered the solution, and the number of subjects in the ACI group (b) that offered the solution.

• Back-track and try again (9/5)

• Walk a little further and keep looking (8/1)

• Ask someone where is a bookstore (4/5)

• Back-track and look down the side streets as well (3/1)

• Go one block South to see if the other One-Way could also be Main St (2/1)

• Take a Right onto Main instead of Left (1/0)

• Look up “bookstores” in a phone book (1/0)

• Is it Brethren Housing? (has some blue and has a store) (1/0)

We marked six solutions as unique to the ACI group:

• Give up (4)

• “I’ve got to look for a house with a blue roof” (1)

• “Is there a bookstore?” (1)

7 Summary and Discussion

Results of this study confirmed the prevalence of navigational challenges faced by brain injury survivors, even on a short pedestrian route. Participants with ACI demonstrated significantly greater on-route navigational challenges – more frequent errors and hesitations – than matched controls. Participants with and without ACI exhibited different types of problem solving. The ACI group requested assistance over the cell phone more frequently than controls, and required more attempts at re-orientation with concrete, salient directions in order to re-orient in the field. Participants in the control group anticipated errors with greater frequency than those with ACI.

We chose to provide participants with a cellular phone for two reasons: (1) we wanted to capture real-time participant insights relevant to getting lost, and (2) we wanted a flexible means to provide route re-orientation to explore effective strategies. All participants highly endorsed the cell phone as a useful tool for both reorientation and reassurance.

Analysis of the phone helper transcripts revealed several important implications for providing on-route assistance to travelers with ACI. Participants reported they might have quit the route without assistance and support. We also discovered that it was important to explicitly ask the participant to stop walking and remain at a given location while the phone helper attempted to provide assistance. When participants continued to move (e.g. cross a street or face a different direction), left/right directional assistance became irrelevant. It was also important to verify the participant’s current location. In one instance, the phone helper was unable to re-orient a participant who reported inaccurate information about his current location. In addition, it was critical to provide specific instructions that utilized landmarks for re-orientation. In several instances during this study, the person with ACI required multiple re-orientations over the telephone when the phone helper assumed to know the participant’s location and orientation to provide left/right street directions. We discovered that the only way to successfully re-orient these participants in the field was to provide explicit landmark re-orientation. For example, one participant with ACI required five attempts at re-orientation before the phone helper successfully described salient landmarks (i.e. face the blue house) to get her back on track. It should be noted that the phone helper in this study had access to photographs at each intersection along and near the route. For care providers who do not know the neighborhood or do not have access to such pre-planned information, GIS technology paired with GPS information may provide a useful supplement, such as the Street View images now available in many metropolitan areas provided by Google maps (maps.google.com). Of course, one might argue that the GPS on the phone could reliably place the participant at a certain point and facing, and that there is no need for the phone-helper to ask the participant to self-locate. We will discuss this shortly.

The current study results demonstrated that individuals with ACI perform pedestrian navigational tasks with greater errors and hesitancy than matched controls. Although there was insufficient power to investigate the relation between severity of cognitive impairment and getting lost behavior (due to the small and skewed sample that included only two participants in the severe range), the trend suggested that individuals with more severe cognitive impairments demonstrated greater difficulty. Additional qualitative research, especially in-depth case studies of individuals who are either successful or unsuccessful navigators, may reveal important cognitive predictors of navigational performance. Future research must also continue to evaluate the potential effectiveness of various assistive technologies to improve navigational performance. One problem is accuracy: a GPS system carried by the traveler may be able to prompt a traveler to pull a bus-cord (our own work, and that of others [13], has born this out), but not be able to provide location or orientation data accurate enough to place a pedestrian at the correct corner of an intersection, and a specific heading [14]. However, even if a GPS device provided accurate enough data, there remains the issue of two-way interaction. We are dubious that an in-car style of assistant, programmed to replan a route when an error is detected, without feedback from the user, will be effective with either (a) pedestrian situations, in general, or (b) travelers with ACI, in particular. It may be worthwhile to discuss our prior “Springfield-1” trial in this regard [1], which used four matched routes of roughly 300 yards each as shown in the Fig. 7.2 (taken from the paper).

Fig. 7.2

Matched routes of Springfield-1

In Springfield-1, we provided perfect, proactive, route-following directions, achieved by using a wizard-of-oz technique to cue instructions at appropriate points. Subjects frequently encountered challenges in following these perfect directions. Researchers stepped in at these points to (a) ask further about what led to the problem, and (b) place the subject back on track. It never crossed our mind to allow subjects to keep walking past the correct choice, and then reroute them around city blocks (as an in-car system would typically do). First, walking is not driving, and blocks can be long. This tends to be a population that does not get out for physical exercise, and taking a rambling tour of the urban core would not be viewed as recreation. In particular, 12 out of 20 of our participants in Springfield-1 had to take significant rest-breaks within the trial. Second, there is every reason to believe that a subject would miss the next choice point if they just missed the current choice point; the problem is not one of re-routing, but debugging why the person is making errors. Also, at least for some, part of the problem is the need for reassurance and emotional support that can overcome their anxiety of walking in an unfamiliar location. In essence, the navigation problems we saw arising with our subjects in Springfield-1, even with perfect directions, led us to want to know more about errors and problem solving, and hence, the study we report here.

Our follow-on to this study has been exploring how we can augment the human in the loop (i.e. the phone helpers in our study) with a computer-based helper. We have found a service point in Eugene that has potential: the travelers’ hotline maintained by the local transit district. Admittedly, it is not always on-call. And it focuses solely on problem-solving in relation to public transportation, and not pedestrian route-following (but does include walking to and from transit stops). On the other hand, we believe it is a good place to start given a bus ride is a likely part of most trips with this population. Following up with our results from this study, we are working with hotline personnel to provide them with scripts tailored to individual travelers. On a call to the hotline, the traveler’s phone can supply GPS information. Just as importantly, the phone can supply personal information about the traveler (e.g., impairments, ability to use landmarks, solutions that have worked in past) that can allow (we conjecture) hotline personnel to provide efficient problem-solving help. The study has just started, and we hope to report results in a future meeting.