Keywords

These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

1 Introduction

Improv…allows the designer to explore the design solution in all sensory and cognitive modalities…in a way that cannot be achieved through mere graphic design or well-plotted “user scenarios.” While the actor uses empathy to perform dramatic characters in scripted situations, the designer uses empathy to perform design solutions that are drawn from deep identification with real, individual people in specific situated contexts in the real world.

-Brenda Laurel

Design Research: Methods and Perspectives Laurel (2003)

Our current work explores, how design engineers can draw out people’s implicit intuitions and expectations for how interactions with machines should transpire. In this paper, we focus on our use of semi-structured, improvised interaction sessions in order to prototype devices and platforms, experiment with prompts and responses, and test social norms.

The approach evolved from our research in physical interaction design and robot mediated communication: domains that rely on designers’ knowledge and intuition of embodied action, which are often only implicitly understood, and therefore can be difficult to verbalize. Our use of collaborative improvisation has therefore been particularly helpful in revealing, and making explicit, these internal thought processes.

2 Approach and Background

Collaborative improvisation engages experts in physical expression, as well as typical users, in brief, everyday interactions with prototype technologies. We improvise prompts to interact, then examine and discuss their responses, to reveal how people perceive these devices, think and feel about interacting with them, and behave towards them. Through these sessions, we learn about the needs of users and cultures whose backgrounds may vary considerably from our own, along dimensions that include remarkably different physical abilities or limitations, notions of rational behavior or use, or expectations of the technology’s sensibility, usability or longevity (Dourish 2012).

Even considering a diverse population of potential users, to each individual, home and work are familiar (and often comfortable) environments, and it can be difficult to question, or evaluate, the role that some novel technology—such as a robotic appliance or furnishing—could take. As Bell et al. (2005) highlight, one challenge in the study of everyday settings is the difficulty of asking questions about what seems obvious [they cite Norman’s (1988) discussion of the affordances of glass “for seeing through and for breaking”], how this naïve questioning defamiliarizes the familiar, and thus supports our efforts to evaluate its significance.

Our approach to conducting improvisation sessions builds upon Gerber (2007), which, in turn, is based upon Johnstone’s (1989) exercises to help stage actors perceive and project roles and relationships, respond with spontaneity, and develop skills in narrative storytelling. Gerber maps these concepts over to design activities, and provides examples of how they can be applied to promote collaboration, learning through failure, and skill in presentation—for example, through a group sketching activity where individuals alternatively add single marks. We then implement these concepts through group storyboarding of potential user scenarios, physical and video prototyping, design improvisation sessions, and exploratory lab or field experiments (Sirkin and Ju 2014).

From a methodological perspective, our approach to field and lab studies is similar to the breaching experiments espoused by Garfinkel (1967) as a way to understand the “seen but unnoticed, expected, background features of everyday scenes,” and used by Weiss et al. (2008). Weiss and her colleagues deployed an ACE robot in a public shopping area, and approached unsuspecting people with tourist guide information. They then subsequently interviewed people about the interaction to gauge how accepting people were to this discrepant event. This type of event tells us something different from longitudinal deployments of robots in shared settings [a nice survey of such research is found in Leite et al. (2013); a strong case study in Kidd and Breazeal (2008)]: it tells us how people who have incidental and short-term interactions interpret and respond to robot overtures. This class of research has relevance not only to situations where robots are novel, or where robot interactions are passing (Lee et al. 2009), but also to situations where people maybe encountering robots in emergency or disaster situation (Bethel and Murphy 2010), when the nuances of first impressions may be critically important.

Most of our improvisation sessions and experiments employ Wizard of Oz techniques, in which participants are given the impression that they are interacting with fully functional systems, while their interactions are actually mediated by a human operator. Depending on whether activities are focused on ideating or testing, participants may be aware of the human wizard or not. The approach allows both experimenter and participant more freedom of expression, or more systematic constraints, than would be possible with a fully realized system (Dahlbäck et al. 1993). From a design perspective, Wizard of Oz permits the rapid deployment of prototype technologies into naturalistic settings early in the design process, to inform the context of use, to mine the real world for naturalistic social interactions that the device will need to generate and respond to, and to understand critical technical limitations inherent to the application (Riek 2012).

3 Case Studies in Design

Our case studies center on the design of interactions between people and expressive everyday objects: specifically, how non-anthropomorphic robots can, and should, interact with people. So what are robotic everyday objects? Our projects include an ottoman that offers to support a seated person’s feet, a dresser drawers unit that opens to reveal the right tool just when someone needs it, and a trash barrel that roams around a dining area and collects trash from diners completing their meals. We study such objects in motion because interactivity implies sociability, and the mindful design of such near-future autonomous technologies can help avoid social miscues, mismatched conceptual models, and unmet expectations. Over the course of these projects, our goals have been (a) to understand how people respond to novel, agentic devices during their everyday activities, as well as (b) to develop a methodology for creating interactions that read naturally to the people involved. This type of design research takes place within the broader context of embedded computing, smart devices and home automation.

The following sections describe how we designed these three prototypes, and used improvisation to evaluate how best to interact with people, as well as the range of responses these devices received in use. We seek to understand both normative behaviors, which are common and expected, and individual responses, which are unique and often more idiosyncratic.

3.1 Mechanical Ottoman

3.1.1 Introduction

The mechanical ottoman is a household robot that approaches a seated person from across a room, offers to support his or her feet, and after doing so for several minutes, requests to take leave of the ongoing interaction. Since the main joint activity in human-ottoman interaction is fairly static, we were primarily interested in the question of how engagement and disengagement occur. We therefore developed the ottoman, and a natural interaction scenario (see Fig. 1), specifically to study how a non-anthropomorphic robot can initiate, participate in, and then disengage from, a joint activity with a human partner (Sirkin et al. 2015).

Fig. 1
figure 1

A participant in an improvisation session accepts the ottoman’s offer to support his feet

Full size image

3.1.2 Prototype Systems

Although we are designing the intended functions and workings of autonomous systems, our approach towards development is to produce quick, inexpensive prototypes that enable us to explore possible forms and functions. We therefore built a low-resolution functional prototype using an inexpensive store-bought ottoman, which we set atop casters, and steered around the floor by hand using 2-m long wooden dowels that we attached to its bottom with gaffer’s tape.

We subsequently built a robotic, teleoperated ottoman, modeled on the earlier prototype. We set the ottoman atop a modified Willow Garage Turtlebot, which is based, in turn, on the iRobot Create robotic base. We attached a servo to the topmost portion of the internal frame to raise and lower the ottoman 2.5 cm (vertically), making sure that we reinforced the internal assembly to support the weight of a person sitting on it. A concealed researcher then remotely controlled robot’s path and speed across the floor, and its vertical motion.

3.1.3 Designed Behaviors

Through early prototyping and improvisation sessions (described next) we decided that the robot would (a) initiate and conclude each movement with a vertical lift or drop motion, (b) start rolling at a speed of about 1 m/s, (c) approach a seated person following a curving path, always within his or her immediate field of view, (d) pause movement at a distance of 1 m for about 10 s, and (e) resume its path toward the person, slowing to about ½ m/s as it drew near [proxemic movement patterns were informed by Hüttenrauch et al. (2006) and Michalowski et al. (2006)].

If the person did not immediately raise his or her feet, the ottoman would begin a sequence of three increasingly assertive actions. First was a brief lift and drop, next was a quick rotational wiggle around its center, and last was a gentle nudge, or bump, up against the person’s legs or feet, alternatively leading from the left or right side. Once someone’s feet were actively being supported, the ottoman would then follow a similar sequence to bid to leave.

3.1.4 Improvisation Sessions

There were two distinct phases of design improvisation for the ottoman: an initial, developmental phase using the hand-puppeted prototype, and a subsequent experimental phase using the remotely-controlled robotic prototype.

3.1.4.1 Phase 1: Developing Behaviors

We held three design sessions, each lasting about 2 h, with domain experts in physical movement and interaction, including (a) a dance choreography instructor, (b) an improvisational theater performer and theater director, and (c) a stage actor. We placed these participants in various individual seating arrangements, and puppeted the prototype (see Fig. 2), exploring ways that someone could beckon or dismiss the ottoman, what personalities different speeds, gestures or angles of approach and departure projected, and appropriate social distances. We engaged participants in role play, using prompts such as “shoo the ottoman away as many ways as you can,” and encouraged them to respond gesturally and speak their reactions aloud.

Fig. 2
figure 2

During an improvisation session, the researcher on the right puppets the prototype using two wooden dowels

Full size image

These sessions often pointed out how social and cultural the interpretations of actions were. At one point, after the robot quickly approached to about a meter away, followed by a pause, then a gentle move closer, the dance instructor declared “Ah, now we’re in India,” evoking the tradition of exemplary service being viewed as an art form in that culture. She elaborated by contrasting India, or England, to other cultures, such as the United States, which have historical frames of service being viewed as closer to servitude. The improvisational theater performer echoed this sentiment when the ottoman withdrew to a ready position in front and to his side, stating “That feels like butlery,” noting how this action showed an intent to be useful, and allowed it to be recalled to service quickly. The stage actor treated the ottoman more like an animate object, which he could wave over if needed, sneer at if he wanted to cross his legs, or kick gently aside if it encroached too closely into his personal space. Each of these perspectives placed the seated person at the top of the social order, with the ottoman alternatively treated as provider of expert service, obedience or pure functionality.

Several participants felt that a brisk, vertical lift movement of several centimeters (which we called “stand up”) suggested attention and a readiness to move, and that the corresponding drop (or “sit down”) movement suggested stability, and a likelihood of staying put after completing some action.

3.1.4.2 Phase 2: Exploring and Interpreting

Next, we engaged 20 participants in a lab study, which recreated a natural interaction setting, and explored how they interpreted and responded to the robotic prototype’s designed behaviors. We asked participants to sit in a lounge chair and watch a video. After a few minutes, the ottoman approached from a position either nearly in front of them, or off to their side (see Fig. 3). The operator improvised interactions with them, encouraging them to engage, without following a script.

Fig. 3
figure 3

A typical path taken to approach a seated person. Squares represent the ottoman’s position, one second apart

Full size image
3.1.4.3 Approaching

Almost all participants recognized the ottoman as a robotic footstool, or as one participant described, “a weirdly sentient footstool.” One or two who were confused at first quickly came to understand its role and intent: “At some point I thought it could be a small chair or something, but I pretty much got it when it was approaching my feet.” This suggests that not only the robot’s form, but its movement, informs how people interpret its role.

Most people accepted the offer to rest their feet on the ottoman, many of them lifting their legs right away as it rolled up toward them (see Fig. 4). About half even held their legs suspended in the air for several seconds, so the ottoman could settle down just beneath them. Those who did not rest their feet knew that it was an ottoman, and that it was offering to support their feet, and described their reluctance as perceiving the ottoman to be alive: “It’s a moving thing that I almost perceive as living. I didn’t want to denigrate it by using it as a footstool.” Another said, “Because it felt like it was alive I didn’t want to put my feet on it.” and “I feel like it communicated with me well, but I would feel uncomfortable doing what it was asking.” One other participant rested her feet at first, but removed them after about 30 s, saying later, “It seemed like it wanted me to put my feet on it, but I didn’t want to constrain it too much. I didn’t want to imprison it here.”

Fig. 4
figure 4

From left to right: one participant anticipates the arriving ottoman by elevating his feet; another pats it to assuage its offer to support her feet; one rests his feet on it while watching a video; and another avoids interacting with nervous laughter

Full size image
3.1.4.4 Taking Leave

Everyone who had rested their feet recognized the quick lift-and-drop motion as a request to take leave of the interaction, although some did not notice it until the second or third time: “It, like, sat up when it was ready to go.” and “When it signaled to leave, it rose and fell, to let me know it was time to go. I wasn’t ready for it to leave because I was comfortable. I’m glad it let me know instead of taking off.” Responses to the ottoman’s request to withdraw ranged from accepting, “I think it wanted to go, so I set it free.” to disappointed, “It made its own decision to leave, like I don’t want to be your footrest anymore.” to mildly annoyed, “I was a little offput when it decided that it wanted to leave. If it was doing that all of the time, then I’m not sure how good of a footrest it would be. I expect a footrest to be there.”

Participants had several rationales to explain its early departure, which ranged from attending to other routine tasks to taking care of someone else: “I thought it probably had something to do, to go do some errands.” “Maybe someone else needed a footrest, or it needed to charge itself.” and “There was probably somebody more important in the room, so it was going to meet that guy.”

3.2 Emotive Drawers

3.2.1 Introduction

The emotive robotic drawers are designed to help us explore how a robot can participate in an iterative, turn-taking activity with a human collaborator, using its movements alone. The shared activity can be any task where the human needs repeated access to objects stored within the robot’s drawers, and where it can open and close in ways that encourage and support task completion. The example that we explored most closely is a cube assembly puzzle (see Fig. 5), which requires six different fasteners, and where the drawers contain all of the tools required to build the cube Mok et al. (2014, 2015).

Fig. 5
figure 5

Emotive drawers performing an animation gesture while a human collaborator assembles a cube puzzle

Full size image

3.2.2 Prototype Systems

We fabricated the prototype from a standard IKEA MICKE 4-drawer unit. To allow the drawers to perform consistent and repeatable motions, we retrofitted the top three drawers with motors and a rack-and-pinion system. Spring-loaded rotary encoders mounted against the unit’s frame allowed us to track each drawer’s position. The bottom drawer was taped off from access during the study and contained parts and hardware that drove the system. Actuation was controlled by an Arduino microcontroller communicating with a local client program over USB cable.

The local client program provided a Wizard-of-Oz style remote control for the drawers. A researcher in an adjacent room operated the drawers via keyboard hotkey with 15 buttons, each of which was bound to a pre-programmed sequence of movements, or “animations,” for the drawers to execute. The researcher had a one-way video feed of both the drawers and participants, which was used to observe participants’ actions and improvise appropriate responses.

3.2.3 Designed Behaviors

Animations were modeled to be either non-expressive or expressive. The non-expressive case used only the most basic, functional movements, and included simple open (Fig. 6a), where each drawer opened at a constant speed, and simple close (Fig. 6b), where all of the drawers closed at the same constant speed and locked after closing.

Fig. 6
figure 6

Drawer animations include (a) simple open, (b) simple close, (c) flair close, (d) wiggle, (e) beckon, (f) chuckle and (g) happy

Full size image

The expressive case added flair to the closing animation, and more communicative gestures. For close flair (Fig. 6c), after one drawer closed, the other two would open and close slightly in sequence, mimicking a ripple effect. Other animations were designed to suggest to the participant that he or she return or remove an item from an open drawer. For the former, the drawers would wiggle (Fig. 6d), where an open drawer moved in and out slightly, but quickly, two times. For the latter, the drawers would beckon (Fig. 6e), where an open drawer closed halfway, and then reopened at half speed. Two final animations suggested positive sentiment. For chuckle (Fig. 6f), all of the drawers opened to random positions, wiggled twice and then closed. For happy (Fig. 6g), all of the drawers mimicked a ripple effect, which traveled down the drawers unit twice.

3.2.4 Improvisation Sessions

Twenty people participated in a lab study, which simulated a mechanical workspace within an office environment, to interact with the drawers and complete the cube puzzle. For each session, the robot was either non-expressive or expressive. Additionally, it took on one of two levels of assertiveness: proactive, where the drawers led the activity by initiating actions, or reactive, where the drawers waited for gestures by the participant before responding. We gave participants very little guidance about how to engage the drawers, hoping that they would negotiate an approach that made sense to them. The operator could then freely improvise the robot’s behavior, using the animations available for that session’s persona.

3.2.4.1 Proactive Action

When the drawers were proactive, participants felt that the robot was not treating them as equals. To one, the robot appeared very much to be “Like a boss.” figure that relegated people to a lower social standing. Another indicated that “I was the builder, the drawers should not command me to do things. I will do it when I am ready.” Yet another felt that “It was distracting when I was trying to understand what’s going on. I knew it was trying to get me to get the tool for the next step, but I didn’t know what to do yet. So I just ignored it until I was ready.” Conversely, when the drawers were expressive, they did not create this feeling of inequality. One participant noted, “It was like a fiery little Scottish Terrier trying to pull me its way.” Several others similarly noted that it was “Like a pet.” So, by incorporating an expressive nature into its actions, the robot can still help lead interactions, while making participants feel more like equals, and assuaging the frustration and discomfort that they might otherwise feel.

3.2.4.2 Expressive Movement

The expressive movement displays were also effective at making an impression on participants, regardless of their immediate focus of attention. For example, participants’ attention was often drawn somewhere other than the drawers—say, toward the cube puzzle task—while an expressive drawer performed an animation. Yet they later recall having seen the animation, with a large percentage realizing that “It was trying to congratulate me.” Regarding the chuckle animation, one participant noticed, “It was warning me of an error.” So, even without looking directly at the drawers, participants still experienced their movements. This reaffirms Hoffman and Ju’s (2014) finding that people are sensitive to movement, and that small, well-designed motions can be used to communicate.

3.3 Roving Trash Barrel

3.3.1 Introduction

We built and tested a roving trash barrel robot to better understand the implicit social protocols, cues and signals of public interactions between people and robots, and to produce movements and actions that people can understand. In eight lunch-time sessions, spread across two heavily populated campus dining destinations, we piloted the robot in Wizard-of-Oz fashion through crowded public areas, initiating and responding to requests for impromptu interactions around collecting people’s trash (see Fig. 7).

Fig. 7
figure 7

A lunchtime diner discards his trash into the roving trash barrel after it approached and gestured to draw his attention

Full size image

3.3.2 Prototype Systems

The trash barrel robot was designed to resemble those commonly found around the university’s campus. Its body is a standard 32 gal BRUTE gray trash barrel from Rubbermaid’s line of commercial products (see Fig. 8). The barrel is mounted atop a robotic base, powered by an iRobot Create, and augmented with a laptop computer (concealed within the trash barrel), two web cameras (hidden within the barrel’s grab handles) and a microphone (mounted under the barrel’s topmost lip).

Fig. 8
figure 8

A standard Rubbermaid Brute trash barrel is mounted atop an iRobot Create platform. A laptop computer, hidden within the trash barrel, handles video and control commands

Full size image

A researcher remotely controls the robot (Fong and Thorpe 2001), issuing commands over a web interface to a remote server, which relays the control commands over WiFi to the laptop, and then by USB to the robotic base (see Fig. 9). The interface shows the operator two video streams from the cameras in the robot’s handles. The choice of teleoperation over autonomous control provides flexibility for real-time improvisation, and permits responsive behaviors to unanticipated events.

Fig. 9
figure 9

The roving trash barrel robot’s control system. Cameras on the robot send video to a remote operator’s interface on a laptop, which sends motion commands back to the robot

Full size image

3.3.3 Designed Behaviors

Owing to its iRobot Create drivetrain, the trash barrel could move forward and backward, straight or along an arc, or rotate in place: it was incapable of lateral motion without first rotating. This limitation, combined with the use of cameras to view and interact with people, created an implicit front for the otherwise cylindrical trash barrel, and meant that the robot “faced” a person or group as it approach them.

Aside from speed and direction control, the robot had three pre-programmed behaviors: wiggling, where it quickly rotated left and right, nudging, where it abruptly moved front and back, and beeping, where it played a neutral two tone beep. At high speed, the wiggling and nudging movements appeared more like shivering and nodding, respectively. The combination of basic drive and programmed behaviors allowed the operator to maneuver the trash barrel, and signal intent, improving the legibility and predictability of its actions (Takayama et al. 2011; Dragan et al. 2013).

3.3.4 Improvisation Sessions

We conducted eight sessions, during lunch at two busy dining locations, in which we explored different ways to approach, gesture, disengage and acknowledge people. Because our goal was to elicit naturalistic behavior, we did not pre-warn potential participants that they could be engaged in an improvisational interaction, but we did interview those who actively engaged with the robot afterward (155 in total). Due to the crowd and noise, people were rarely aware of the trash barrel until it approached within an arm’s length distance. It could therefore wander throughout the area, interacting opportunistically with nearby people along its path, without their anticipating its arrival.

There were two phases of improvisation. In the first, exploratory phase, the trash barrel wiggled, nudged, bumped and beeped to attract attention. It could even interrupt conversations and make excessive noise by dragging along empty chairs. In the second, goal-oriented phase, the trash barrel followed a loose script to encourage people to discard their trash. Scripts called for the robot to either (a) visit every table along a set path and enact a simple stop-and-move-on behavior at each one, (b) initiate and respond to requests to engage people by responsive movement, or (c) purposely appear to struggle, by bumping into chairs or uneven pavement, with the intent to elicit empathy.

3.3.4.1 Interrupting Activity

People who were alone were less likely to engage socially with the robot, but if several people in a group noticed it, they would engage extensively. Overall, people did not readily interrupt ongoing activities, whether they were watching a video alone or socializing with friends. This correlates with results by Hüttenrauch and Hüttenrauch and Severinson-Eklundh (2003), who found that people do not attend to robots when they are already engaged in their own activities.

3.3.4.2 Overt Non-interaction

Most people appreciated the robot’s presence and offer, particularly when they were in small social groups, where the person closest to the robot could check the responses of the others before deciding that it was okay to engage. The most common way for people to signal their unwillingness to interact was to ignore the robot’s presence (Fischer et al. 2014), although some people overtly avoided interacting by averting their gaze or turning their faces away, and not responding to the robot’s actions as it drew near. One woman kept a hand by her face, as if to hold her hair back from her eyes, but always at an angle that kept her from visually connecting with the robot. Others avoided looking toward the robot as it approached, but turned around and recorded video as it departed to interact with others. Still others performed short, curt interactions, and then pointedly turned away to indicate that the engagement was over. More observations from this study have been written up in Fischer et al. 2015.

3.3.4.3 Ascribing Desires

People often waved to attract, or shooed to dismiss, the robot. The most common gesture to call it over was to wave trash in its line of sight. This pattern is consistent with the material signals that people use when coordinating joint action, as pointed out by Clark (2005). But we also observed actions that went beyond just signaling for coordination. People appeared to ascribe desires and motivations to the trash barrel, that it intrinsically desires trash rather than just performs a fixed collection routine. One demonstration of this belief involved a family with a 5-year-old son. Over 30 min, as the robot interacted with others, the child re-approached the trash barrel on four occasions, waving trash directly in front of its cameras before slowly pulling the trash back, in an attempt to coax the trash barrel to follow him, as if the boy’s mental model of the robot included its intent to consume trash, despite his never having been informed as such. Adults also seemed to hold this mental model. Several described throwing away trash as “feeding” the trash barrel, and expressed a desire for it to acknowledge, or thank, them for having given trash to it.

3.3.4.4 Empathy and Altruism

Both accidentally and intentionally, the trash barrel made a series of mistakes and exhibited struggling behavior—where it repeatedly tried to reach an objective, but was unable to overcome obstacles, such as furniture or pavement edges. Some people felt that the robot was “Not very smart.” and exhibited “Erratic behavior.” But others found struggling to be endearing, characterizing the robot as being like “A puppy or a toddler.” One said, “When it ran into the garbage cans, I thought, ‘silly robot!’ It was adorable.” This attitude appeared to encourage them to move obstacles, such as chairs, out of the robot’s way. When asked later why she had done so, one person replied, “I don’t know, it felt like it was a team effort.” Another person “Noticed that the robot made people laugh and smile, and when it got stuck, I wanted to help it because I thought, ‘the show must go on.’”

4 Next Steps

The subject of our embodied design improvisation approach has been to build interactive robots that co-inhabit spaces with people. As a next step in extending this method, and in order to design gestures and movements that are more meaningful and socially appropriate, we propose to utilize the “wisdom of crowds” to help teleoperate robots. We are interested in better understanding when the crowds are indeed wise, and when they need assistance. We would like to know how best to aggregate crowd input, to handle transfers of control, and to give remote workers situational awareness.

We expect that experimentation with this crowdpowered embodied interaction approach will help us to elicit context and culture-sensitive rules of interaction that are key to designing new products; that these techniques will lead to insights that are more easily codified into machine action than those generated from live, in-person improvisation; and that end users will respond more positively to ideas generated from more people than to those developed without the benefit of as much collaboration.