Keywords

These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Introduction

As computing becomes embedded in the physical environment, we need to ­understand how to support users to enact appropriate input actions and understand related system outputs [7, 9]. Image schema and conceptual metaphor theory may provide insight into underlying motor-cognitive mechanisms that can be leveraged in interaction design for embedded systems [1, 6]. Most of the research to date on metaphor in human computer interaction has focused on the use of metaphor in the design of visual communication elements of graphical user interface or in understanding users’ mental models of such interfaces. In this chapter, we focus on one way that metaphors may be used to design the interactional mappings between input actions and output responses for whole body interaction in an abstract domain.

In previous works we explored the idea that conceptual metaphors, derived from image schemas and operating outside of conscious awareness, could be used to create systematic and predictable or “intuitive” relationships between specific human actions and specific system responses for a whole body, audio environment called Sound Maker [2]. We called these embodied interactional models and claimed that they constitute a design strategy that has several benefits [1]. Based on our investigation of Sound Maker, we found evidence that an embodied interactional model made it easier for both adult and child users to learn to use the system and resulted in an enjoyable experience [3]. We also found that users tended to interpret the input space, their actions in it and their effects on the system using a spatial perspective on meaning making rather than a body-centric (physical) interpretation.

In order to validate our previous findings and extend our work to include the visual modality and a very abstract domain, we have created another interactive environment, called Springboard. Springboard is a room-sized interactive environment where users can explore how solutions to issues in social justice can be balanced through balancing their bodies spatially (i.e. in a space) or physically (i.e. centre of gravity). This paper briefly describes the theoretical foundation, the design rationale, and the system implementation. Springboard is a research instrument that allows us to continue to empirically investigate the details and benefits of embodied interaction in embedded computational systems. We asked the following questions: What are users’ general attitudes to this embodied interactive environment? Do they find it easy to use? Does incorporating an embodied metaphor for balance into the interaction model impact users’ understanding of how to use the system? Is there any difference between users’ interactions with and understandings of the system where its interaction model is structured using a spatial schema for balance compared to a body-centric schema for balance?

Theoretical Underpinning

Embodied Metaphor Theory

A conceptual metaphor is the interaction between a target domain and a source domain that involves an interaction of schemas and concepts. Image schemas in the source domain are used to structure understanding of concepts in the target domain through metaphorical elaboration. Johnson claims that metaphors arise unconsciously from experiential gestalts relating to the body’s movements, orientation in space, and its interaction with objects [8]. He called these fundamental gestalts embodied schemas, also called image schemas. Johnson suggests that a cornerstone of human meaning-making is our ability to form conceptual metaphors by using the structural and inferential properties of image schemas to structure and organize abstract concepts.

There are only a few studies that apply embodied metaphor theory in human computer interaction. The general premise of this work is that interfaces or interaction models that are consistent with metaphorical elaborations of image schemas will be more effective, efficient and satisfying to use. For a general discussion of the role of image schemas and conceptual metaphors in user interfaces see [7], and in interaction models see [1].

The Meanings of Balance

Johnson presents an analysis of the meanings of balance that include experiences, perceptions and concepts of balance. He suggests that our experience of balance is so pervasive and basic that we are seldom aware of its existence. The structure of balance is a key element that pulls our physical experience together as a coherent whole. For example, a toddler learning to walk immediately experiences various states of bodily balance and imbalance (Fig. 2.1, left). We learn about balance with our bodies. The meanings of balance emerge through acts of balancing. Long before we have grasped the meaning of the word balance, we develop several image schemas for balance based on our physical experiences.

Fig. 2.1
figure 1_2

Image schemas form from experience and are elaborated through embodied metaphor to structure abstract thought

Full size image

As children develop, image schemas related to balance begin to structure, and give coherent meaning to their perceptions. In the realm of visual perception, we soon learn to interpret visual imagery as balanced or imbalanced. An image with a black circle placed at the interior edge of a square is interpreted as less balanced than an image where the circle is in the middle of the square. The image schema for bodily balance (around a point) structures this interpretation. Balance or imbalance does not objectively exist in the images. Balance comes through our act of perception and our interpretation that utilizes a balance image schema.

Balance image schemas are also used to give meaning to balance in abstract domains such as psychological states, legal systems, mathematics, and social justice (Fig. 2.1, right). Through metaphorical elaboration, we interpret an abstract concept of balance based on its similarities with one or more image schemas for balance. For example, when we speak of social justice, we infer that justice involves a balance of factors such as rights, privileges, damages, and duties. Our understanding and judgments arise from the twin-pan balance schema (Fig. 2.1, centre). We treat factors metaphorically like forces or weights in the pans of a scale. The scale can be imbalanced by either side of the fulcrum having too much or too little metaphorical weight or force. For a detailed discussion of balance schemas and their metaphorical elaborations, see [8].

Design Rationale and System Implementation

Our multimedia interactive environment, Springboard, uses a camera vision system to sense and analyze data about a user’s centre of gravity and foot position on an input platform. The system interprets sensed data to determine the degree of the user’s body and/or spatial balance. This information is sent to the image and sound display engines that use it to determine which images and sounds to display. The design rationale and a brief system description follow.

An Abstract Domain: Social Justice

In our earlier work we explored the benefits of an embodied metaphorical interaction model in an acoustic environment. As pointed out by several reviewers, changes in sound parameters are more perceptual than conceptual and are more physical than abstract. The concept of balance in social justice is very abstract. It also lends itself well to visual representation. As such, it was chosen as a suitable domain to use in this phase of our investigations.

We chose to design a system based on three different issues in social justice so that we could create three content sets for our user study. We repeat the same task three times, each time with the same system but a different content set (and issue) in order to improve validity of the study. The issues we chose can each be conceptualized based on the metaphorical elaboration of the twin-pan balance schema. Each issue was represented through visual media and was complex enough to sustain user interest. Based on these criteria, we chose the following three issues that each have two main factors or metaphorical forces which must be balanced to achieve a socially just solution:

  1. 1.

    balancing ethical farming practices versus adequate quantity of food production (abbreviated food)

  2. 2.

    balancing resources used to build shelters versus quality of shelter production (shelter)

  3. 3.

    balancing the authority of state control versus community safety (safety).

We acknowledge that we have vastly simplified these complex social issues by conceptualizing them as the balance of two main factors. However, for the purposes of initially studying metaphor-based embodied interaction, we feel that the conceptualization is consistent with the twin-pan balance schema and adequate to reveal benefits or limitations to our approach.

Design Goals

Our main objective was to create a system that we could use as a research instrument to explore the benefits and limitations of using an embodied interaction model in a whole body environment that enabled users to explore the abstract idea of balance and imbalance in social justice. The interaction model is the mapping layer that relates input actions to changes in displayed images and sounds. A second objective was to explore the similarities and differences between using a spatial and a physical image schema as the basis for the embodied interaction model. From these objectives and our user study design requirements, we set four primary design goals.

The first goal was to create an interaction model that used the twin-pan balance schema to structure the input sensing space. The twin-pan schema can be enacted by a user either physically (e.g. moving or standing with their centre of gravity in or out of balance) or spatially (e.g. moving or standing in the centre of the input space or at the edges of the input space). The second goal was to create an interaction model that used the following three schema-concept pairings:

  1. 1.

    a spatial twin-pan balance image schema (i.e. based on body position being balanced or imbalanced in a space) that was linked to displayed images and sounds representing the concept of balance and imbalance for each issue (abbreviated “spatial” mode);

  2. 2.

    a body-centric twin-pan balance image schema (i.e. based on centre of gravity being balanced or imbalanced) that was linked to the same displayed images and sounds (abbreviated “body” mode); and

  3. 3.

    a combination of both spatial and body-centric twin-pan balance image schemas that was linked to the same displayed images and sounds (abbreviated “spatial  +  body” mode). The interaction model links image schematic input states to changes in output imagery and sounds, depicting various degrees of balance in each social justice issue. The system can be run using spatial, body or spatial  +  body modes of interaction.

A third design goal was that the whole body environment should support users to both move and think without privileging one modality more than others. For example, input movements should not be too difficult or too trivial to enact. Similarly, changes in output images and sounds should be fairly easy to perceive and interpret (while moving). A fourth design goal was that the environment should support both imbalanced and balanced bodily states without causing physical harm to participants.

The Springboard Whole Body Interactive Environment

The Springboard whole body environment supports users to interactively explore images and sounds related to three social justice issues. The system is designed to be used by a single user. The active input space is a small raised platform (132  ×  71  ×  20 cm) made from a crib mattress spring, a board and black cloth (Fig. 2.2). Since standing in a balanced way is a normal state for most adults, we required an input space that upset this balance but not so much as to distract the user’s attention away from the image display space. When a user steps onto the platform, their centre of gravity immediately becomes slightly out of balance and they will likely wobble on the platform. The rectangular design of the platform also supports lateral movement. By moving left or right, the user can also be out of balance spatially and the design of the platform ensures that it is even more wobbly at the edges than in the centre. States of bodily balance are determined as the user moves their body’s centre of gravity and spatial position on the springboard input platform.

Fig. 2.2
figure 2_2

Springboard input platform made from a crib spring, a board and black cloth

Full size image

The spatial and body balance parameters for the input space are determined using a camera vision blob tracking and analysis system developed in the Max/Jitter programming environment. The user stands in front of a black background on the black Springboard platform. This setup allows a simple background subtraction process to be used to isolate the user’s image. The “balance” of the user can be calculated using a body balance index, which is based on the relation between the user’s centre of gravity and feet, or a spatial balance index, which is based on their spatial position on the Springboard platform, or the combination of both. For the combination, the two indices are combined using a scaled addition process. The total balance index ranges from −10 to +10 where 0 reflects a completely balanced state. The total balance index is used to control the image and sound display engines.

A description of the image display engine follows. Each of the food, shelter and safety issues has two factors that can be balanced or imbalanced to various degrees, as described above. The factors are depicted with pairs of images displayed on a large vertical screen as shown in Fig. 2.3. For each issue, a set of images depicting different aspects of that issue were sourced and then tagged through a collaborative sorting process in order to categorize them in five numbered bins. For each issue, the bins range from (1) too much of the factor, to (5) too little of the factor with a central bin (3) for balanced factors. For example, for the shelter issue, an image of opulent interior of a private residence was tagged 5 (too much quality/resource consumption) and an image of a person sleeping in a cardboard box was tagged 1 (too little quality/resource consumption), as shown in Fig. 2.3. Each of the bins contains many images to support variation and multiple interpretations.

Fig. 2.3
figure 3_2

Vertical display with pairs of images for issue: shelter (out of balance, bins 5 & 1)

Full size image

The display engine uses the current value of the total balance index (described above) in a random process controlled by a bell distribution curve to select pairs of images from the five bins. The center of the curve is determined by scaling the data from the total balance index (−10 to +10) to a floating point index from 1 to 5. The deviation of the bell curve is set so that when the control index is at x.0 the engine presents images only from that bin. For example, if the balance index is 3.5, the engine will display images from bin 3 and bin 4 with equal probability. To select pairs of images that represent the two factors or sides of twin-pan balance schema, the second image is selected using a mirror index derived by subtracting the balance index from the maximum for the index. For example, a balance index that would be mapped to bin 2 produces a mirrored index mapped to bin 4. For the food ­production issue, this might result in an image of mono-culture farms on one side, and chock full produce section at a large chain grocery store on the other side. Pairs of images synchronously fade in and out as the user’s centre of gravity over their feet (body) and/or spatial position on the spring board platform (spatial) change. The end result is that the user’s movements in and out of balance trigger metaphorically related changes in the images depicting the balance of two factors for each social justice issue.

The sound display engine for Springboard utilizes several approaches to representing the concept of balance through sonic aspects. Sound provides constant ambient information, responding to and guiding user actions. The obvious choice of a left-right channel panning was discarded. Panning is a representation of balance based on a cultural invention associated specifically with the technology of headphones. It is not based on a metaphorical elaboration of bodily or perceptually formed image schema. In addition, panning does not provide a clear resolution of change. We focus on more primary perceptually-based sound parameter changes such as pitch, timbre, and phase to achieve a sense of sonic balance and imbalance. For more details on sound design for Springboard, see [5].

User Study Methodology

An observational and interview style user study was conducted with 45 participants using Springboard as part of a larger study. The component of the study we report on here provides qualitative observations and verbal responses to post-task interview questions, which enable us to explore the impact on user experience of using image schematic input patterns to produce metaphorically related changes in image and sound content. We also compare the similarities and differences between using spatial and physical (body-centric) twin-pan balance schemas to structure the interaction modes.

Study Details

Twenty two (22) females and 23 males were recruited from the Simon Fraser University Surrey campus. All but three participants were aged 18–25 years old. All participants were daily computer users and 63% had been exposed to whole body interactive environments before. Participants were randomly assigned to one of the three interaction modes: spatial (n  =  15); body (n  =  14); and ­spatial  +  body (n  =  16).

The study took place in a self-contained research lab that contained the Springboard system. Participants completed a training task and three identical tasks, each with a different content set (food, shelter, safety). Session duration ranged from 30 to 40 min. The tasks involved having the participant use Springboard to use their body to interact with each multimedia content set. Users were told to explore the images and sounds for as long as they liked, and to indicate when the images and sounds represented “balance” for each issue. They were not told what the issues were or how the system worked.

Qualitative Data Collection and Analysis

During each session we took detailed observational notes using a semi-structured data collection sheet. After each session, we conducted an interview with each participant. We asked open questions about their general likes and dislikes (attitudes), how easy or difficult they found Springboard to use, and how they thought the system worked.

Participant’s verbal answers were recorded on audio and later transcribed. We used a bottom-up thematic analysis approach to identify themes that were related to the benefits or limitations of using an embodied interaction model. We also compared and contrasted responses between the three groups (spatial, body, spatial  +  body).

Results and Discussion

We present the results of our qualitative analysis, which address our research ­questions, through a discussion of the following themes.

General Attitude to Springboard

Overall, most of the participants were positive about their Springboard session. They found it “interesting”, “novel”, “cool”, “thought provoking” and found the whole body interaction “a welcome change from sitting at a desktop computer” or being in a classroom. However, about half the participants also had criticisms of the session and Springboard. Many felt that they wanted more details of how to interact and instructions for the task. The open exploratory nature of the task was problematic for some participants. For example, when asked what they disliked, one participant said, “Well not really dislike but I guess if you weren’t here I would have no idea what I’m supposed to be doing so it wasn’t really clear, instructions or anything.” Springboard is essentially a black room with the Springboard platform and a large display that offers participants few affordances or cues about how to interact with the system. Most participants quickly understood that they should position themselves on the Springboard facing the display. We designed the system intentionally to offer few clues to operation because we wanted investigate if participants would come to understand how their bodily and spatial states of balance impacted the depictions of balance and sonic balance based on metaphorical elaboration, either unconsciously or consciously. However, this raises the issue, also identified in [4], that physical objects (e.g. tangible user interfaces) can be designed or naturally provide affordances that encourage or constrain specific interactions such as image schema enactments. More unstructured whole body interactive environments (without controller artefacts) do not always have these affordances.

Enactment Versus Verbalization of Image Schemas

About a third of the participants were able to both enact and express verbally how the balance schema structured their interaction with Springboard. However, many other participants expressed that they “didn’t really understand how the system worked”. Despite this, we observed that most of these participants were able to use the system successfully to complete the tasks. Without being aware of it, participants enacted the spatial and body-centric balance image schemas as they interacted with Springboard. However, they were not able to verbalize a mental model of the system that included these balance schemas. This is consistent with image schema and conceptual metaphor theory since these mental structures are processed below the level of conscious awareness [8]. From this we suggest that if there is a specific user experience goal for whole body environment (e.g. explore balance in social justice) then the design must enable or afford users to enact image schemas and interpret metaphorically related system responses in ways that the designers intended. This can be done through the design of physical affordances and perceptual cues structured using one or more image schemas. For example, the spring board afforded moving in and out of balance, and the side-by-side visual layout mirrors the twin-pan balance structure. However, supporting users to enact or perceive image schematic structures does not necessarily require that users understand how the interaction model or system works. We suggest that it is important to consider these distinctions when designing whole body interaction based on embodied metaphor.

Image Schemas in Mental Models of Interaction

Participants provided verbal evidence that other image schemas and related metaphors occurred in their mental models of how they thought the system worked. For example, several participants initially thought that the system worked like an interactive video controller (metaphor: time is a linear path). “I was thinking that if I moved to the left the images would scroll backwards so if I shift left I could see previous images and if I moved my body to the right it would scroll forward but it wasn’t doing that.” Some participants thought that they should move towards images they wanted or liked. Similarly, one participant said that they should interact “by leaning towards the side that I think is social justice” (metaphor: force is ­emotional ­attraction). Some participants commented that they used force or pushed down to change the images (metaphor: force is change). Some participants thought that if they stepped to one side it would make images bigger. What is evident here is that participants used a variety of image schemas and metaphorical elaboration in their interpretation of the interaction model. However, they did not necessarily use the balance image schema! We suggest that cueing participants might be necessary to help them determine which schema they should use in cases where the physical characteristics of the system afford many input actions and interpretations.

Image Schemas in Interpretation

Many participants provided verbal evidence of using the schema of balance to structure their interpretation of Springboard content. For example, they used the word “balance” metaphorically in their verbal responses to our questions. They also used other words related to the twin-pan balance schema (e.g. “weight”, “midpoint”) in their responses. For example, one participant said “It was kind of like a teeter-totter, the more balanced you made it, I guess, the more the images represented a more balanced society.” Another participant said, “It lets people know that society has to balance more, distribute more balance on the different classes of people.... to be equal to everyone.” A third said, “Based on balance left and right you could weigh the options sort of in one direction or the other.” This again raises the issue of whether participants need to be able to explicitly understand how the system works (with respect to the balance schema) to use it and make sense of content. We suggest, consistent with metaphor theory, that users do not need to explicitly understand metaphorical relations to use them to interpret their perceptions.

Spatial Versus Body-Centric Balance Image Schemas

There were no large differences in participant’s responses between the three groups (spatial, body, spatial  +  body). There were two small differences that require further investigation. First, we found that more participants in the spatial group used words like “easy”, “intuitive”, and “simple” to describe what they liked about Springboard. Second, more participants in the spatial and spatial  +  body groups gave explanations of how Springboard worked that focused on the content rather than interaction. We interpret this cautiously as further evidence of the primacy of spatial schematic interpretations in whole body interaction, as seen in [2].

Conclusions

We present the theory, design, and results from an observational and interview style user study in which we investigate the benefits and limitations of using image schemas and embodied metaphors in an interactive environment that uses whole body interaction and depicts images and sounds from the abstract domain of social justice. Our findings suggest that using image schemas and related metaphors to structure interaction with multimedia content in an abstract domain like social justice is a promising approach. However, there are limitations. In particular, participants are often unaware of using schemas to structure interaction and interpretation, and use a variety of schemas and metaphors in their sense making. More work is needed to continue to explore the benefits and limitations of using embodied interaction models for whole body and tangible interaction in abstract application domains.