Playful Ambient Augmented Reality Systems to Improve People’s Well-Being

Abstract

Intense stress, discomfort or sedentary behaviour are factors that have a negative impact on people’s well-being and efficiency. A number of prototypes have been developed to help people address these issues. In parallel, gamification techniques have been successfully used in applications that aim to improve people’s health by encouraging them to change their behaviour. In this chapter, we focus on the use of Ambient systems and/or Augmented Reality systems and discuss the potential of playful and gamified systems to improve people’s well-being. We first describe several prototypes that aim to improve people’s well-being. Then, we discuss the potential of gamification and projection-based AR systems in the context of well-being, and introduce the notion of playful projection-based ambient systems. We conclude by describing a scenario and discussing technical considerations for its implementation.

1 Introduction

Intense stress, discomfort or sedentary behaviour are factors that have a negative impact on people’s well-being and efficiency. Therefore, it is not surprising that researchers from various fields (e.g. medicine, psychology, sociology, ICT, etc.) have been looking for innovative solutions to help people address these issues and increase their well-being.

In relation to ICT and well-being, several research topics have emerged in the past 30 years, including affective computing (Tao and Tan 2005), physiological computing (Fairclough 2009) and even Human-Building Interaction (Alavi et al. 2016). Within the Human-Computer Interaction community, a number of interactive applications and physical artefacts have been proposed and developed. These tools leverage on the benefits of new technologies to provide people with the possibility to positively impact, adapt to and cope with their environments. In particular, working environments have been largely adopted as an interesting “playground” for the deployment of such prototypes. In parallel, gamification techniques have been successfully used in applications that aim to improve people’s health by changing their behaviour patterns or mitigating stressful situations through playful experiences.

Although the spectrum of tools designed is very large, the underlying approach is often similar and can be presented in a simplified way as follows: (1) collect data related to the user’s mental and physical states (e.g. physiological measures such as heart rate, respiration or skin conductance response) and/or to the user’s environment (e.g. temperature or air quality); (2) process data in order to inform or alert the users and/or promote behavioural changes—be it by supporting the acquisition of self-regulating techniques (e.g. meditation), suggesting actions (e.g. opening the window if the air quality is poor), or even to directly alter the environment to impact users’ comfort.

In this chapter, we will focus on the use of Ambient systems and/or Augmented Reality systems that have been designed to provide users with data related to their mental and physical state as well as to their environment, and we will discuss and illustrate the potential of gamification for such applications. Given the prevalence of systems intended to be deployed in the workplace, and in particular for white-collar workers, we mainly focus on built and working environments. We utilise the first section to delineate a number of factors that impact people’s well-being, and briefly present different ways of assessing it. In the second section, we describe Ambient systems as well as projection-based Augmented-Reality systems. In the third section, we discuss the potential of gamification in the context of well-being. In the fourth and last section, we propose the design of Ambient Augmented Reality systems to improve people’s well-being, focusing on the idea of peripheral interaction, calm technologies and gaming.

2 Well-Being

2.1 What Is Well-Being?

Definitions of well-being in the literature are not consistent, and the term is often used interchangeably with terms such as comfort and health (Pinto et al. 2017). Danna and Griffin (1999) view well-being as a broader concept that includes non-work satisfactions, job-related satisfactions and general health. Health is thus considered as a sub-component of well-being and is associated with physiological and psychological factors. Nevertheless, both are affected by three types of interrelated antecedents (the work setting, the personality traits and the occupational stress) and both can affect individuals and/or organizations (Cooper and Dewe 2008).

In this chapter, we will use the term “well-being” in a broad sense and we will consider that well-being can be affected by, among others, stress, health and comfort. We will first address occupational stress, followed by other health-related issues conditioned by the work setting, and finish with the concept of comfort. The notion of comfort constitutes a well-documented and delineated topic of research that is being studied within a variety of fields, including Architecture, Human-Building Interaction and Built Environment research.

2.1.1 Stress

Stress is a physiological and psychological response that happens when the environmental demand exceeds one person’s adaptive capabilities. Stress is triggered by a “stressor” (an environmental stimulus or situation) and results in behavioural consequences that will eventually help to recover homeostasis. The stress responses involve the Autonomous Nervous Systems or ANS (e.g. through increased blood pressure, heart rate and breathing, sweating and muscle stiffening) as well as the Hypothalamus-pituitary-adrenocortical axis or HPAa, that is responsible for releasing hormones (and notably cortisol, that will redistribute energy). The different types of stress result in different patterns of activation of the ANS and the HPAa system (Jeunet et al. 2014). These types of stress include: physical stress, psychological stress (e.g. when a task is cognitively demanding) and psychosocial stress (triggered by a social-evaluative threat that “occurs when an important aspect of the self-identity is or could be negatively judged by others” Dickerson and Kemeny 2004). The detrimental effects of repeated stress on health are well-documented, and include depression, hypertension and various forms of cardiovascular diseases (McEwen and Seeman 2003).

2.1.2 Other Health-Related Factors

Nowadays, with the prevalence of computer stations at work, people spend most of their time sitting, and more than two thirds of office workers spend six hours or more sitting at their desk.¹ This has led researchers to investigate the effects of sedentary behaviour on people’s health and, consequently, to propose solutions to encourage office workers to be more active. Sedentary behaviour is defined as any waking behaviour characterized by sitting or reclining while expending little energy (Mansoubi et al. 2015). Sedentary behaviour is known to be associated with obesity, type 2 diabetes, cardiovascular disease and premature mortality. Another aspect of modern workplaces that has been addressed in the literature is the effect of incorrect sitting postures and extensive computer usage on people’s health, which can lead to the development of musculoskeletal symptoms, e.g. for the shoulders, necks, lower back and upper limb regions (Lapointe et al. 2009). Other factors that have been addressed include chronic water deficiency (e.g. Chiu et al. 2009) as well as the computer vision syndrome (Blehm et al. 2005). This term refers to various problems such as eyestrain, tired eyes, irritation, burning sensation, redness, blurred vision, and double vision. It affects 70% of computer users and can result in headache and fatigue (e.g. Dementyev and Holz 2017).

2.1.3 Comfort

Although comfort includes several aspects, such as physical comfort or contact comfort, we will here focus on the notion of comfort as defined in research related to built environments. In that context, comfort is an essential aspect of indoor environment quality that includes four components: thermal comfort, visual comfort, acoustic comfort and respiratory comfort.

Thermal comfort is related to four physical variables (air temperature, radiant temperature, humidity and air velocity) as well as two personal parameters: the clothes worn by the people and their activity level (Fanger 1984).

Visual comfort includes the illuminance, luminance, colour of light and glare, amount of daylight, etc. Visual discomfort can be provoked by non-uniform illuminance, and by discomfort glare, which is “a sensation of annoyance caused by high or non-uniform distributions of brightness in the field of view” (Kaufman et al. 1981).

Acoustic comfort refers to the intensity and frequency of noise. Three issues can lead to acoustic discomfort: (i) too much noise entering the person’s space outside the building; (ii) too much noise from adjacent spaces; (iii) lack of sound control in the space itself (Paradis 2016).

Respiratory comfort relates to the air quality of the environment. More precisely, the indoor air quality can be defined as “the physical, chemical and biological properties that indoor air must have in order: not to cause or aggravate illnesses in the building occupants; to secure high level of comfort to the building occupants in the performance of the designated activities for which the building has been intended and designed” (Bluyssen et al. 2003). Respiratory comfort also relates to “the absence of discomfort due to odour and sensory irritation” (Frontczak and Wargocki 2011).

2.2 Measuring Well-Being

People’s mental and physical states can be assessed using a variety of methods, including:

• biochemical sensors that “require collection, analysis and disposal of body fluids” (Patel et al. 2012) (e.g. to measure salivary free cortisol quantity);

• physiological sensors (e.g. to obtain electrocardiograms or respiratory signals, see (YU et al. 2018a, b) for a review);

• neurophysiological sensors (e.g. electroencephalogram);

• indirect sensing (e.g. see Tao and Tan 2005 for a review of techniques to identify and convey emotions, and Hernandez et al. 2014 for a review of the use of computer mouse and keyboard to assess one’s stress);

• questionnaires such as the State-Trait Anxiety Inventory (Spielberger et al. 2017), the Self-Assessment Manikin (Bradley and Lang 1994), the Daily Stress Inventory (Brantley et al. 1987) or the NASA-TLX (Hart and Staveland 1988).

Sedentary time and sitting posture can both be detected by sensors embedded into the worker’s chair (e.g. Tan et al. 2001), wearables (e.g. Dunne et al. 2007), cameras (e.g. Pinto et al. 2017) or commercial products (e.g. PostureMinder² or VISOMATE-USB³).

Occupants’ comfort within the built environments is usually determined using ambient sensors (e.g. to measure temperature, humidity, light intensity, concentration of CO2), cameras or microphones. In addition, physiological measures can be used (e.g. De Dear et al. 2013; Zhang et al. 2010; Liu et al. 2008), as well as questionnaires, such as the ASHRAE Standard 55 (of Heating and Engineers 2010) and the Post-Occupancy Evaluation questionnaire (Riley et al. 2010). Dedicated tools can also be used (e.g. Comfort Box by Alavi et al. 2017).

3 Ambient and Augmented-Reality Systems to Improve Well-Being

3.1 Ambient Systems

Ambient systems rely on lights, projections, sounds or objects moving at the periphery of the user. Pousman et al. (2006) discussed the various definitions of “ambient displays” that have been used in the literature, and proposed the term “ambient information system” to refer to systems that present the five following characteristics: (1) they display information that is important but not critical; (2) they can move from the periphery to the focus of attention and back again; (3) they focus on the tangible representations in the environment; (4) they provide subtle changes to reflect updates in information (they should not be distracting); (5) they are aesthetically pleasing and environmentally appropriate. The first two characteristics are closely related to the notion of “glanceability” discussed by Rogers et al. (2010): “just as we momentarily look at a clock on the wall, [ambient displays] too are intended to be looked at occasionally and peripherally without distracting us from our ongoing activities, or compromising the objects or spaces in which they are embedded”. The second characteristic is also related to the concept of attentive interfaces (Vertegaal 2002). In that sense, ambient displays can support the unobtrusive display of information, while subtly promoting behavioural changes. In this section, we describe several ambient displays that aimed at improving people’s well-being. Instead of providing an exhaustive list of ambient prototypes, we chose to focus on illustrative examples.

3.1.1 Stress and Arousal

Several ambient lighting systems have been designed to display biosensor data such as the levels of arousal or stress (Roseway et al. 2015; Snyder et al. 2015; Yu et al. 2018a, b). For example, MoodLight (Snyder et al. 2015) consists of a lamp that can produce up to ten different hues: the higher the level of arousal, the warmer the colour of the light. This system was designed to enable “students to develop awareness of communication practices related to self-revelation”, but also to help therapists introducing stress management techniques to students. Another example is BioCrystal (Roseway et al. 2015), a small lamp that can diffuse five colours in order to reflect the user’s affective state along two axes: valence (positive or negative) and arousal (calm or energetic). A two-week user study revealed that “the Crystal helps to control stress, and can be successfully used to increase users’ awareness of their affective state, which could lead to behavioural change”.

The DeLight system (Yu et al. 2018a, b) is similar, but in addition to providing users with feedback about their level of stress, it also offers tools for relaxation assistance. Unlike previous prototypes, DeLight was composed of one central light and several ambient lights installed on the ceiling or projected on the wall (see Fig. 6.1). The authors investigated the design of warm-toned versus cold-toned colour mappings. Results showed that both conditions enable users to control their response to stress, although participants “were more likely to associate a warm-toned light with their increased stress when it turned to orange”.

Fig. 6.1

Illustration courtesy of the authors (Yu et al. 2018a, b)

The DeLight system provides biofeedback through ambient light for stress intervention. Light colors change according to the user’s level of stress. a Low-stress; b high-stress (warm-toned mapping); c high-stress (cool-tone mapping).

Dišimo (Mladenović et al. 2018b) is a physical artefact that relies on multimodal feedback to encourage users to breathe more regularly and deeply. When the system detects a decrease in heart rate variability, it emits a sound to subtlety alert the user, who can decide to take the device in his/her hands (see Fig. 6.2). Audio sounds are played as a breathing guide and physical particles flutter inside the device when heart rate variability increases. Several Dišimo can be used simultaneously, for collaborative relaxation sessions. In that case, light colour and brightness provide information about the other participants’ activity.

Fig. 6.2

Illustration courtesy of Jérémy Frey (Mladenović et al. 2018b)

Dišimo provides multimodal biofeedback about heart rate variability in order to encourage and help users focusing on their breath.

3.1.2 Sedentary Time

Other prototypes aimed at encouraging people to be more active, such as HealthBar (Mateevitsi et al. 2014), an ambient display with the shape of a light-tube that changes its colour depending on the user’s physical activity (see Fig. 6.3). Similarly, in Fortmann et al. (2013), the authors investigated the use of a peripheral lamp to inform users about their physical activity, through variations in light. Similar to the HealthBar system, their design was based on the metaphor of a battery: the light colour progressively changes from green (fully charged) to red (discharged); when the user is active, the battery charges again and progressively switch from red to green. Results showed that participants increase their number of steps when using MoveLamp.

Fig. 6.3

Illustration courtesy of Lance Long, Electronic Visualization Laboratory

The Health Bar prototype encourages users to take a break to reduce sedentary behavior (Mateevitsi et al. 2014).

BreakAway (Jafarinaimi et al. 2005) is an actuated sculpture that relies on “body-language” notions to convey information about the time people have spent sitting. The sculpture reflects a body’s shape and can look as if it were standing upright or slouching (see Fig. 6.4). Therefore, it is meant to encourage users to move before the sculpture reaches the slouching position. An initial evaluation with one office worker revealed that the time at which the worker took breaks was correlated with the sculpture’s positions.

Fig. 6.4

Illustration courtesy of the authors (Jafarinaimi et al. 2005)

The different postures of the BreakAway system, an ambient display that encourages people to take breaks more frequently.

3.1.3 Sitting Postures

In Haller et al. (2011), the authors investigated different ways to interrupt people when their sitting posture is inadequate in order to provide them with training sessions. One of the interfaces consisted of a physical and ambient plastic plant with leaves and petals that could be automatically changed to reflect the user’s posture. A user study showed that participants found this ambient display less disruptive than vibrations. A similar prototype was designed by Hong et al. (2015) to improve people’s posture at work.

3.1.4 Active Behaviour

Rogers et al. (2010) investigated whether an ambient display could influence people’s behaviour by encouraging them to use the stairs instead of the elevator in their place of work. They designed three different ambient installations. Follow-the-Lights consist of a set of LEDs embedded into the carpet and that subtly guide people towards the stairwell instead of the elevator. The Cloud is a large display composed of two sets of coloured spheres hanging from the ceiling: the higher the number of people taking the stairs, the higher the cloud formed by the grey spheres. The History is a large screen showing pie charts that represent the ratio of people stair/elevator usage for each day. A 8-week study showed that despite the fact that people were not aware of changing their behaviour, there was a significant change in the use of the stairs. The authors suggest that this could be due to the fact that the installations raised people’s awareness of their decisions.

3.1.5 Hydration

Ambient displays have also been designed in order to encourage workers to drink more frequently. Playful Bottle (Chiu et al. 2009) consists in an ambient smartphone application that detects the amount and regularity of water consumed by the users and that use these pieces of information as part of two games. The first game simply displays a tree that changes its aspect depending on the amount and regularity of the water consumed: if the user does not drink enough water, the tree’s leaves change their colour from green to yellow and fall down. The second game is a multi-player game that connects together several Playful Bottle: all players’ trees are visible and players can compete as well as send “hydration reminders” to others. However, in order to be able to send these reminders, they must first earn some credits by drinking. A 7-week user study showed that both games led to a higher amount of water consumed and that the multi-player game lead to better results in terms of amount and regularity. A similar metaphor was adopted for the design of the Mug-Tree prototype (Ko et al. 2007). More recently, Kaner et al. (2018) also used a tree metaphor: their GROW prototype consists in a bottle of water that embeds a thermo-chronic tree printing on its surface (see Fig. 6.5). Its aspect changes depending on the level of water in the bottle: the image is fully displayed only when the user drinks a sufficient amount of water.

Fig. 6.5

Illustration courtesy of the authors (Kaner et al. 2018)

GROW is a smart bottle that uses its surface as an ambient display to indicate the amount of water intake. The entire tree is shown when the user has drunk enough water.

3.1.6 Comfort

Although all the above-mentioned prototypes play a role in improving people’s overall comfort, there is little work concerning the design of ambient displays to improve people’s comfort at work, as defined in the context of the built environment research (i.e. thermal, visual, acoustic and respiratory comfort). One interesting example is a commercialized display developed by Jabra and called NoiseGuide.⁴ It consists of a peripheral display that can be placed on office workers’ desks and that indicates whenever the noise level is above a specified limit, through different colours (green, orange, red). Another commercial product is CubeSensors,⁵ which consists in a set of small smart cubes that can be placed inside people’s home or office. When they are shaken by the users, their colours change to indicate the air quality.

Moreover, it is worth mentioning that a number of ambient displays have been developed to encourage people reducing their energy consumption; in particular, some of these displays aimed at improving existing Heating, Ventilating and Air Conditioning systems, which could in turn directly affect people’s thermal and respiratory comfort.

3.2 Projection-Based AR Systems

Although ambient systems often rely on dedicated artefacts or additional displays, it is also possible to design ambient systems using Augmented Reality. Augmented Reality systems combine both physical elements and digital elements, by enhancing the former with the latter simultaneously (Van Krevelen and Poelman 2010). Systems are defined as “AR systems” if they fulfil the three following criteria (Azuma 1997): (1) they must combine real and virtual objects; (2) they must be interactive in real time and (3) they must be registered in 3D. Although current AR systems mainly rely on the visual modality, auditory, olfactory and tactile stimuli can also be used as a way to augment the real world.

Nowadays most often used AR systems are handheld systems such as mobile phone cameras that show the real world together with digital objects on the screen of the mobile phone. Another alternative is head mounted displays, which are commonly divided into video see-through and optical see-through displays. This division is based on the method used to achieve display transparency. Video see-through displays use a video camera and an opaque screen, whereas optical see-through displays are transparent. Examples of such displays are HoloLens⁶ and MagicLeap.⁷ Irrespective of recent advances in display technology, current see-through displays are not appropriate for the design of ambient systems, either because they require the users’ full attention to hold the device in hand or because they require the users to wear a bulky headset or glasses. In addition, such systems can only augment a small field of view of the human eye viewing frustum, which makes it impossible to create ambient displays that sit in the periphery of the user.

Visual projections are another display technology used in AR systems and are commonly referred to as Spatial Augmented Reality or Projection-based AR systems. These systems present several advantages: “an improved ergonomics, a theoretically unlimited field of view, a scalable resolution, and an easier eye accommodation (because the virtual objects are typically rendered near their real world location)” (Bimber and Raskar 2005). Unlike see-through displays, projection-based systems can be used to enhance the user’s environment in a very unobtrusive way and are therefore particularly adapted for the design of ambient systems. For example, the Dynamic Projection Institute company sells the Mirror Head,⁸ a digitally-controlled mirror that can be attached to a projector in order to project images and textures into various elements of a room, such as curtains, floor, wall and ceiling.

In the context of well-being, two illustrative examples of projection-based systems are TOBE and Inner Garden. TOBE (Tangible Out-Of-Body Experience) (Gervais et al. 2016) is a tangible avatar onto which visual animations that reflect the user’ inner state can be projected, using Spatial Augmented Reality (see Fig. 6.6). The physiological data being captured include: breathing, heart rate, perspiration, eye blinks and brain activity. Interestingly, the users are free to decide in which way they want their physiological data to be visualized on TOBE. An application enables them to associate one metric (e.g. arousal level, workload, vigilance) with one support (e.g. tangible avatar or sound) and one shape (e.g. a heart that changes its colour depending on the arousal level), in order to design tailored feedback. TOBE can be used as a biofeedback device and was also successfully used as a relaxation device for multiple users who had to synchronize their heart and reach cardiac coherence.

Fig. 6.6

TOBE (Gervais et al. 2016) is a tangible avatar onto which visual animations that reflect the user’s inner state can be projected, using Spatial Augmented Reality. An interface enables users to customize the visualizations. Illustration courtesy of Potioc project-team (Inria, Universite Bordeaux, CNRS)

A similar approach was explored by the same group of researchers and resulted in the design of Inner Garden (Roo et al. 2016), a tool to support mindfulness practises. This prototype is an augmented sand-box: users can first create a landscape by shaping the sand; then, various elements such as water, trees, clouds, etc. are projected onto this environment according to different measures of the user’s mental state (see Fig. 6.7). For example, the breathing patterns are directly mapped to the water level of the mini-world. This mini-world continuously evolves based on the measured data. The users can also decide to immerse in this world using a VR helmet. (Although they are out of the scope of this chapter, it is also worth mentioning that a few VR systems have been developed to improve people’s awareness of their mental states using biofeedback techniques and gaming approaches, such as Gaggioli et al. 2014.).

Fig. 6.7

Illustration courtesy of Potioc project-team (Inria, Universite Bordeaux, CNRS)

The Inner Garden system is an augmented sandbox, the appearance of which changes according to the user’s physiological signals. Inner Garden supports mindfulness practices (Roo et al. 2016).

In addition to these prototypes related to stress management, a few prototypes have been developed that enable users to interact with their built environment using see-through displays in order to, for example, collect information about the temperature of the room (see Rauhala et al. 2006) for example and Wang et al. (2012) for a review of potential applications of AR within built environment).

4 Gamification and Playfulness for Ambient and AR Systems

4.1 Ambient and AR Displays as Persuasive Technologies

In the previous section, we described several prototypes that have been designed to improve people’s well-being, especially by trying to change their behaviour in the long term. As such, these systems can be defined as “persuasive technologies”. A persuasive computing technology is “a computing system, device, or application intentionally designed to change a person’s attitudes or behavior in a predetermined way” (Fogg 1999). Halko and Kientz (2010) identified different strategies that have been used in the field of persuasive technology: giving instructions through an agent (authoritative or non-authoritative); using social feedback (cooperative or competitive); enhancing motivation (intrinsic or extrinsic); using reinforcement techniques (positive or negative). Some of these strategies were used by the prototypes that we described, and are also core concepts of gamification. Gamification relies on the use of gamified elements (such as competition between participants) to reinforce behaviour, especially by enhancing the users’ motivation, and it has been widely employed to promote behavioural change. In this section, we first define what is gamification, before discussing its potential to improve people’s well-being and identified the “playful” and “gamified” elements of the persuasive systems that we described.

4.2 What Is Gamification?

According to Deterding et al. (2011), gamification is « the use of game design elements in non-game contexts”. Gamification can improve user experience and user engagement but also positively influence users’ behaviors and motivation. Gamification is different from playfulness, as it involves explicit rules and competition. It has been used in a number of contexts, such as commerce, education, health and work (Seaborn and Fels 2015).

Gamification builds on two main mechanisms: reinforcement and emotions (Robson et al. 2015). When the users perform the desired outcome, they are rewarded and experience positive feelings, which encourage them to perform the desired outcome another time, until the desired behavior becomes more habitual or even automatic. Motivation is also a crucial aspect to ensure that the desired behavior will continue over time. In fact, Ryan and Deci (2000) identified two types of motivation. Intrinsic motivation refers to activities that are being performed for their inherent qualities, e.g. because they are pleasant or necessary (Ryan and Deci 2000). On the other hand, activities that are performed with extrinsic motivation are performed not for themselves, but for another outcome, such as earning points in a game (Ryan and Deci 2000). One interesting characteristic of gamification is that it can positively affect intrinsic motivation if the participants perceived the game elements (e.g. points, levels, leaderboards) as informational. However, when perceived as controlling, these incentives may also impair intrinsic motivation (Mekler et al. 2013).

In order to trigger positive emotions and reinforcement, gamified applications and tools use a variety of building blocks, that can be described as Mechanics, Dynamics and Aesthetics, according to the MDA framework (Hunicke et al. 2004). Mechanics are “the various actions, behaviors and control mechanisms afforded to the player within a game context” (Hunicke et al. 2004). Dynamics are the player’s interaction with the core elements of the game over time (i.e. with the mechanics). For example, a mechanic of badges can trigger competitive dynamics. Dynamics lead to particular emotional responses that are called aesthetics.

In Robson et al. (2015), the authors identified three types of mechanics: setup mechanics (e.g. the setting, the objects used, the mode of playing, such as competition or cooperation), rule mechanics (e.g. the set of actions that the player can choose and the constraints of the game), and progression mechanics, which affect the experience over time (e.g. achievement rewards). More generally, game mechanics can be very diverse and include the use of avatar, rewards, points, badges, level, challenges and time constraints (Zichermann and Cunningham 2011). The use of feedback and reinforcing game elements is also crucial for the overall user experience. In addition, social game mechanics such as leaderboards or the use of “bragging” notifications on social media can also be used as game mechanics to further engage the players and trigger reinforcement.

4.3 Gamification and Well-Being

Gamification has been particularly investigated in the context of health and well-being (Klasnja and Pratt 2012; Lister et al. 2014; Sardi et al. 2017). Johnson and al. identified several advantages of gamified applications and tools in such a context: they can positively influence intrinsic motivation, they can be broadly accessible and appealing to a large audience, they can be efficient in terms of cost, they fit into people’s everyday lives (instead of games that require people to dedicate some time and space to play them) and they can inherently support well-being by triggering positive emotions (Johnson et al. 2016).

One well-known example is the prototype developed by Hall et al. (2013) to encourage participants to report on their well-being more frequently. The tool was implemented as a gamified Facebook app that includes various incentives (scores, points, stars, badges and social incentives), which enable participants to send points, brag or invite friends. The use of gamification elements proved successful to encourage people to answer the required questions and compute their Human Flourishing Scores. UbiFit (Consolvo et al. 2008) is another illustrative example of a mobile application to increase physical activity. It uses the wallpaper of the user’s phone as a glanceable display, the appearance of which changes depending on whether the participants reached their goal or not.

Johnson et al. (2016) reviewed 19 studies related to well-being and gamification (most of which involved mobile applications and websites) and found out that positive effects of gamified interventions were reported in 59% of the studies, as well as neutral or mixed effects for 41% of the studies. Example of issues addressed included health monitoring, stress management and increasing physical activity. Rewards (including points and badges) were the most commonly used game element and were often used in combination with other game elements such as avatars, challenges, feedback, leaderboards, levels, progress, social interventions and stories/themes. The use of rewards was found to be efficient “to drive health behavior” such as an increase in physical activity; tools that use avatars were associated with positive outcomes; social interventions led to a better user experience.

Klasnja and Pratt reported similar positive results in their review of mobile apps for healthcare (Klasnja and Pratt 2012). They discussed the fact that several studies found that the use of entertaining mobile apps helped people to “engage […] with their health goals” and keep them “interested in the intervention”, notably through the introduction of fun content into reminders and informational messages. Overall, the authors highlighted the need to further explore “strategies for motivating health behaviour and education through games and other forms of entertainment” (Klasnja and Pratt 2012).

4.4 Playful and Gamified Ambient and AR Systems

The Playful Bottle prototype (Chiu et al. 2009) described in a previous section is one of the rare ambient and AR systems that propose to improve people’s well-being using gamification. In fact, the authors proposed in a previous publication a “play-based behavior modification model” that encompasses theories of play and playfulness as well as acquisition theories that relate to the reinforcement of behavior (Lo et al. 2007). This model illustrates how people can adopt a new behavior in the long term by being actively engaged into playful activities that rely on the idea of “partial reinforcement”, i.e. positive reinforcement that does not occur systematically. The Playful Bottle prototype includes several game mechanics: progressive levels (the tree has five distinct appearances); use of avatars; social persuasion through competition within players; use of credits and rewards (a loving heart and animations).

Other prototypes described in a previous section did not integrate game elements into their design; however, most of them were designed with the aim of providing a playful experience to the users. For example, the design of Inner Garden (Roo et al. 2016) was inspired by “the playfulness and experimental nature of sandboxes”. The three ambient displays designed to encourage people to use the stairs instead of the elevator were “designed to inform in subtle and playful ways” and the observations showed that the installations elicited curiosity and playfulness (Rogers et al. 2010). The HealthBar system (Mateevitsi et al. 2014) was inspired by the “life bar” that is used in video games to indicate the player’s status. Although it did not incorporate a specific game, MoodLight (Snyder et al. 2015) was first presented to users using a game-like approach, during which participants were asked to “playfully engage the system with their partner”. Dišimo (Mladenovic et al. 2018a) encourages cooperation between participants through different light colors (a mix of each user’s color) and light brightness (the higher the number of participants who increase Heart Rate Variability, the brighter the light).

Interestingly, several prototypes included an anthropomorphic component in their design, such as flowers with leaves and stem that respectively represent the arms and the body of the user (Haller et al. 2011), tangible avatars (e.g. Gervais et al. 2016) or sculptures that can adopt a standing position or a slouching position (Jafarinaimi et al. 2005). Such an approach might positively affect the overall playfulness of the prototype and the users’ willingness to engage with the systems.

5 Towards the Design of Playful Ambient AR Systems to Improve Well-Being

In the previous section, we described several ambient displays that aimed at improving people’s well-being, especially in their working environment. Although very little research has been conducted on the design of AR systems to improve people’s well-being, the two projects that we described (TOBE and Inner Garden) open interesting avenues. Among these two, Inner Garden is particularly appealing as it also acts as an ambient display. Building upon this work, we believe that using AR to design ambient displays is a promising approach to improve people’s well-being in a non-obtrusive but yet efficient way. In addition, given the potential of gamification for well-being that we described in the previous section, we also believe that the integration of gamification into Ambient AR systems would be worth investigating further.

In this section, we first discuss why AR is an interesting medium to design ambient displays. Then, we discuss several design opportunities for Ambient AR systems, including the use of playful and gamified elements. Finally, we illustrate how these design considerations could be instantiated in a concrete way through the depiction of a playful Ambient AR office.

5.1 Ambient AR Systems

AR technology offers two key properties that could be particularly relevant for the design of ambient displays. The first one relates to the fact that AR systems support spatially-distributed information and interaction, while the second relates to the multimodality of AR systems.

5.1.1 Projection-Based AR Support for Spatially-Distributed Information and Interaction

A common approach for projection-based AR systems is to project images, textures or videos directly onto the users’ environment. This offers several advantages. Firstly, as pointed out by Bimber and Raskar (2005), this makes it possible to display large field-of-view images instead of small-size pictures displayed on monitors, tablets or smartphones for example. At a larger scale, these images can be projected onto room’s walls, doors or desktop, further enlarging the size and type of surfaces that can be used and, in turn, the amount of information that can be conveyed. Secondly, this technology makes it possible to display images at various positions, instead of using a single-location display such as a screen or a physical artefact.⁹ Information can therefore be displayed far away or close to the users, depending on the context. This can support the design of interaction at the periphery or focus of attention of the user–a key characteristic of ambient displays. Thirdly, visualizations can be projected on real-world objects, which can enhance the sense of immersion of the user and make information more meaningful or easier to interpret, if it is closely related to the object it is displayed on. We can therefore tap into the spatially-distributed characteristic of AR systems to design peripheral interaction, calm technologies and embedded representations–the three of which we discuss in the next section.

5.1.2 AR Support for Multimodality

Another interesting aspect of AR systems is that they can provide multimodal interaction. In fact, AR encompasses a variety of sensory modalities, including visual, auditory, haptic, somatosensory, and olfactory. Although those last three modalities have not seen many implementations, the multimodality of AR systems opens new avenues for the design of ambient applications, especially by taking into account users’ perceptual abilities. For example, tactile and visual acuities are different, and using one modality over the other for conveying a particular piece of information could minimise the user’s workflow disruption by requiring less or more attention. Multimodality is also important to make the virtual elements more integrated into the real world and to increase the sense of immersion of the user, using a combination of lights, spatial sound and projections. Finally, multimodality can be an efficient way to increase the sense of “playfulness” of a system, for example by combining sounds and animations to design an ambient game. Nevertheless, it is important to note that current technology is limited in how well the sense of touch, somatosensory, and olfactory can be augmented, particularly if no modification to the environment is made.

5.1.3 Ambient Projection-Based AR Systems: Proposed Definition

The term ‘ambient AR applications’ was first used in Billinghurst et al. (2009), in order to define applications that “use AR technology to represent context information from an Ambient Interface”.

According to the definitions of ambient displays and of AR systems that we discussed previously, and also building on the two properties of AR systems discussed, we propose to define Ambient projection-based AR systems as a subset of ambient information systems (in the sense of Pousman and Stasko 2006) that present the following additional characteristics:

• i. they augment existing environment with digital content, focusing on tangible elements;

• ii. they use digital content that is registered in 3D and can be moved around from one object/surface to another;

• iii. they can be interactive in real-time.

We use the term “digital content” to refer to multimodal media such as pictures, videos, sounds, lights or even olfactory elements. This definition also highlights the fact that unlike most ambient displays, AR ambient displays do not require dedicated artefacts and are not limited to a dedicated space within the user’s environment.

5.2 Design Opportunities for Ambient Projection-Based Augmented Reality Systems

5.2.1 “Mobile” Interaction: From Periphery to Centre and Away

Both the spatial flexibility and multimodality of projection-based AR systems provide interesting means to design “mobile” perceptions and interactions. This term can first be understood from a spatial and cognitive perspective. In the first case, the information or means of interaction can be “physically” located at different positions and distances to the user, depending on the context or the relevance of the information. In the second case, the information or interaction can happen at the periphery or centre of attention, using different locations (close vs. far away) or different modalities (e.g. audio vs visual, light vs. dark colours, feeble vs. loud sounds, etc.). This is the core idea of peripheral displays and of peripheral interaction, which both build upon theories of divided attention that concern the use of multiple stimuli and performing of various tasks at the same time.

From a more abstract perspective, ambient AR systems could also support (sub)conscious “moves” between involuntary state and voluntary state, as described in the user’s model of comfort proposed in Alavi et al. (2017). According to this model, comfort is continuously evaluated in an involuntary way by the occupant’s sensory system (e.g. photoreceptors, haptic memory, etc.) and autonomic actions take place to maintain homeostasis, such as shivering or sweating. When the outcome of these actions is not sufficient enough or when the occupants are asked to assess their comfort, the occupants become aware of their state of (dis)comfort and evaluate it voluntary. Then, they can adapt their behaviour in order to recover their sense of comfort, e.g. by opening the window (i.e. behavioural response instead of automatic response).

This echoes the idea of “scalable notifications” (Matthies et al. 2017), which “can on the one hand, provide very subtle feedback to the user while, on the other hand, can intervene and force the user to take action”. For example, Matthies et al. (2017) designed a haptuator device to help people adopt a correct posture while sitting. The device notifies the user through mecano-pressure ranging from subtle feedback to forcing feedback (which triggers a natural reflex that straightens the back). Other modalities were also successfully investigated such as thermal feedback and electrical feedback. This research opens new avenues for the design of scalable and multimodal notifications to promote behavioural or autonomic responses.

Another interesting property of projection-based AR systems is that they can easily be embedded within the existing user’s environment and that they do not require a lot of additional hardware or physical modification of the environment. Therefore, in situations where the user does not want or need to use the system, it can simply disappear, visually and cognitively. Unlike projection-based AR systems, ambient displays that use physical artefacts cannot take advantage of this kind of “spatio-temporal” mobility.

5.2.2 Calm Technologies

The concept of “calm technology”, first described by Weiser and Brown (1996), integrates two important notions: they inform and encalm the user. “Calm technology engages both the centre and the periphery of our attention, and in fact moves back and forth between the two” (Weiser and Brown 1996). In that sense, they are closely related to the idea of peripheral interaction. However, by enabling the user to take control of the technology by recentering it, the “periphery is a fundamental enabler of calm through increased awareness and power”. By supporting peripheral interaction, ambient AR systems could then also support calming interaction, and affect people’s level of stress and overall well-being.

The idea of “calm technologies” can also be interpreted through the prism of restorative theories, and in particular attention restoration theory. The core idea behind these theories is that (views of) natural environment can help individual recover from stress or restore their attention. For example, Raanaas et al. (2011) showed that the presence of plants could positively impact the performance of participants in terms of attention. View of nature in the office environment can also decrease job stress. Brown et al. (2013) showed that viewing pictures of natural scenes could improve the recovery process following a stressor. Although the benefits of digital natures scenes as compared to real natural scenes cannot be systematically observed (see Kahn et al. 2008 for example), ambient AR systems could be used to project large natural scenes into the user’s room and/or to alter the users’ environment to make it look and sound more natural (e.g. by projecting wooden-like textures onto the user’s desktop—wood is known for its restorative effects (Burnard and Kutnar 2015)—or by playing natural bird sounds in the background), once again building on the spatial and multimodal characteristics of AR.

5.2.3 Embedded and Situated Representations

AR enables the design of “embedded” data representations, as formalized by Willett et al. (2017). Embedded representations “display data so that it spatially coincides with data referents”, i.e. with “the physical object of space to which the data refers”. The Playful Bottle prototype (Chiu et al. 2009) is an illustrative example: the data representation (i.e. the tree showing the amount of water drunk) is embedded into the physical object from which the data is sensed (the bottle). This differs from traditional ways of visualizing data, such as screens which usually present non-situated data representations. In fact, screens are often used to display information in a spatially-remote way (because the screen is not close to the object to which the data refer) and/or in a temporally-remote way (because what is being displayed is not live data). The design of embedded representations could be beneficial in terms of (visual) attention and cognitive demand, as it can make relationships between data and physical objects more explicit and therefore easier to perceive, interpret and understand. This could also decrease the perceived obtrusiveness of the applications.

The seamless integration of representations into people’s physical spaces (and for example into workers’ desktops and offices) could also lead to a more integrated approach of tools for improving people’s well-being. Instead of designing dedicated physical artefacts or installations for various aspects of people’s well-being, we could rely on physical objects such as desktops, chairs, clothes, walls, etc., that already exist within people’s environment and use similar metaphors or means of interaction for different aspects of well-being across the whole environment.

5.2.4 Playful Technologies

Finally, we see in the use of AR for ambient systems a good potential in terms of playfulness and gamification. Using different modalities can increase the user experience. It can also be a particularly interesting approach for the design of playful games. Another interesting aspect is that AR supports the design of animations that are not limited to one screen only and that can span the whole user’s environment, for example to draw the user’s attention from her laptop to a specific object in the room, such as the door (if the user needs to be more active) or the window (if the air quality is poor). Such multimodal and spatial animations could therefore be used to design playful ways of notifying or informing the users. In addition, when the whole room is augmented, it is possible to trigger cooperation or competition between people present in this room, using projections and/or spatialized sounds for example. This opens interesting avenues for the design of full-scale games or gamified applications.

5.3 Scenario

To better illustrate the potential of AR for ambient systems, we present a scenario of an envisioned system that builds on the design considerations previously discussed (see Fig. 6.8). Such a system could be implemented, for example, by using the above-mentioned Mirror Head system, thanks to which “projections can be moved around from one surface to another or remain statically projected onto one specific surface”.¹⁰

Fig. 6.8

Illustration of a playful ambient AR system to improve people’s well-being in the office. Projections can be used to highlights some elements (e.g. the window if the air quality is poor or a bottle of water if one person does not drink enough), to display footprints to suggest actions to users (e.g. taking a break), or to provide users with relaxing pictures (e.g. flowers on the desktop). Ambient lights can be used to help users focus on their work, or to indicate if someone is too noisy

Anna is an office worker who spends around 8 h a day in her office, alongside her two co-workers. Their office is equipped with different ambient lights as well as a set of projectors.

The system continuously projects an avatar onto each co-worker desktop that reflects their activity in a humorous way (playfulness). When Anna is working, she can also see the avatar working. However, the avatar sometimes moves around her desktop and her office to suggest her to do something (mobile interaction). For example, the avatar walks from its home place (a piece of paper or another element of the desktop) to Anna’s bottle of water to indicate that she must drink. The bottle is then highlighted and augmented with a line that tells Anna the amount of water she should drink (embedded data representations). The avatar can also move towards the windows to tell Anna that she needs to open it in order to lower the temperature of the room. If Anna does not notice the avatar, the objects or the furniture can be highlighted using ambient lights, and the avatar can come closer to Anna’s laptop to attract her attention. It can even “enter” the laptop for more obtrusive notifications when necessary (mobile interaction).¹¹ The avatar also mimics Anna’s sitting posture. Whenever Anna’s sitting posture is incorrect, the avatar shows her how she could correct her posture.

The avatar can also move towards the door (its footprints are projected onto the floor) to tell someone that he/she must take a break and be more active. This is in fact part of a game–each co-worker earns or loses points depending on their level of activity and the total number of points is continuously projected on the ceiling (gamification). At the end of the week, the less active co-worker must do something (e.g. bring a cake).

Tomas, one of Anna’s co-worker, is often stressed. His avatar reacts accordingly, showing signs of excitation and irritation. Tomas tries to breathe more deeply to help his avatar recovering a calm state. To help him achieving it, the system uses projected animations to display blooming flowers and growing plants on and around Tomas’ laptop. Tomas can also use his headphones to listen to natural sounds such as bird sounds (calm technologies).

As for Alice, the third co-worker, she is doing a demanding task and she needs a silent environment. The system detects Alice’s activity as well as the noises around her. The system first uses ambient lights to make Alice’s desktop more cosy (calm technologies) but also to indicate to Anna and Tomas that Alice needs to be very focused. It also highlights Anna and Tomas’ desktops to reflect how noisy they are. As Tomas’s desktop is very red, he understands that he is speaking too loudly and starts speaking more quietly.

All of them are also regularly encouraged to look away from the computer to prevent visual fatigue.¹² Twice an hour, animated images are projected onto an existing frame placed on the wall and are used to subtly attract the co-worker’s gazes. The images disappear when the workers have been looking at them for a certain amount of time, or become more and more visible otherwise.

The above scenarios illustrated some possibilities of what could ambient projected interface provide. In Table 6.1 we categorised these and other possibilities based on the projection type and the type of intervention as described in this chapter.

Table 6.1 Examples of playful ambient augmented reality systems to improve people’s well-being divided by projection type and intervention targeted

5.4 The System

While designing a system as described in the above scenario, several issues have to be considered. As visible in Table 6.1, we have not used any example of constant animation in periphery. Animation grabs attention, it can disturb users’ workflow and it should be used sparingly and only when needed. For example, in our scenario we have described that the avatar starts moving from the periphery into the field of view only when there is a need for it. It is important that every intervention starts in an inobtrusive way in the periphery and that the changes are not sudden but happen over a period of time and slowly move into the users’ visual field to engage them in interaction. Moreover, the systems should not overwhelm the users with suggestions and they should learn what kind of intervention the users react to. Users in a study of triggers have for example reacted more positively when triggers were adapting to their behaviour, as compared to non-adaptive triggers that have led to trigger overload and consequently to users ignoring them Kljun et al. (2018). Similar information overload has been shown also for other information types such as email, social networks (Maier et al. 2015), and other fields (Eppler and Mengis 2004).

The need for users to have a control over their environment has also been stressed as important in the literature (Ghani and Deshpande 1994). Being unable to control the system or interacting with a system that behaves in an unpredicted way (e.g. not notifying the user of what has been done and why) often leads to frustration and helplessness. Even worse, it can decrease users’ comfort and lead to stress. It is thus necessary to give users the possibility to refuse suggestions made by the system, if they are temporarily not able to or do not want to interact. Users should also be able to turn off the system if necessary. In addition, there are some tasks that should not be interrupted such as meetings. The system should be aware of such occasions and start engaging the user only after such tasks are over. Overall, the system should be designed to help overcome frustration rather than create it Klein et al. (2002).

For the implementation of the above-mentioned projection types, we propose a mirror head projection system which moves the projection around the room by rotating a mirror (see Fig. 6.9). In such a system the mirror rotates around two different axes, on Fig. 6.9 marked as pan and tilt angle. For the above scenarios an ideal mirror head system would enable 180° of tilt and pan rotations, enabling one to direct the centre of projection to any point on half sphere (see Fig. 6.9). Existing commercially available systems, such as the Mirror Head¹³ from Dynamic Projection Institute can pan the projection in the range of 180° and tilt the projection in the range of 162°. Even though this is not ideal, such a system provides sufficient capabilities for building a prototype system, particularly when the projector is mounted close to the top of the ceiling.

Fig. 6.9

Projector with a digitally-controlled mirror that can project images and textures into various elements of a room

The second consideration one needs to make is the fact that the projection distance, defined as the distance between the surface of the projection and the projector, changes as the mirror head moves the projection around. This introduces two additional points to consider: the projection size and the projection focus. As the projection distance changes, so does the projection size, which in turn affects the pixel density of the projection.

Here we estimate the projection distance for the proposed scenario of the office space. If we consider mounting the projection in the centre of the room with a standard ceiling height of 2.4 m¹⁴ and a room size of 4 × 4 m, the projection distance varies from 1.4 m (when projecting directly down onto the table surface) to 3.7 m (when projecting into the farthest corner of the room, cf. (6.1)).

$${\text{Max}}\,{\text{ projection}}\,{\text{ distance}} = \sqrt {\left( {\frac{width}{2}} \right)^{2} + \left( {\frac{length}{2}} \right)^{2} + (height)^{2} }$$

(6.1)

If the throw ratio (defined as projection distance/width of projection) of the projector’s optical system is not changed, the size of projection changes linearly to the projection distance. For the proposed room size, the size of the projection from min projection distance to max projection distance would increase by a factor of 2.6. To control this change of projection size and keep the projection in focus, we propose to use a projection system with a motorized zoom lens and focusing. These features are available on some commercially available projectors, such as Optoma zu1050.¹⁵

Another consideration is related to the required brightness of the projector. Ambient light, projection surface, size and desired contrast of projection are key factors to estimate the required brightness of a projector. For the scenario above, we limit the projection size to 1.2 m in diagonal and calculate the required projector luminance as follows.¹⁶

As stated by the guidelines in a guide published by Dnp danemark,¹⁷ the projection luminance should be 2–3 times the luminance at the desk in order to provide the users with a comfortable viewing experience. In addition, according to US General Services Administration,¹⁸ 500 lx of light is deemed appropriate for the workspace. Hence, we require a brightness of 1000 lx. To estimate projector brightness, we also need to consider other factors such as: surface reflectance (we take reflectance factor of white wall which is 0.25), projector contrast (2.000.000/1),¹⁹ desired contrast (20/1), image area (0.848 m²)²⁰ and adjustment for calibration loss (10%). According to Eq. (6.2) and (6.3), the minimal required projection brightness for our scenario is therefore 7000 lm.

$$\begin{aligned} {\text{image}}\_{\text{brightness}} & = \,{\text{light}}\_{\text{at}}\_{\text{projection }}*{\text{ surface}}\_{\text{reflectance}}*{\text{projector}}\_{\text{contrast}} \\ & *\, ( {\text{image}}\_{\text{contrast}} - 1)/\left( {{\text{projection}}\_{\text{contrast }} - {\text{ image}}\_{\text{contrast}}} \right) \\ & = 500*0. 2 5* 2{,}000{,}000*\left( { 20 - 1} \right)/\left( { 2{,}000{,}000 - 20} \right) = 2 3 7 5\,{\text{lx}} \\ \end{aligned}$$

(6.2)

$$\begin{aligned} { \hbox{min} }\_{\text{projector}}\_{\text{brightness}}\, & = \,{\text{image}}\_{\text{brightness }}*{\text{ image}}\_{\text{area }}*{\text{ PI}}/{\text{screen}}\_{\text{gain }} \\ & *{\text{ adjustment}}\_{\text{for}}\_{\text{calibration}}\_{\text{loss}} \\ & = 2 3 7 5\,{\text{lx}}*0. 8 4 8* 3. 1 4/ 1* 1. 1= 6 9 5 6\,{\text{lm}} \\ \end{aligned}$$

(6.3)

The above considerations illustrated key characteristics to build a system that is capable of creating the projections described in Table 6.1, allowing one to implement the foreseen scenario.

6 Conclusions

In this chapter, we discussed the potential of gamification of the design of AR ambient systems for the improvement of people’s well-being. We first briefly described several aspects of well-being that have been addressed within the HCI community, such as stress, sedentary behaviour and hydration. We also included the concept of “comfort” in the scope of this chapter. We believe that the continuously growing number of pervasive (bio)-sensors provide excellent opportunities for the development of tools that could improve people’s well-being, in particular within the context of work.

Although these tools are often implemented as mobile/wearable apps or websites, researchers have also investigated other ways of exploiting these data in innovative ways, using different types of interaction. In the framework of this chapter, we focused on Ambient displays, including augmented-reality based displays. In fact, we highlighted two properties of projection-based Augmented Reality that make it an efficient medium for Ambient displays: it supports spatially-distributed information and interaction, as well as multimodality. We thereafter proposed the notion of Ambient projection-based AR systems as a subset of Ambient displays that (1) augment existing environment with digital content; (2) use digital content that is registered in 3D and can be moved around the user’s environment; (3) can be interactive in real-time.

In addition, we discussed how gamification has been successfully used for well-being and health purposes, in particular for mobile apps and websites. We also reviewed the playful and gamified elements of existing Ambient AR tools. Given the promising results, we suggest the use of gamification with Ambient AR systems as an efficient way to improve people’s well-being. To better illustrate our vision, we concluded this chapter with the description of scenarios related to the use of a playful AR ambient office that promotes healthy behaviour and attempts to improve workplace comfort level.