Keywords

1 Introduction

Augmented reality (AR) technologies have become popular for developing a variety of entertainment services. AR is typically used to spatially present virtual images or textual information superimposed on the real world either to help people make better decisions or to present remarkable expressions [1]. The potential power of AR makes it possible to change the semiotic meaning of the real world, strongly influencing human attitudes and behavior to adopt a better lifestyle, thus solving common social problems such as environmental sustainability. Typical AR approaches can simply integrate computer-generated virtual images into the real world. The images typically present invisible information that cannot be seen by a user in the real world. Although it is possible to show fictional images to enhance the meaning of the real world, this approach limits the potential power because only some pieces of real-world objects can be enhanced. In contrast, current virtual reality (VR) technologies make it possible to create a new fictional (but realistic) virtual world by using 3D models of real persons, objects and landscapes in the real world; it is possible to present fictional events in the virtual world because believe that the events actually happened in the real world.

This paper proposes a new design strategy named Alternative Reality, which overcomes the drawbacks of AR techniques by integrating VR techniques; a new type of ambient intelligence experience can be offered. Although a user watches the real world, some scenes of the real world may be temporally replaced in favor of virtual scenes that are actually represented as the virtual world for a short period; however, he/she is not aware of the inserted virtual world and feels that the fictionality in the virtual world actually occurs in the real world because real landscapes, persons and objects are used to construct the virtual world.

Because the hybrid real world can be represented abstractly, remarkably, or ironically as fictional virtual scenes through framing to either simplify or exaggerate essential concepts in our daily lives, people easily notice important concepts that are relevant to achieving an ideal, sustainable society through fictionality [17]. Accordingly, this approach can be used to overcome serious social problems through human behavior change [21].

In Alternative Reality, a user wears a head-mounted display (HMD) and the virtual world watched by the user consists of real landscapes and real persons constructed as a 360-degree video movie; thus, the user can interactively navigate the virtual world. Because the virtual world is seamlessly integrated with and temporally connected to the real world, a user feels that fictional events in the virtual world are performed in the real world even though the virtual-world events are not real. This approach provides a novel method of enhancing the meaning of the real world to develop advanced ambient intelligence experiences, thereby integrating the transmedia storytelling approach to incorporate fictionality into the real world [8].

In this paper, after presenting an overview of Alternative Reality, we introduce its basic concept. To demonstrate the feasibility of the concept, we have developed two case studies in which fictional events occur in a real location. The first case study investigates whether a user can believe that the improbable behavior of a moving object is realistic, where the user can interact with the object. The second case study investigates whether a user can experience fictional occurrences in the virtual world as in the real world. In both case studies, a user wears an HMD to increase the immersions of the hybrid world created by Alternative Reality, in which the virtual world is created by capturing the real world with a 360-degree camera. We conducted user studies to extract insights that would help us investigate the feasibility of the Alternative Reality concept, even though the current work is still at a preliminary stage. The insights of the experiments with the case studies show that Alternative Reality effectively augments the real world without losing sight of reality.

2 A Framework for Augmenting Real World Through Inserting Virtual Scenes Temporally

There are several ways to incorporate fictionality into the real world. One typical approach is to use live action role playing (LARP) [12] or alternative reality games (ARG) [10]. During LARP, players play fictional roles based on a pervasive role-playing concept [11] and a game master to control the gap between fiction and reality. ARG adopts a concept named transmedia storytelling [18], using multiple media to incorporate fictional stories into the real world. These approaches are promising, but the approach requires a rigorous plan that requires a long time to reduce the gap between fiction and reality. Augmented reality and virtual reality technologies offer another possibility to incorporate fictionality into the real world. For example, in [14, 22], by using head-mounted displays, a user immersively changes the meaning of the real world to alter his or her attitude and behavior.

In Alternative Reality, a user watches a sequence of scenes on an HMD. As shown in Fig. 1, the sequence consists of several scenes. Some scenes are captured from present scenes in the real world (Real Scenes in Fig. 1). The scenes are recorded by a 360-degree camera, and shown on the HMD in real time.Footnote 1 However, some scenes in the sequence are not real; the scenes might be actually constructed through VR techniques and are fictional (Virtual Scenes in Fig. 1). Additionally, in the virtual scenes, several events might not occur in the real world as it is now. In particular, Fig. 1 presents that there are two persons in the room. One person watches the room through an HMD. Real scene A or C is a video that captures the room as it is in the present. Virtual scene B or D is a video that captured the room in the past. The person who wears an HMD watches the virtual scenes and he/she can feel that the other person is in front of him/her. However, in the real world, the person is actually behind him/her.

Fig. 1.
figure 1

A timeline model of alternative reality

Full size image

Typically, the scenes are constructed using the 3D models of real persons, objects and landscapes in advance, but some real persons who might not be actually present now can appear. One of the important requirements of Alternative Reality is that a user feels that these real and virtual scenes are continuous and thus is unaware of the boundary between the two scenes. The magic circle is defined as the boundary between the real world and the virtual world [15]. If a user is not aware of a magic circle between the worlds developed by Alternative Reality, the user cannot notice that the virtual world generated by Alternative Reality is not real. Therefore, he/she feels that the virtual scenes actually happen in the real world. The most important issue in realizing immersion blurs the magic circle. The use of an HMD offers a better immersive experience by showing a video stream that captures the real world and then replacing some real components in the video stream with fictional components [16].

Several previous investigations used a 3D model composed from real scenes. For example, in [2], a user interacts with the 3D model of a building to learn routes inside the building. The user can learn the real routes in the real town in the virtual world. Alternative Reality similarly uses the virtual world composed from real persons and landscapes, but the virtual world may contain probable events in the possible future that either might not actually occur in the real world or are fictional events. The events and the stories created from them are perceived as an experience in the real world. By integrating possible future or fictional events into real events, a user can accept the events as his/her own real, present experience; thus, the presented stories can influence the user’s future behavior [7].

Here, we show two typical examples of the benefits of adopting the Alternative Reality concept:

  • Pervasive Game: Pervasive games enable us to play games in the real world [12]. Extending a user’s experience from the pervasive game’s world to the hybrid (real and virtual) world would be easier with Alternative Reality.

  • Overcoming Social Problems: Behavior changes are essential to overcome various social problems [6, 26]. Alternative Reality enables an awareness of social problems such as environmental or health issues because people notice more easily the effect of the future events shown as Alternative Reality.

In this paper, we focus on the feasibility of the Alternative Reality approach to influencing human behavior and extract some useful insights to develop services based on Alternative Reality.

3 Case Studies

The current case studies demonstrate the feasibility of Alternative Reality, for which we have developed movies that connect real and virtual scenes. In the case-study experiments, we presented the movies to participants by wearing HMDs and interacting with the 360-degree movies by moving the participants’ heads. Although the case studies did not present the full potential of Alternative Reality, we can still extract useful insights and potential pitfalls to guide our approach towards the next step.

3.1 Interactive Improbable Object

“Interactive Improbable Objects” has been developed as a first case study, which is a movie in which a moving object with which a user can interact behaves in an improbable way. A user watches the movie through an HMD (Oculus RiftFootnote 2). This experiment investigates whether a user believes an object’s improbable behavior to be realistic when the object reflects his/her interaction.

In the “Interactive Improbable Objects” experiment, participants watched the movie as scenes taking place before them in the present. Because the movie is captured with a 360-degree camera (BublCamFootnote 3) from the participants’ locations and they watched it through HMDs, they can feel the movie as though it is the present real world. In the movie, virtual scenes show a moving object, but that object’s movement usually is not natural in the real world. However, the object does return feedback with the participant’s interaction. For example, when he/she asks a question of the object, it stops to listen to him/her and turns its head toward him/her. The movie is captured in advance as time-lapse imaging and is shown with 15 fps. Figure 2 shows some of the time-lapse images in the movie. An object moves from left to right (left scenes in Fig. 2). When a user asks the object a question, the object stops to listen to the user’s voice and turns its head to him/her (right scenes in Fig. 2). When a participant speaks to the object, as shown in the right-hand scenes of Fig. 2, the object reacts to his/her voice.

Fig. 2.
figure 2

Interactive improbable object

Full size image

Eleven participants participated in the experiment and were the subject of semi-structured interviews. In the interviews, we asked participants whether they feel the moving objects to be strange. In addition, we asked whether the movement becomes realistic by allowing them to interact with the object.

Most of participants said that at first, the object’s unexpected movement amazed them; however, they gradually felt that the movement was realistic and that they experienced no uncomfortable feeling related to the improbably moving object. Moreover, some of them said, “The object was like a machine because it was moving regularly”. However, once they interacted with the object, most of participants said that it looked like a living thing. Offering interaction with an artificial object makes it believable as a living thing with its own will.

One design issue that is important for increasing the reality of an improbable object is the fact that people usually predict how to move the object based on either their technological predictions or past experiences. For example, we can easily predict the movement of an object that has wheels or whose shape is like a humanoid robot. From the results of the experiment, the movement also becomes acceptable when it obeys physical laws. In the case study, if the object moves with a simple harmonic motion, a participant feels the movement as though the object is a living thing; however, he/she could find the movement unnatural or mechanical if it moves with uniformly linear motion.

To achieve interaction with an object as a living thing, its latency in returning feedback is also important. When there is a time lag before the object’s head turns when a participant asks a question, the participant will not feel that the object is truly a living thing.

3.2 Fictional Future

“Fictional Future” has been developed as a second case study that is a movie containing both present and possible future occurrences to demonstrate the Alternative Reality concept. The case study is performed by real persons whom know well each other and in real locations known to the participants such that the users believe that these occurrences are real. This case study is similar to the one developed to demonstrate the feasibility of the Substitutional Reality (SR) system [25].

In the “Fictional Future” experiment, a participant watched three movies that were merged into one movie through an HMD similar to “Interactive Improbable Objects”. The first movie is a real-time captured movie that shows the current real location in detail. The second and the third movies are constructed based on the Alternative Reality concept and the scenes in the movies are scenes performed in the past as a possible future. The second movie was performed by an experimenter and captured the scene of the experimenter and participant entering the room. The third movie was performed by the experimenter and consisted of possible future occurrences, such as gratefully giving money, which the experimenter believes has happened. Figure 3 shows the screenshots of the second and third movies. In Fig. 3, the first photo shows the second movie where a participant and an experimenter entered a room as a past event, the second photo shows third movie where the lady moved ahead as a possible future event. Finally, the third photo presents the third movie where the lady explained the occurrences as a possible future event. After watching the merged movie, when the participant removed the HMD, the experimenter performed the same activities performed in the third movie. Thus, a participant watches the same scene shown in the third movie.

Fig. 3.
figure 3

Fictional Future

Full size image

Eleven participants participated in the “Fictional Future” experiment, and we conducted semi-structured interviews with the participants. In the interviews, we asked participants whether they feel that the possible future in “Fictional Future” actually happened in the real world. If their answers were positive, we also asked them why they felt that the potential future depicted represented reality.

Some of them said, “After taking off the HMD, when I was viewing the scenes shown in the third movie, I felt déjà vu to be seeing the same scenes again.” This means that the participants experienced the merged movie made by the Alternative Reality concept as real scenes and that after they removed their HMDs, they felt that they had seen the scene before. Additionally, some participants said, “I felt and expected that the activities in the future movie would happen because the activities are desirable and make me happy”. Similarly, after the experiment, a participant asked, “Can I get some money?” This is because we explained the concept of money given in gratitude in the third movie; this comment means that the participant believed in the possible future presented in the third movie.

One interesting finding of the experiments is that people tend to think that the performed events are real if the activities are common or desirable. This enables us to believe that the events occur in the real world as it is right now. Additionally, the realistic landscape is a key to making the depicted events realistic. However, of course, it is important to ensure a sense of the reality of the virtual scenes. If a user loses a sense of that reality, the user confuses what happens in his/her present life and is aware of the “magic circle” boundary because he/she may feel that even real scenes are not real. A user also tends to remember remarkable situations. When a participant feels déjà vu in the experiment, it is typical that these events, such as putting a doll on a desk, touching a doll, or leaving a room, are noticed.

3.3 Design Implication

One important design issue implicated by Alternative Reality is whether a user feels that virtual scenes represent reality. As shown in [7], increasing visual reality is not the only way to make a user sense reality. The feasibility studies described in this section present some evidence that even events that did not actually occur in the real world can be incorporated into the real world without losing reality. More specifically, the results from the first case study demonstrate the feasibility of an unrealistic fictional event that can offer very strong stimuli for designing advanced, pervasive games. However, our current participants may be the most accustomed to virtual content such as animations or movies making improbable objects feel realistic. For example, one participant said, “I have seen this movement in some movies”. Conversely, another participant from a foreign country said, “In my country, a moving object by itself is thought as something like a ghost”. He also said that the experience was scary and uncomfortable. We need to design fictionality in Alternative Reality according to a user’s experiences and culture.

As shown in [17], fictionality has a powerful persuasive effect to overcome serious social problems. Alternative Reality offers powerful persuasiveness through immersively incorporated fictionality. More specifically, as shown in [13], the persuasive power of fictionality can be increased by presenting it as transmedia storytelling. The second case study demonstrated how to insert virtual scenes so that they would not lose their reality. The virtual scenes may be portions of transmedia stories that can teach people the importance of overcoming their problems. Persuasiveness designed as digital rhetoric is a promising direction for offering abstractions to design the enhancement of meaning in the real world [20]; the abstraction can be used to design virtual scenes that influence human attitude and behavior more easily.

4 Related Work

Augmented reality techniques can be used to enhance existing artifacts. For example, [27] describes several augmented reality games that are enhanced versions of traditional physical games. Specifically, Augmented Go [9] demonstrates a promising approach to maintaining the advantages of the physicality of the board game while adding virtuality. Virtual Aquarium [13] shows a virtual fish tank in which the movement of the fish reflects a user’s tooth-brushing behavior. Enhanced TCG [23] enhances our real world by replacing a real-world component with a fictional component for changing the semiotic meaning of the real world.

Most recently, digital marketing and social media practitioners have referred to this approach as gamification. The addition of badges and leaderboards is a typical approach to achieving gamification [3]. The idea is to use game mechanics, such as those in online games, to make a task entertaining, thus encouraging people to conscientiously complete target goals. In [5], gamification is defined from a service-marketing perspective as follows: “A process of enhancing a service with affordances for gameful experience to support user’s overall value.” In traditional gamification, a set of game mechanics is widely adopted for motivating human behavior; however, incorporating game mechanics into the real world is not easy. Thus, simple mechanics such as badges, leaderboards and points are typically used.

When attempting to solve serious social problems such as sustainability issues, health issues, and happiness [6, 26], guiding people’s attitudes and behaviors is an important design issue. Enhancing the semiotic meaning of the real world through incorporated virtuality is a powerful technique for altering people’s attitudes and behaviors [19]. As shown in [24], procedural rhetoric is a promising theoretical foundation to increase persuasiveness by making the enhanced real world meaningful.

Several case studies use augmented reality technologies to enhance the meaning of the real world, thus influencing people’s behavior. For example, in [22], the authors propose a service to both implicitly influence people’s satisfaction while drinking a beverage and to control beverage consumption by creating a volume perception illusion using augmented reality technologies. The system proposed in [14] realizes a method for modifying perceptions of satiety and controlling nutritional intake by changing the apparent size of food with augmented reality technologies.

In [4], Dunne and Raby use design to offer new forms of expression for complex and critical issues; these forms of expression are grounded in the most abstract, speculative and future-focused considerations. Critical questions about emerging technology in everyday situations have presented preferable futures as opposed to predicting the future. They call this design approach Speculative Design. The approach taken in Alternative Reality can be considered an example of Speculative Design because the aim is to investigate whether a user feels as though he/she is watching a future scene when the scene uses only components that exist in the real world.

Some works produced by Sputniko! are similar to Alternative Reality. For example, popular works such as Crowbot Jenny and Menstruation Machine, Takashi’s Take use only existing persons, landscapes and prototype artifacts to create futuristic movies (http://sputniko.com/works/). Although they are similar to science fiction movies or fantasy dramas, these movies do not use unrealistic artifacts; instead, they present influences and debates by introducing emerging new technologies in our real lives. However, these art works traditionally do not offer the artists’ explicit interpretations; instead, interpretation is open for audiences. Thus, audiences may feel that art is boring because they cannot understand its meaning.

5 Conclusion

In this paper, we proposed Alternative Reality, which enables us to change the meaning of the daily urban world by inserting fictional virtual scenes among real scenes. The virtual scenes are constructed from real-world components so that a user believes that the virtual scenes actually take place in the real world. to demonstrate the design strategy of Alternative Reality, we have developed two case studies. The first case study investigates whether a user can feel that the improbable behavior of a moving object is realistic, where the user can interact with the object. The second case study investigates whether a user can experience fictional events in the virtual world as they are experienced in the real world. We extracted some insights that can be used to develop future services based on Alternative Reality.

As future directions, we consider the following two issues. The first issue is to discuss the relationship among similar frameworks, particularly the gameful digital-rhetoric framework [21] and the value-based analysis framework [19]. We need to discuss how these frameworks are mutually related, along with possibilities to integrate these frameworks into a more generalized framework. The second issue is to explore the possibility of using advanced ubiquitous computing technologies for designing methods of persuasion. Sensing technologies enable us to develop persuasive methods that are customized for each person. Moreover, advanced wearable technologies may change the perceived meaning of the real world. The studied technologies allow us to develop more effective persuasive methods. We also need to discuss how procedural rhetoric concept is related to persuasive methods that are enhanced through ubiquitous computing technologies.

In the next step, we will try to develop and evaluate a new prototype service based on the Alternative Reality concept. This service will present a long fictional scene to a user, where the scene embeds several persuasive messages ambiently. We like to evaluate whether he/she feels that the scene occurs in the real world and how the scene influences the user’s behavior changes when the user watches the scene. The new service will show the essential persuasive power of Alternative Reality and indicates the potentiality that the current case studies could not clarify.