Keywords

1 Introduction

Prototyping and exploring worlds and environments for creating exhibitions with and through technology we sometimes use consumer technology that shortens and speeds up our work process, and we can start sketching in hard- and software already early in the conceptual phase of a project. Even consumer products today come with a developing kit and widely open APIs which make them suitable also for experiments and research. One risk although here is that one as an artist, designer and/or researcher gets trapped inside of the presumed future use of this technology, with all limitations that comes with that [9]. Relying on “produced” technology we clearly see a need for a set of tactics to bring our work in our own direction. In our recent practice based work and research, that focuses on exhibition work that spans from 2015 and onwards, we have returned to the idea that if we want to have an audience or a visitor experience 3D spaces of different sorts in an exhibition format, we need to find ways for the visitor to enter, interact and experience these virtual worlds, without too much former knowledge how to enter, navigate and perceive them in order to have the visitor feel both present and immersed by them. Because there is always a risk with new, not seen and experienced technologies that we have the visitor exploring, the technology itself instead rather than the topic of our attention, e.g. spatial constructions. Through our work we gained knowledge that many forms of interaction takes hours of practice to learn and even longer time to master, and therefore are not fit for use directly in an exhibition, but valuable tools for us as artists, designers and technicians to help us create things that have not existed before. In this line of work, we have learned that configuring space via user participation and interaction is not easy, but crucial for an immersed experience to take place. Throughout our work, developing, methods, tools and processes we try to emphasize the importance of a multiperspective view of space and its entities based on the idea to transcend merely scientific or artistic approaches into a more comprehensive and immediate approach and working practice, in which we try to use different forms of interaction and telepresence to create a state of immersion. It is about symbolic objects and entireties (the issue of “Gestalt”, not about mere construction and functions). To have the created worlds to stand out as something believable (the issue of “representation”) and how through thoughtfully designed user interaction can create an immersive experience for the visitor.

2 Our Three Use Cases: Step in, Fly over and Cruise

The origins of the environments for this article: The used spaces are all derived from the exhibition at the Biennale of Architecture in Venice 2016. Starting from this basis, selected places (i.e. the shown worlds) were made accessible with new media devices in order to explore how the spatial impression is changing as a result of this new approach.

Fig. 1.
figure 1

User’s perspective: (a) “step in” (b) “fly over” (c) “cruise” as WiP.

Full size image
Fig. 2.
figure 2

Setup: (a) “step in” (b) “fly over” (c) “cruise”

Full size image

2.1 Step in: The CAVE Installation

The installationFootnote 1 was shown from May to November as part of the Architectural Biennale 2016 in Venice. The setup consisted of three projectors displaying a seamless image onto a tryptic screen (total size: 3.60 \(\times \) 6.0 m) (See: Fig. 2(a)). The screen itself consisted of three canvases, each at an angle of 150\(^\circ \) to each other. This CAVE-like installation—where also the user’s peripheral viewport was covered—allowed each visitor to “dive” into the shown environments and therefore enabled an immersive experience, and to create a bodily notion of that featured space. On this projection screen, pre-rendered movies were shown. Due to the fact that the camera movement was not bound to any (simulated) physical or time constraints free movement in all dimensions was possible: The environments were shown from different perspectives, but always with a constant focal length (35 mm). This approach was chosen in order to offer visitors a realistic size estimation and a feeling of “being there” (dweller/pedestrian).

Findings: A: Size matters, being almost inside of a projection in a cave projection creates a presence of the spaces shown. B: Since every visitor individually can approach the cave projection and also share the experience directly with others. C: No technical barrier for the users to overcome. D: Immediateness: No latency/lag and low-threshold for a majority of users.

2.2 Fly over: The Balloon Installation

For the ZKM—Center for Arts and Media in Karlsruhe, Germany we wanted to set up an installation that is connected to the place where it is exhibited. Because of its special kind of city planning (“fan city”) we already presented a version of Karlsruhe in our CAVE Installation and therefore it was of interest for us to make it accessible (here: Karlsruhe from 1834) from another perspective by riding over it via a balloon. In general spaced VR installations have some inherent disadvantages for each individual visitor: The action space available to the user is very limited and any danger of collision must be prevented in advance, e.g. by displaying warnings (“chaperones”). How useful these interventions might be, they strongly interrupt the immersive experience. Therefore, a concept is required that plays creatively with the limitations of VR and simultaneously offers firm support in case of need. Hence we developed an installation that naturally and credibly shapes the playing space without being perceived as a limitation. We chose to build a replica of a balloon-basket that is also mapped 1:1 in the virtual environment (See: Figs. 1(b) and 2(b)). With this setup our installation features: 1. birds-eye perspective for experiencing the special kind of Karlsruhe’s architecture 2. slow movements to reduce motion sickness 3. When wanting to ride/fly over a city there currently is a rise in hardware that require users to strap themselves onto it, which might be not perfectly suitable for (semi-) public space and also for physically handicapped or introverted people (see [21] or [17]) and therefore we thought a balloon might suit better. In addition, clinging to the balloons railing also reduces motion sickness and increases immersion [3]. To start the ride, visitors had to enter the nacelle, which is also represented within the virtual environment and pull down the rope (represented in physical and virtual space) to start the ride/virtual burner (See: Fig. 2(b)). The ride took place on a pre-given path with the possibility to change height by pulling down the rope to fuel the burner. In the real environment, the burner was represented by spotlights which radiated enough heat and was augmented with vibrations of the nacelle to foster the illusion. Another trick to further improve the illusion of flying, was to add an airstream using a fan. This, of course, is not correct from a physical point of view, but fits well to most of the visitors (most of them who have not yet experienced a real balloon ride).

Findings: A: One can increase immersion using elements that might not be physically correct (wind in a balloon, vibration when pulling the trigger). B: One can regulate viewing direction by implementing spatially located audio files. C: Intentionally regulating the given space by using a physical restriction also displayed in the virtual world (nacelle) helps people to orientate oneself within the virtual reality. D: Although HMDs are becoming increasingly popular and we added clear instructions next to the exhibition, museum guides still had to attend the exhibition to help out the visitors.

2.3 Cruise: The Bike Installation

The bike installation was shown to a broader public in an art/design performance/exhibition in two cities (Kristianstad and Copenhagen). In this art/design project with the bike we returned to Jeffrey Shaw’s original idea from 1986, “Legible City” [18]. The Bike installation was one of five stations in which the visitors could explore different ideas about urban development, here in this installation we had the visitors to visit 15th century Milano to experience a conceptual city space never realized. “As a space, [it] is constructed in such a way that we always have the impression to see only a fragment, a more or less small section of it—because at every meters x, the next structural element can appear, ad infinitum” [5]. For this project we use a VirZoom bike with its speed and direction sensors to explore two of the worlds from the 2016 exhibition. Here we want to see what happens when you take some worlds developed for animation/film and translate that world into real time graphics (Unity3d). What specific qualities get lost and which transfers well?

Findings: A: Direct immersion for the visitor when entering a virtual world with a familiar navigational device such as the bike. They know how to direct themselves almost immediately and there is a low threshold to learn how to navigate oneself. B: Biking is not walking, therefore a space needs some adjustment: this was a bit tricky since this world needs to be changed to fit navigation by bicycle rather than by walking, for example have the bicycle go from one floor to the next, using stairs could have been an option, but does not relate very well to cruising—it will be a bumpy ride. We also wanted to keep the original plan of the city and not introducing “modern” or “alien” elements into it. So to keep the flow in constantly biking through Milano, we used ramps as discreet as possible integrated in the original environment to move the visitor on it’s bicycle in between the different levels of the city, and at the same time point the biking experience for the visitor into new directions. C: The bike soon became an important tool developing the worlds—iterating between modeling/texturing/lighting and the experience of the changes made by bicycling in that area of attention. D: It made us aware of the difference between seeing something on the screen (editing) and experience it in full VR [19]. E: The bike itself is as we found out also a camera rig and can be used for classical camera work to get camera data out to other 3d programs but also to produce animation paths for Unity3d itself.

3 Thoughts on Audience Participation

One downside that comes with head mounted devices (HMD) in (semi-) public spaces is, that they can only be used by one person at the time. In addition, while experiencing the virtual world, the person who wears the HMD is almost completely isolated from his surrounding although other visitors can easily observe him. This “voyeurism” behavior is quite similar to the one described as one of the first parts of the audience funnel framework by Michelis and Müller 2011 [15]. Their framework is based on observations made in regards to interactions with public displays, but while they argue that visitors either pass by the display or view and react, we would like to add another category: The voyeur implicitly does not want to experience the VR exhibition, but wants to passively observe and find out what is happening. While in general, the term voyeurism has a rather unpleasant notion, in this setting, this behaviour can lead to a honey-pot effect, which then will bring other people to observe and interact with the current user/voyeur or to try it by oneself. There are several possibilities one can come up with to include bystanders into the installation (see Fig. 3): 1. Directly display the virtual world onto a 2D display placed so that bystanders and voyeurs can see it. 2. Include the actual user into the 3d world using a greenscreen as described by Intel in 2017 [22]. 3. (Floor) projections [7, 23]. 4. Project facial expression on the front of the VR headset for bystanders to see it [11, 13]. There is also a lot to decide about on how to add auditory cues, but it would extend the scope of this discussionFootnote 2. In our current state, we are considering several possibilities in engaging visitors to interact with the current user. For example: should the visitor be able to see the virtual content on a 2D screen before he got fully immersed into the virtual environment via HMD or should one try to only make the content visible for participants after they went into the virtual reality itself?

Fig. 3.
figure 3

Audience “funnel” for “Biketopia” VR exhibition.

Full size image

4 Thoughts on Immersion

By using familiar or at least conventional devices, such as the CAVE, the balloon and the bike, the users’ direct physical presence in the virtual environment reinforced the sense of physical presence of the virtual world and the interactivity of the respective space or device used in our case studies enhance their sense of immersion. We tried to lay the “focus on the activity, not the technology” and made “the control mechanism obvious” [14] when designing the necessary technical part of our installations. Conversely, this means that immersion cannot be designed directly, but that we have ensured that as many interfering factors as possible are eliminated. As an artist/designer one should always be aware that one never designs the actual immersed experience itself, but only the framework wherein that experience can take place. Since the amount of immersion is not easily measurable, we classify our three works by using the parameters Slater and Wilbur came up with [20]. As one can see in Table 1 the range of sensor modalities (Extensiveness) used, varies. Also the idea on how visitors can interact (Interactability) with the different installations is altered between the exhibitions.

Table 1. Immersion in dependence on to the parameters defined in [20]
Full size table

5 Thoughts on Interaction

On interaction we played safe and in each of our examples we deliberately used well familiar forms of interaction by having the visitor, “step into”, “fly over” or “cruise” our different environments. We wanted established forms that helped the visitors more directly make the transition from the physical world on a device they already used or at least seen before. Therefore interactivity and navigation used in both the balloon and the bike was directly added to the experience by a tight coupling between display, the movements of the user’s body (when maneuvering the device) on which they made the transition from one world to another. In the case of the CAVE example there were no interaction through any technology, here we instead used the design of the physical space itself to have the visitor just step into the worlds showed. Fundamentally, we can state that the concept of affordance [6] from interaction design can also play to its strengths in our context: Although it is ultimately just a sophisticated controller for a virtual environment, the bicycle is clearly recognizable to every visitor. This simplicity and accessibility effectively prevents fears of contact and takes little time to get used to. The balloon can show these strengths even more effectively: The basket restricts the possible degrees of freedom very effectively without actually being perceived as a plain restriction or overruling. We believe that the museal context demands this mixture of strict constraints and clear affordances to be successful. As a designer it is our job to find concepts that integrate these principles into a harmonious environment that plays creatively with these limitations and, at best, uses them as framework for conceptual design.

6 Developing Visual Style and Aesthetics

Developing through iteration is common practice in many art/design disciplines and also for our type of work in which we strive to facilitate an openness towards what happens in-between the design cycles. Similar to the OODA-Loop [2] in which one Observes, Orients, Decides and takes Action, in our case to challenge the machine and software logic and our own limits and conventions, to produce something that is both unexpected and valuable, and in the end will point out possible new directions. So using this loop switching between the state of editing (tweaking the parameters) and experience (being there) help us develop and put forward qualities (that sometimes is produced by errors or wrongdoings) that are hidden from the concept’s point of view and by doing so—have the concept redeveloped itself through the results and experiences we achieve by iteration. We try to explore what is being shadowed by the concept itself. Therefore we do not formulate any detailed specification in relation to the concept beforehand, or rather loosely “not photo real” or “do not use textures”. But at the same time it is important for us to be able to control or at least understand the in- and outputs of the development environment itself—from bits to spaces. The shortcomings, errors and quirks are vital resources that suddenly can reveal themselves as major feature with specific new qualities. When trying to follow the design intention or concept, the materials/methods and prototypes themselves and when we are “bending” [4] them, can produce qualities not known beforehand. In our line of practice-based work for this to happen, the production needs to reach a certain state of complexity, to be able for all of the possible parameters to be explored thoroughly and have an impact on the details as well as the whole. Therefore a production environment itself is crucial to facilitate practice based research in the area of art and design, where one can game the rules of play [9], otherwise there is a risk that art/design projects are just illustrations of technology, and technology driven projects unreflected use conventions and qualities from art. In the three projects we specifically address this in the visual styles we tried to develop. At present, one of the most important parameters to benchmark an experience in VR is “realism”. But realism even technologically advanced is still a cultural construction [12]. And realism in relation to digital simulated ones, is not realism, it is reality seen by the camera lens, as Allen points out “The intention of all technological systems developed since the beginning of the 1950s has been towards reducing the spectators sense of their real world, and replacing it with a fully believable artificial one” [1]. And even if realism would be possible in a perfect replica, it is still a blunt showcase of technological frameworks and not an independent aesthetic quality. If one really wanted to depict physical realism in its original form, then a theoretically infinitely level of detail would be necessary, which not only seems technically impossible, but in particular also not desirable from an artistic perspective. So when designing our environments in the first we have deliberately decided to work in favor of this non realistic approach, to find a level that is believable rather then realistic. For example we used the same light and shaders to develop a common style that connected the seven disparate worlds from our first project with each other, to find a reasonable level to avoid the uncanny valley—a mismatch or break in presence where believability does not clash with falsehood [16]. And it is also important to avoid that, “the virtual environment becomes less immersive since they lose the interest and engagement of the user envision the reconciliation of immersion and interactivity” [15]. During our iterative design process, a visual aesthetic gradually emerged that portrayed the spatial situation in a very minimalistic, reduced yet consistent way. In the context of the “hunt for realism”, we constructed just the opposite: a world that deviates completely from this ideal and only shows details where they are indispensable. And that’s why we assume these environments worked so well in our use case: because we found that consistent and credible design is not tied to photorealism–although this hypothesis needs empirical foundation in our future user tests.

7 Future Work

Currently we are actively working on Motopia—an environment designed by Geoffrey Jellicoe in the 1960ies [8]—to enable this space for the VR-bike as well as for a VR-car simulation. Since this conceptual environment was once meant to be the city of motor-mobility we are interested if the 3D bike is able to tap these potentials. For the Biketopia installation we are currently thinking about letting the user take pictures from within the virtual environment and only present those pictures to the audience (bystanders, voyeurs etc.). This could improve conversation between the user and the bystanders in such a way, that they want to ask more questions and maybe want to experience the world itself. In addition, the pictures might help the authors to get an idea of what the actual users think might be the most interesting parts of their journey. We then can evaluate this and improve our work based on this feedback. For the balloon flight we implemented an Immersit Shaker System, which simulated the vibrations of a balloon’s basket and ensured direct haptic feedback for pulling the drawstring. We expect that the repetitive vibration that occurs when the (virtual) expansion joints of the roadway are passed once again clearly emphasizes the spatial impression as the underlying concept of Motopia as a modern era functional city. In regards to our observations on audience participation, the next step will be to monitor visitor-behaviour via top-down camera to generate heat-maps in regards to where they are while they are in the balloon’s “sphere of activity” We will continue to further investigate how space in different media is represented via media technology and how it is used.