Keywords

These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

1 Introduction

This chapter addresses the emerging trend favoring socially immersive artworks via mobile Augmented Reality over sensorially immersive AR that was based on historical Virtual Reality ideals. It will first define the various types of immersion being discussed and make distinctions in how the same term has taken on a new connotation with modern information technology. Then we’ll explore socially immersive artwork by looking at both the empowering head-mounted technologies as well as some early examples using Google Glass. This will lead to a discussion on collaborative locative media by investigating enabling toolkits and tracing its roots back from Land Art of the 60s through early community-created gaming platforms to massively multiplayer online games and ultimately to the current phenomenon of collaborative “Field Art” taking place in the AR game Ingress. Then we’ll conclude with a better understanding on how artists are exploring mobile Augmented Reality within an internet native culture focused on connecting people with each other through information sharing.

2 Definitions/Distinctions

2.1 Sensorial Immersion

The goal of immersion within Virtual Reality has been to surround participants with visual (and sometimes auditory, tactile or other) stimuli to evoke a sense of displacement whereby the physical world is supplanted by an alternate reality. Sensorial immersion can be applied to Augmented Reality in a manner that bases this reality on the nearby physical world, but adds virtual objects into that world. Technically speaking, Tobias Höllerer (2004) defines an AR system as “one that combines real and computer generated information in a real environment, interactively and in real-time and aligns virtual objects with physical ones” (Höllerer and Feiner 2004). This form of mobile AR has been typified in science-fiction film and literature with examples such as Princess Leia’s holographic avatar, William Gibson’s AR tableau of River Phoenix in Spook Country and the user interfaces in Minority Report and Iron Man, to name a few. AR interfaces would know your physical position and head orientation so that an onboard microcomputer could determine how to correctly “insert” objects into your visual perception in such a manner as to convince you that they are as real as the physical objects they are next to.

2.2 Social Immersion

Margolis et al. (2012) expanded upon Jonathan Steuer’s (1992) definition of Virtual Reality (Steuer 1992) to describe an emerging form of Social Immersion used in mobile AR as “the social richness of a mediated environment as defined by its interactive features, that is, the way in which an environment layers networked information via real and virtual means” (Margolis et al. 2012). This form of media is not simply virtual object overlays, but of stories that connect these objects to the people and places that imbue their meaning. This new view of immersion enables us to describe participatory practices within contemporary culture that use mobile social media networking applications which give us an unprecedented ability to share information with social groups and the world instantaneously. For example, Wikitude streams information about nearby events, tweets, Wikipedia articles and user reviews into the live camera view on smart phones. Also, transmedia storytelling creates a sense of social immersion by unfolding elements of a fictional story across different platforms to create an interactive narrative over time and space.

3 Socially Immersive Art

3.1 Head Mounted AR (Wearable Smartphones)

Head Mounted Displays (HMDs) have been used as a means of displaying Virtual Reality content for decades. These systems present 3D computer graphics by placing small screens very close to the eyes. There are several devices that have come to market over the last few years such as Epson’s Moverio, NEC’s Tele Scouter and the Oculus Rift which can be described using the sensorial form of immersion for AR since they overlay graphics onto the real world through see-through optics or by rendering CG over live video to opaque displays. It’s important to understand these devices and the affordances they provide in order to explore what differentiates them from new platforms that are now forthcoming which are better categorized as providing social immersion for AR.

While technically the Oculus Rift is a VR HMD, given the fact that there are other vendors (Ovrvision 2014) offering camera modification attachments to enable real-time video overlays within the Rift display, it is worthwhile to compare and evaluate the Rift as a viable head mounted AR device. The latest version of the Rift dubbed Crystal Cove incorporates markers on the outside of the Rift along with an external camera that enables head tracking so that as users move their heads in the real world the virtual environment responds accordingly. This is a key feature for designing an AR system since the computer needs to know exactly where to position virtual objects on top of the live video stream as users move around. The AR-Rift (AR-Rift 2014) is also particularly interesting as it enables users to create virtual desktops within AR to create entirely new ways to work with standard applications.

As one of the most commercially successful HMD manufacturers, Oculus Rift raised ten times more than the original $250,000 Kickstarter campaign goal. It is therefore important to understand how they are focusing their business interests on the gaming sector as evidenced by hiring several of the most influential game developers such as Epic’s John Carmack and EA’s David DeMartini as key company strategists. This will likely make the device much more powerful and user-friendly than if it were intended primarily for scientific or military use like so many other VR headsets. It’s then worth considering how game consoles have become increasingly social as they have become inherently networked. Both the PS4 and Xbox One have added built-in DVR systems to enable gameplay recording and playback as well as enabling sharing those videos to social media sites like Facebook. Perhaps even more importantly, Sony has added broadcast capabilities to the PS4 so that users can stream their live gameplay (with live picture-in-picture camera view of the player) to sites like twitch.tv and ustream.tv with the push of a button on the controller. Some of these home broadcasts reach tens of thousands of viewers and in the first 6 weeks since launch, PS4 users have already broadcast 1.7 million streams generating more than 55 million minutes while being watched by 22 million sessions (Twitch: PS4 streaming 2014). These broadcasts are accompanied by chat windows to enable discussion both within the virtual community as well as with the original player. It’s then very likely that these highly established game consoles will influence the Rift and so we can consider how these emerging video sharing platforms may be integrated with a HMD like the Rift.

Google Glass on the other hand is a new device that might be best described within AR as a heads-up display (HUD) rather than a HMD like the Rift. HUDs were originally designed starting around WW II for military use to overlay navigational information onto aircraft windshields or helmets, but are now commonplace within most gaming user interfaces to display information such as health and inventory as well as chat boxes and mini-maps to collaborate with teammates. Google Glass works in a similar way by showing various types of information in the upper-right section of your field of view. It can provide you with directions, look up facts, translate your voice or physical signage and of course help connect you with others. One of Glass’s most powerful features is the ability to live-stream video from your own point of view anywhere and anytime. It’s similar to a Google Hangout, but it’s completely untethered. These features quite distinctly differentiate Glass from other forms of AR by creating what we can describe as Social Immersion rather than Sensorial Immersion like the Rift.

3.2 Socially Immersive Artworks

Google Glass art is like the new reality TV – Glass art through livestreaming ties back to artist and Internet pioneer Josh Harris and his various ‘90s projects like “Quiet: We Live in Public” which surrounded 100 secluded artists with webcams and broadcast their every move onto the Internet. Now Glass ups the ante by enabling completely untethered high quality directorial power to broadcast your own life or the lives of those you think the public may covet.

Helpouts (Helpouts by Google) were released in November 2013 in order to create a public marketplace for purported experts to provide real-time video chats for a fee. They are in essence an extension of Google Hangouts which have been continuously evolving to provide better quality multi-user video chat rooms with included screen sharing and embedded text chats. While the marketplace is only a couple months old, there are already thousands of advertised services listed there grouped into the following categories: Art & Music, Computers & Electronics, Cooking, Education & Careers, Fashion & Beauty, Fitness & Nutrition, Health and finally Home & Garden. The specific services being offered tend to range from pre-made lessons (e.g. over 50 people offering Helpouts to Learn to Play Guitar) to more personal problem-solving sessions (e.g. over 70 people offering Helpouts to solve peoples’ Computer Issues). These live video sessions are particularly well suited for use with Glass in that many of them require the instructor to be hands free in order to help execute the Helpout.

New York artist Molly Crabapple made a drawing of porn star and circus performer Stoya while wearing a pair of hacked Glasses in a project entitled Glass Gaze (Artist gazes at porn performer 2013). The Glass enabled Crabapple to live-stream what she saw and thereby “to take the most classical, thousand-year-old way of looking: an artist looking at a model. It’s the most undistracted, direct, unmediated thing. And then run it through this super-mediated, captured, commodified way of seeing that Glass represents.” Crabapple reports that wearing Google Glass while drawing made it tempting to constantly look at the image in the corner of her vision of what was being recorded. Crabapple describes the uncanny drawing process by stating “Glass is all about distraction, and it’s the ultimate in unphysical – it’s light on a piece of clear plastic in front of your eye.”

New York city curator Samantha Katz created Gallery Glass in September 2013 as a YouTube channel (Gallery Glass – YouTube) to highlight Brooklyn artists by filming 30 artists over 30 days while wearing Google Glass (Bushwick Artist Uses ‘Google Glass’ 2013). Although the videos posted to YouTube are still edited and generally feel like they could have been filmed with a handheld camera, as shown in Fig. 8.1, Katz argues that Glass can give viewers a first person perspective that presents “what it’s like to walk in a studio.” At times Katz hands her Google Glasses over to the artists as they draw or paint, but unfortunately the videos have been edited so that the audio from the interviews are overlaid onto the visuals of the painting and drawing. While this is a standard editing technique for cinematic film, in the case here for Gallery Glass it tends to disrupt the attempt at experiencing the creation of an artwork from the artists’ perspective. Katz admits the project was a “beta” test and is interested in continuing to explore the potential use of Glass for learning about art.

Fig. 8.1
figure 1

First person perspective via Google Glass from an artist Katz interviewed as she created a new drawing

Full size image

It could be argued that David Datuna’s Viewpoint of Billions is perhaps one of the only works of art to date that is truly unique to Google Glass (David Datuna 2014). Datuna worked with mobile app developers BrickSimple to create an interactive installation that at first glance represents an American flag – see Fig. 8.2. However, as one approaches the artwork, you’ll see the surface is covered with found eyeglass lenses. Beneath the lenses is a collage of photographs and newspaper clippings as well as hidden cameras and microcomputers. The cameras can record visitors as they experience the artwork and then live stream that video back to those same visitors wearing the Google Glasses. This creates a sort of feedback mechanism in which the artwork appears to be watching you back as you watch it. While other video artists like Steina and Woody Vasulka pioneered using video feedback, Datuna’s use of Glass to transmit a live video of one’s self via a camera hidden directly within the artwork perhaps takes this concept to a new level.

Fig. 8.2
figure 2

Viewpoint of billions by New York artist David Datuna at Art Basel Miami (Datuna Glassfeed)

Full size image

4 Collaborative Locative Media

4.1 Handheld AR Platforms

In order to understand how mobile AR is evolving into a socially immersive medium, we will first present two exemplars of popular mobile Augmented Reality applications for smart phones and tablets. Both of these began with a traditional attempt at fulfilling the original AR goal of providing a platform for developers to create sensorially immersive AR experiences. However, they both continue to evolve to enable authors of mobile AR experiences to produce content in an increasingly more accessible manner as well as provide more and more hooks into social media platforms for quickly sharing content and experiences with friends, colleagues and collaborators.

Layar is a good example of a popular consumer mobile AR browser that allows developers to geotag a location with images, text, sound or other media (Layar). When a user is exploring the physical world through their phone or tablet, they are able to view nearby AR content overlaid on top of a real-time video feed via the device’s camera. Layar enables developers to encourage end-users to easily share their AR experiences with friends on various social media platforms such as Facebook and Twitter.

Over time, Layar has changed its business model as to how to best support mobile AR technologies. They have added computer vision based scanning to their toolkit of sensing technologies. So instead of finding content solely based on your physical location, you can now point your phone or tablet’s camera at an image target. Historically, AR marker targets tended to be black and white geometric patterns that resembled 2D bar charts. Nowadays, Layar and many other computer vision manufacturers use markerless targets that can be defined by any natural image as long as it adheres to a few basic technical requirements such as non-repeating patterns, no motion blur, avoiding reflections and transparencies and has generally flat illumination. This new sensing platform enabled Layar and others to support AR experiences within interior spaces where GPS satellites would not work reliably. However, from a demographics perspective, this new feature provides for countless new advertising and marketing opportunities as now companies can integrate mobile AR with their brand experience much easier.

Qualcomm’s AR Software Development Kit, Vuforia, enables Android and iPhone developers to incorporate sophisticated natural image based tracking into their mobile AR applications (Qualcomm Vuforia). Vuforia also requires developers to specify an image target that users can then point their mobile phone or tablet towards to view a real-time augment. However, Vuforia also adds extended tracking to enable users to have continuous visual experiences even when the tracked target is outside of the camera view. While this might seem to be a trivial technicality, it provides for a much more seamless and immersive AR experience since now you aren’t required to keep your mobile device pointed at a particular physical object, but instead one can simply use their phone or tablet to explore the entire surrounding environment as well.

It’s important to understand that Layar and Vuforia have taken different approaches to mobile AR development and publishing. As an AR browser, Layar requires that end users install a single official Layar application onto their smart device. Once the user has installed this app, they have access to all of the AR “channels” that developers have published using the Layar platform. Users can either scan an image target or search through keywords and categories to find results of what AR content might exist near them. Vuforia on the other hand is meant to be packaged as a custom application for each AR experience. This means that end users must download and install a different app for every application built using Vuforia. However, it also provides developers with much more control over the application and enables apps to utilize all available Android or iPhone services and libraries to build a highly rich and engaging mobile experience.

4.2 Handheld AR Media

These new socially based immersive media platforms provide affordances for new forms of locative artworks that were not possible before. Here we will discuss how geo-locative artworks and experiences have been emerging and where there may be precedents of artists and gamers working collaboratively within large scale environments (both physical and virtual) that perhaps have laid the cultural landscape for artists today to explore these new socially immersive AR collaborations.

In the late 1960s Joseph Beuys developed a theory of Social Sculpture that describes a society in which “Everyone is an artist”. He illustrates the idea that art is meant to be participatory and can hold the power to transform society. Like Richard Wagner, Beuys strongly argued for a Gesamtkunstwerk in which society as a whole was to be regarded as one great work of art. This helped to lay the foundation for Land Art in the late ‘60s where the landscape and the work of art were inextricably linked as seen in Fig. 8.3 documenting Spiral Jetty. Much like contemporary AR artwork, land art was meant as a protest against the traditional museum and gallery art worlds in order to bring power back to the community.

Fig. 8.3
figure 3

Spiral Jetty by Robert Smithson (1970). 1,500 ft in Utah, US

Full size image

Perhaps inspired by the well-known artists who have been transforming the landscape of our planet for the last 50 years, other lesser known artists have been transforming the landscapes of virtual environments for the last decade. Perhaps most famous of these online virtual worlds is Second Life (SL) which has had over 20 million subscribers. Many SL users pride themselves on the creative design of space and objects in SL and collaborate in groups on large-scale installations. Some of these installations mimic spaces from the real world such as in Becoming Dragon (Cárdenas et al. 2009) while others create psychedelic art installations like StormEye (Story and Enfield 2009) out of giant moving shapes coated in video and sound. Artists build these virtual installations and then share them with other citizens of Second Life to experience.

However, this idea of locative art within a virtual world is not new to online platforms. Gamers have been designing their own levels for decades. Lode Runner from 1983 was one of the earliest games to come with a level editor that enabled level designers to create their own custom environments for playing within. The website worldofleveldesign.com is built exclusively for this community to aid sharing articles and tutorials for environment artists and level designers to help with all aspects of working on level designs and 2d/3d environments. Also, LEVEL-DESIGN.org has a reference database of tagged image screenshots to help share ideas on level design.

Now, there are countless massively multiplayer online (MMO) games where large groups of users collaborate together in teams and factions to move through the game story. In addition to solving the challenges presented by the original game designers, players often choose to create their own virtual art installations within the game world. LittleBigPlanet is an example launched in 2008 on the Playstation 3 that focused primarily on user-generated content as evidenced by their tagline “Play, Create, Share” and over 1 million user-created levels to play.

Ingress is a transmedia game by Google played out through live events, the web and mobile applications. The players’ goal is to capture “portals” which are co-located with objects in the real world such as public art sculptures, landmarks and libraries. Once captured, three or more portals can be linked together to create fields that give users points. There are two factions players must choose between, the enlightened and the resistance, which battle each other to control these portals and fields. As of January 2014, over one million people are walking through cities around the planet capturing portals and linking fields.

There are no pre-defined rules on what the shape of these portal fields should look like and most of them are abstract shapes designed to maximize the amount of underlying land controlled with no consideration given to aesthetics. However, having reviewed the history of locative art in Social Sculpture, computer gaming and science fiction, we might find it inevitable that players have begun to create specific designs within these augmented reality platforms so that they link portals together in a strategic manner as to form a representational image across actual cities. This so called Field Art may connect dozens of points spanning entire cities. Given that every portal and field is in constant danger of being attacked, it is easy for one to appreciate the transient complexity of these undertakings and the need for large groups of people to help create them. We might perhaps expect to find that several Field Art installations utilize collaborations between factions to create even more complex designs. In the screenshots shown below in Fig. 8.4, you can identify the two factions contributions by looking at the two colors on the maps.

Fig. 8.4
figure 4

Field art created with Ingress. (a) Flower in Germany. (b) Christmas tree in London is over two miles long (Monnington 2013). (c) Butterfly in Heidelberg, Germany (RNK Field Art). (d) Woodpecker in Germany (Stadler 2013)

Full size image

In addition to aesthetic designs created within Ingress, players have used the game platform for political and social commentary. After the Boston marathon bombing, MIT Ingress players from both opposing factions agreed to a temporary ceasefire on the MIT campus as well as erected a dual owned virtual memorial for the slain MIT police officer Sean Collier.

5 Conclusion

We are beginning to experience mobile Augmented Reality in a new context within a culture that has come to expect (and rely upon) persistent connections between people and information. Naturally then we architect systems to better connect with each other through the Internet. These new platforms are enabling us to create new worlds that bring us together through our common interests. So whether it is learning how to paint nudes with 100 strangers via Google Glass Helpouts or touring CERN via Glass or creating virtual landscape art installations on top of and in the middle of New York City, these new socially immersive media are changing the way we can create, share and play.