Keywords

1 Introduction

Mass adoption of applications such as Pokémon Go and SnapChat filters has proven there is a considerable interest in the use of AR. Despite the fact that AR technology has been present in different formats for a long time (Azuma et al. 2001; Rogers et al. 2002), it is still considered a novelty. There is, to date, a significant lack of understanding to what extent AR is actually being used outside of the academic lab and how the real world environment influences such use.

AR by definition functions differently than other interactive technologies in terms of its “reliance” on the physical world. It is based on the fusion of the physical environment with the virtual to a much larger extent than is the case for more traditional interactive technologies, such as the established mobile applications, social media platforms or emails. Given that the physical surroundings are much more important for AR experience, the contexts where AR is used—for example the positioning of markers or placement of AR screens—greatly impact the nature and quality of experience.

This study aims to provide a more holistic overview of the components that need to be taken into account when designing a user journey with AR that takes place in public. For that purpose, this paper indicates which factors precede the experience which are central to the experience and what the potential outcomes are. It draws on findings from media studies, human-computer interaction, marketing and computer science, underlying the necessity of an interdisciplinary approach to the study of AR. Furthermore, we shortly discuss two recent placements of AR in public context and illustrate the framework’s applications through those examples, along with the research questions that could serve as guidance for further empirical studies of AR in public.

2 Background

Studies of consumer and user experience with other interactive technologies (internet, social media, mobile phones, virtual reality, wearables) focused predominantly on factors such as: technology affordances related to the user interface, user demographics and psychographics, different types of engagement such as social, affective or cognitive and, evidently, behavioural in terms of purchases, performance and reuse.

This rich body of research related to these technologies has for instance proven that affordances like interactivity or modality play a major role in online consumer experience (Sundar et al. 2015) and that they have further impact on variables such as website attitude, e-retailer trust and purchase intentions (Dennis et al. 2009; van Noort et al. 2012). Moreover, perceptions of characteristics such as ease-of-use, usefulness, visibility and privacy further impact consumer willingness to use websites, mobiles and wearables (Rohm et al. 2012; Lunney et al. 2016; Chuah et al. 2016; Pagani and Malacarne 2017), along with consumer characteristics like demographics, tech savviness or personal traits (Dennis et al. 2009). The nature of engagement needs to be observed both on an individual and social level (Pagani and Mirabello 2011), in relation to different values such as enjoyment, utility and community participation (Calder et al. 2009).

Many of these aspects remain relevant for studying AR. Rauschnabel et al. (2015) for example showed that consumer characteristic such as introvertism vs. extrovertism significantly impact willingness to adopt wearable AR. Scholz and Smith (2016) emphasize the relevance of engagement between users and brands and the support of meaningful content when creating immersive AR experience.

However, additional insights are required. Some digital technologies—mobile, social media, wearables—became ubiquitous in terms of their use in the sense that individuals would be accustomed to constantly interacting with them, for instance just before going to bed, when walking down the street, when they are spending time with their friends and so forth (Cecchinato et al. 2014). On the other hand, numerous AR apps or set-ups of AR experiences have until now not been perceived as a technology that one would feel like using all the time or anywhere, but more like for specific tasks and in specific environments (Rehrl et al. 2014). That has partially to do with some technical limitations—for example, holding a smartphone in the air to keep seeing the overlaid information is not an intuitive use of technology that one would engage in for an infinite amount of time, but in some contexts it can appear both useful and hedonic. Wearable AR is facing other issues, too. Wearing HoloLens for a longer period can prove to be challenging because of its weight and also because of the users’ concerns how it might affect their appearance (Rauschnabel and Ro 2016). Furthermore, it can potentially still lead to some motion sickness, even thought the issues are not as severe as, for instance, with VR headsets (Hern 2017).

An interesting exception to this was of course Pokémon Go where people were willing to play an AR game when walking down the street or when in an office. However, safety and social issues arose around it, along with its decreased popularity. Google Glass was designed with the intention that people would continuously wear it, but there was little evidence that users were comfortable with constant virtual overlay, as it could quickly appear intrusive, besides the issues related to the social acceptance of technology.

While cases of AR apps that are not related to specific context might (continue to) rise, it is currently equally or more relevant to discuss how to contextualize AR experience, set up in public spaces.

3 Dimensions of User Experience with AR

User experience with AR thus requires revisiting some of the established factors and rethinking them in the context of AR, but it also calls for understanding some of the factors that have not been focused on to such an extent beforehand.

As indicated above, more than around its ubiquitous deployment, a deeper appreciation is required around the specific contexts in which AR can enhance the experience in a meaningful way (Scholz and Smith 2016), also referred to as situatedness (Javornik et al. 2017). Due to the nature of how it functions (i.e. its ability to enhance physical contexts), AR works very much in conjunction with the physical environment. In public places such situatedness proves even more complex because the external influences can be unpredictable and thus may interfere with the experience. For example, if in an art gallery the overlaid information appears based on markers of the painting, the constant flow of people passing by might interrupt the augmentation. Or if a virtual mirror is situated opposite a strong light, this can interfere with a user’s reflection in the mirror and consequently with the try-on. As observed with Pokémon Go and with other AR apps destined for outdoor use, physical context can also lead to safety issues—if a user gets so immersed in the screen that he forgets about his surrounding, that can jeopardize his safety. On the other hand, the physical context can very well play into the experience. If appropriately integrated, the situatedness would enhance the experience.

Some previous studies have looked at the user side. Rauschnabel et al. (2015) for instance showed the impact of psychological characteristics such as introversion and extroversion on a user’s propensity to adopt AR wearables. Technological savviness can also be expected to have multileveled influence on the quality of the experience. On one hand technologically savvy users will be more willing to experiment and will also be capable of having a fuller exploration of the technology, but that affinity will be probably coupled with higher expectations. However, technologically less savvy users might express more admiration and find themselves more in awe of different formats of augmentation, resulting in a higher level of surprise. With public AR mirrors, moreover, the person would see herself virtually enhanced in a mirror and if the people passing by could see that reflection, the social embarrassment can represent an obstacle in such situations (Wouters et al. 2016), which could have different effects based depending on the level of shyness.

Moreover, more needs to be understood around the interface features and the related affordances that are suitable for AR. Interactivity was defined as one of the key characteristics of the internet (Song and Zinkhan 2008). Virtuality or vividness is seen as central for creating immersion into virtual worlds (Jennett et al. 2008). Because AR appears in a camera-view, other features—interactivity, virtuality etc.—need to adapt to that in order to not take over the screen and obstruct the view on which virtual elements are overlaid.

Also, the rules for designing the content for AR mode (camera view with overlaid content) differ substantially from designing the content for websites or mobile applications. What combination of text and image might be most suitable for the users to engage the most with the content? We might expect that viewing the augmented content might make the users become progressively more used to seeing content overlaid on museum artefacts or when exploring a new area while travelling. Would they adopt the virtual content as part of their viewing and, as a consequence, feel impoverished when not having access to it in certain situations?

Overlaying of the physical environment with virtual annotations has been referred to as augmentation. This can entail augmentation of the products (for instance overlaying a Lego box with an image of how the Lego construction will look like when built) or the person herself. Research so far has shown that consumer enjoy such visual simulation and that they also find it useful when shopping (Javornik et al. 2016), discovering a touristic or cultural site (Kourounthanassis et al. 2015; Leue et al. 2015) and learning (Chang et al. 2014). The perceived value of such augmentation can be related to the quality of virtual augmentation—for example, a suitable alignment of the physical and the virtual, the quality of the content or the system itself (Jung et al. 2015). That can be especially important for certain types of AR apps like virtual try-on, while Pokémon Go was rather based on an overlay instead of on perfect alignment. With the increased level of quality of emerging AR apps, the expectations of users with regards to such alignment might increase.

Designing AR experience in a store or in a museum for people to take part thus always requires taking into account the elements that will signal to the user that they can step into an AR experience. What will be used as triggers? If there is a virtual try-on mirror situated in a store, how will the shoppers understand that the mirror will augment their reflected image? What is required for the set up of an intuitive experience with a mirror, be it in a museum or in a store? Is it more suitable to allow for an element of surprise when a person suddenly sees herself in the mirror with a virtual hat on, or should the person first approach the mirror and initiate the overlay by a movement or by tapping on the screen? If so, how can it be ensured that many people would not just walk by? Is it a role of store assistants or museum curators to point the technology to the visitors? If the human interaction is a crucial element, then further research is needed about how it can be best integrated in the experience. Would it rather be the role of advertisement and PR to raise the profile of stores or spaces where AR installation can be sought out? If so, how to combine the promotion through additional advertising channels with this novel technology? Some screens are so huge that they immediately attract the crowd because they simply can’t be overlooked, but the cost associated with such set-up is not compatible with the wide public deployment.

As already underlined by Scholz and Smith (2016), engagement with AR can take different forms. The users can engage with content, other users and brands or institutions behind the AR. Qualitative research for example should unveil what meanings users attach to these experiences. Does AR enhance cognitive engagement and facilitate decision-making? In what way do AR set-ups allow users to perform better in specific tasks or lead to behavioural change by visualizing certain content in a manner that it makes a difference? Moreover, interacting with installations or technologies in public spaces often caries a social component– such as the honeypot effect where those passing by are drawn to observe or to take part in the experience, but also elements of social embarrassment, feeling uncomfortable about knowing that others are observing you. The social component in AR experiences needs to be explored further.

Research shows that the younger generation no longer distinguishes between digital and physical content. If AR use shall continue to rise, perhaps the generation post-Y might no longer draw clear distinctions between virtual when overlaid on the surrounding and the physical surroundings. But currently, it is more crucial to investigate what is the perception of such augmentation and what are the potential consequences. In our recent study, we for instance show that such visualization can facilitate an artistic process and allow users to view themselves as opera characters (Javornik et al. 2017). To which extent will the augmentation increase imagination and creativity and to which extent will it actually have the opposite effects? In what ways can AR empower human activities without making the users relying too much on its visualisation techniques? (Fig. 1).

Fig. 1
figure 1

Framework of AR experience in public (adapted from Javornik et al. 2017)

Full size image

The impact on the subsequent behaviour is one of the most under-investigated areas in AR and user behaviour. Once the episode of interaction is completed, it is crucial to understand what difference it made in the long run. Did the AR app increase the amount the users were willing to spend on a purchase or did it convince them into buying an item? Chang et al. (2014) demonstrate direct impact of AR on learning and art appreciation when visiting a gallery. More such studies are required to unveil if AR installations can truly make a difference in for instance how visitors relate to cultural events in the long run and if AR can actually modify how we perceive our skills or if it can impact certain attitudes and beliefs that would lead to positive behavioural change. In the following two cases, two AR public installations are shortly discussed in the view of this framework.

4 Charlotte Tilbury’s Magic Mirror

Virtual try-ons are proving to be increasingly more popular additions to retail environments. Make-up artist Charlotte Tilbury set up one such “Magic Mirror” in her flagship store in Westfield that allowed the visitors to try on ten of the artist’s signature looks (Arthur 2016). Charlotte Tilbury is positioned as an up-market brand that offers high quality make-up and the retail stores reflect the luxury brand image.

As seen in the Fig. 2, the virtual try-on mirror was situated in a manner that it appeared as a part of the store. Such integration in terms of style and aesthetics allowed for a wholesome retail experience for visitors, as the mirror was embedded as one leg of the retail journey. Furthermore, looking at oneself in the mirror with make-up is an activity that does not need to be learnt, as it’s a rather intuitive reaction for shoppers to do—which allowed for appropriate situatedness and interaction triggers.

Fig. 2
figure 2

(Copyright Holition Ltd)

Magic Mirror in Charlotte Tilbury store in Westfield (London).

Full size image

The fact that the visitor could only try the ten predefined looks poses an interesting question about the level of interactivity and personalization and branding strategy. This allowed the brand to visualize the combination of products of their own choice and retained control over the displayed images. Such a set up can represent a certain advantage, as it prevents the shoppers from designing looks that the make-up artist would not perceive to be of sufficient quality or appropriate appeal. Furthermore, the ten signature looks are associated with celebrities and represent an important asset to the brand image, therefore the mirror permits the brand to raise profile and increase the awareness and knowledge with the shoppers. Finally, the signature looks—The Ingénue, The Golden Goddess, The Uptown Girl, The Rebel, The Glamour Muse, The Vintage Vamp, The Bombshell, The Dolce Vita, The Rock Chic and The Sophisticate—add elements of storytelling to the virtual try-on.

Moreover, such a set-up simplifies the interaction, as the mirror interface does not require any other features in addition to the buttons for the ten looks. On the other hand, the set up limits further interactions with the mirror in terms of personalizing the looks and trying on a palette of products, as is usually the case for virtual mirrors.

More investigations would require insights into the following questions: what role does such particular set-up of the mirror play in the decision-making process? In what manner does the mirror change consumer experience in the store? What impact does it have on product attitude, especially given the fact that the products were presented as parts of a signature look? Does the mirror enhance interactions with shop assistants or interactions among the shoppers themselves?

Semi-public space such as retail make such set ups less challenging in terms of controlling the external influence and contextual factors. Similar was not the case for the Christmas campaign set up by Blippar, designed for a public site of Covent Garden.

Fig. 3
figure 3

(Copyright Holition Ltd)

The shoppers and store assistant in front of the Charlotte Tilbury Magic Mirror

Full size image

5 Blippar Christmas Covent Garden Campaign

In the period leading up to Christmas 2016, Blippar launched an app that would augment the visit of Covent Garden. Designed as a gaming application, its purpose was to engage the visitors into a game of collecting eight different reindeer (Fig. 4). When all the reindeer were collected, the user could enter a prize draw to win £200 voucher for dining at a popular Covent Garden restaurant. Coupled with the prize were also other promotional offers that one could benefit from when scanning the tag “Exclusive rewards”, related to specific stores at Covent Garden.

Besides the game, there were also other features, such as visualization of the reindeer Rudolph flying towards the sky and sparkling, as well as an appearance of The Northern Lights (Fig. 4, Right). In order to access all this content, the visitor had to download the Blippar app and “blip” (i.e. scan) the markers that were organised either as labels on windows of stores or cafés or at some other prominent places. As the notion of “blipping” might not have been familiar to the visitors, additional instructions were required in such instance. The experience designers solved this issue by printing the instructions in large font next to the blipping images (Fig. 3 and 4).

Fig. 4
figure 4

Cardboard with Rudolph that was augmented once it was “blipped”

Full size image

The content of this particular campaign was aligned with the time and the place—Covent Garden being a popular destination by many during Christmas period. Seeing Rudolph in the sky with Aurelia Borealis enhanced the magic of the visit at that time of the year. Furthermore, the central Covent Garden area was closed for traffic therefore the visitors had the chance to stop in front of different points without jeopardizing their safety. Also, the site is a popular destination among families, thus the gaming element resonated well with children.

Some of the related challenges of such a set up have to do with the fact that “blipping” is not an intuitive experience (i.e. not something many visitors are used to doing) and therefore a certain amount of explaining is required for the visitors to take part in the experience. A user study would be relevant to explore to which extent the visitors perceived the experience to be easy and intuitive and if the app had an impact on the character of the Covent Garden visit experience. Were these perceptions different across demographic groups and did they differ depending on the user being on her own or with friends and family?

Designing such an experience in a public space (and not semi-public) carries numerous challenges. For example, the labels for blipping should be easily accessible, but should at the same time not create awkward social situations, where others visitors might mistake the “blipping” for someone taking a photo of them while they are enjoying their coffee (Fig. 5). The blipping should not obstruct other people’s activities, which is why the alignment of the digital/virtual and the physical requires such attention with AR experience in public.

Fig. 5
figure 5

One of the labels on the window for the visitors to “blip” and collect reindeer

Full size image

Also, further research would be required to understand the perceived appeal of the interface design. Which were the successful solutions of the user interaction and which points created a bottleneck, if any? Moreover, did the gaming elements in some way enhance the visits, both on social and/or individual level? And, most importantly from the commercial point of view, to which extent did the exclusive rewards offer drive the shopping and purchase behaviour? Finally, did the user acquire interest and desire for such a “blipping” experience also in other contexts—if so, which ones and for what purpose?

6 Conclusion

This paper outlines the main dimensions of AR experience with a special emphasis on public interactions. By doing so, it contributes to the field by pointing out the relevance of elements such as situatedness, design of interface features for the AR mode, augmentation and some others. The placement of AR in the physical surrounding and the manner in which virtual overlay relates to the physical context carries high importance for the quality of AR experience. The adoption of AR is starting to come of age and such holistic approaches can help the designers, researchers, marketers and others to seize the potential or AR and deploy it in efficient and appropriate manners.