Keywords

These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

1 Introduction

Mobile devices have been rapidly growing in power, capabilities and features. As a result, opportunities for designers and engineers to create exciting new experiences and applications are emerging. However, while the potential for innovative service design is great, there are also challenges when it comes to designing effective, useful and usable services that offer users long-term value. Challenges derive from smaller screen sizes that offer reduced real estate for content presentation and interactive elements. Challenges also arise from the complexity inherent in most usage contexts, and the fact that users are more often than not actually in motion—that is, moving through different contexts and set-tings [1]. Acknowledgement of form-factor, interaction and context-of-use challenges has led to the establishment of a new field of study called Mobile Human–Computer Interaction (MobileHCI) or Mobile Interaction Design [2]. The MobileHCI research field started emerging in the late 1990s with the advent of the second generation of cell-phones available to mainstream users [3]. Initially focused on how users inter-acted with hardware (e.g., keypads) and on designing usable interfaces (e.g., UI for the limited interaction modalities and small screens) [4], the field has evolved rapidly to encompass consideration of use context through reports of field deployments, the design of development environments, and the creation of innovation methodologies. Mobile HCI as a field is the fastest growing in the broader human–computer interaction (HCI) community.

Our recent work in this space has focused on mobile augmented reality (“MAR”). Augmented reality offers new affordances for interaction; it has potential to enhance users’ experiences through provision of digital information relevant to real world surroundings, without depriving users of their context [5–8]. Through placement of virtual objects or feedback over reality users are provided access to information that is typically not available to them using their own senses [6]. Augmented reality and mixed reality are established areas of investigation [6, 7], but only recently has the emergence of mobile augmented reality been made possible owing to the advent of increasingly powerful smart-phones and smaller, more effective sensors (e.g., GPS, cameras). Mobile augmented reality thus enables users to move around and explore what is around them freely within real world settings [7, 9]. While the promise of MAR has been discussed on the research world since at least 2000 [10], we are only just seeing the promise turning into a reality. MAR applications are becoming more sophisticated, and re-cent research suggests that MAR will be responsible for amassing an enormous amount of profit in advertising, games and applications in general [11].

In this chapter we present an overview of the challenges posed by MAR to designers and researchers and discuss on the benefits and drawbacks of the most used techniques and approaches, from a design and HCI perspective. We discuss the current state of the art and most commonly used methods for the design and evaluation of MAR. In addition, we present our own design methodology, through the description of our approach to designing a MAR application. Our application focuses on allowing users to find whom of their friends are in the local vicinity by displaying icons and avatars overlaid on the current camera-rendered location, showing distance and direction in relation to the user. We utilized three different techniques to prototype our concept: (1) a low-fidelity prototype (i.e., mock-up and Wizard of OZ); (2) a mixed-fidelity prototype using video; and (3) a high-fidelity working prototype used on an actual smart phone. Using these three prototypes, we con-ducted a set of field studies, gathering feedback from users in real life scenarios. We detail the benefits and drawbacks of each approach and the settings, scenarios and phases of the design process (such as during early ideation or later stage evaluation) in which they perform most effectively.

2 Mobile Augmented Reality

Early experiments with MAR required considerable amounts of equipment in their creation [6, 8]. Use testing relied heavily on complex, often unreliable infrastructures, using laptops and heavy equipment or requiring Wi-Fi connectivity in order to function [12, 13]. Such equipment arguably interfered with the user experience and therefore potentially invalidated the user’s assessments of the benefits of such applications.

Things have changed: current smart-phones can easily support this type of experience in real world settings. In addition, utilizing GPS and integrated compasses, the combination of AR and location-enabled and positioning services [5, 14] means that it is now possible provide users with ways to use AR and gain easy access information about their surroundings while on-the-go. Today cumbersome, intrusive equipment [15] is no longer required to create these experiences [16].

Thus far, the majority of literature on MAR has been strongly focused on technical challenges and constraints: issues to do with set-up and management of equipment, issues with quality of image rendering; consideration of marker- or marker-less detection; and development of detailed and reliable location-based services [10, 12, 15, 17]. Location detection and/or image recognition algorithms, information retrieval from different data sources and computational efficiency continue to be the main focus for researchers working on MAR. Recently, however, the research area has been starting to address user experience. This has been driven in large part because of the proliferation of end-user services and commercial applications using MAR (i.e., Layar,Footnote 1 YelpFootnote 2). Research reports increasingly relate users’ experience while interacting with MAR, investigate usability issues and pose design challenges for this technology.

2.1 Design Challenges Posed by Mobile Augmented Reality

While innovations in sensor technologies, developments in interaction and presentation modalities and increasing access to data streams in real-time are inspiring services and applications that were previously only imaginable in futuristic films, these new features and capabilities are not always ready for everyday use [18]. Challenges include:

  • Discoverability—many users, even savvy smart-phone aficionados, are not aware of what services and applications are available.

  • Interpretability—many users are unclear about the value these services and applications offer and most services and applications are unclear in their presentation of value beyond immediate entertainment and “wow” factor.

  • Usability—many users find learning to use the applications challenging and find interaction features and interaction paths cumbersome, and often context of use is not well enough considered.

Usefulness/utility/meaningfulness—once in use, many users do not find these applications offer long-term value. While it is clear there are many opportunities for deeper interaction and experience design engagement for MAR services and applications, there are still few guidelines for would-be designer-developers and few documented accounts of how to design such rich experiences [18]. Although some reports on designing MAR can be found, most refer to highly complex settings and infrastructures, defined for very specific purposes [5, 6, 14, 19] or, as we note above, are strongly focused on the hardware constraints [9, 20]. Studies on design or from a user-centred perspective are scarce [18].

One reason for this scarcity of studies, we purport, is that testing and validating ideas at initial stages of the design process is hard. Assessment of the usability of specific interface or interaction design features yield well to laboratory-based evaluation of static of minimally interactive mock-ups (e.g., is the target area large enough to be selected?, is the font readable and in what lighting conditions?, is the image occluding other content?), but full service ecosystem design, such as those imagined for mobile AR applications do not yield so well to such methods. To be able to offer effective feedback, users need to be able to experience MAR services and applications as they are intended to be experienced: “in the wild” with interactive experiences and data presented in real-time. However, development costs can be high, even if all that is developed is a simple prototype. Further, the field is not mature enough yet, nor are user experiences sufficiently common-place for there to yet exist “discount methods” (e.g., guerilla evaluation sessions, simple heuristics) for creating informed design elaborations; typically in more mature interaction paradigms, these discount methods appear in the form of design guidelines and heuristics wherein are codified typically successful or problematic design options. But to avoid making premature commitments to interaction design choices that require costly back-end infrastructure development which, in the end, may not actually be useful or usable, designers and developers in the emerging field of MAR need grounded design guidelines and methods for evaluating design options.

Our research faces this paradox head on. As is often the case, designers are required to come up with new ways to convey their visions and test their concepts when designing new services [18, 21]. To prototype the envisioned, fully-fledged service is obviously costly and will result in premature, often non-retractable, commitments within the service design; such premature commitments are precisely what iterative design and evaluation are aimed at circumventing. However, to get useful feedback from users, it is necessary to create an experience that is “realistic” or provocative enough of the envisioned scenario of use such that users are able to give meaningful and actionable feedback.

We ask: how do we simulate the experience in a realistic enough way to get feedback early in design on concepts and on interaction designs without spending too much effort on building working prototypes? What are the key features a prototype needs to have to simulate the experience sufficiently for users to be able to give us meaningful feedback?

To begin addressing these questions, in this chapter we offer an introduction to relevant papers that do try to address at least some of the issues above: discoverability, interpretability, usability, usefulness. We describe projects where user-centred approaches, embodied mostly through user studies and participatory design, have been used. We follow this overview of related work with a description of our own experiments with prototyping and evaluation of MAR during the early stages of design ideation where we attempt to address the issues of interpretability and usability.

2.2 Summary

In summary, the design of MAR services and applications faces the following issues:

  • Owing to novelty and relatively low adoption, the number of available MAR applications is limited. Therefore, awareness and understanding of MAR is still relatively low. We need compelling ways to “tell the story” about and illustrate MAR services and applications so that users understand the concepts well enough to offer meaningful insights.

  • MAR usually requires sophisticated infrastructures to be built before users can fully experience their capabilities and features. Guidelines and design heuristics are lacking for approaching the design of MAR services and applications.

  • Validation and testing are complex due to the richness of the experience, especially out of the lab.

In this chapter we ask:

How can we create a user experience that is sufficiently “realistic” and provocative for users to envisage the final service experience and thus give meaningful and actionable to developers and designers, even at the earliest stages of design.

3 Mobile Augmented Reality Trends

Technological advances in sensor technologies (e.g., GPS, accelerometers, gyroscopes, cameras) and their increasing inclusion in consumer mobile-phone handsets mean that MAR applications can utilize a number of modalities in their content presentation: vision, sound and haptics are the main modalities.

The most commonly explored form of AR is visual-based. Indeed, the majority of existing commercial applications used visual augmentation overlays. Examples are Yelp’s Monocle,Footnote 3 Layar,Footnote 4 and various research/artistic projects where visual effects are used to create artistic renderings of reality or augment monuments and art installations with colours, drawings and animations [22–24]. Visual MAR applications typically overlay additional information on top of what the smart-phone’s camera captures, displaying a combination of the real world with added information on top of what is around the user. Some experiments have even utilized MAR as way to present information that is concealed by physical structures or simply by the users’ orientation, displaying information and directions to content and data that is located behind the user or off the screen, exploring different types of presentation [25]. Typical use-cases fall into the categories of information seeking/search, navigation, push content such as on-the-spot content recommendation and advertising, entertainment, gaming, or information augmentation such as provision of historic details [15]. The majority of such applications draw geo-­referenced data from a variety of services (e.g., restaurant listings, pharmacies), providing information on the user’s surrounding [9, 17] and augmenting the location context with relevant data [15, 26].

Two other trends within MAR, much less explored, make use of the additional feedback channels available on most smart-phones—haptics and sound [27–29]. The most common example is the eyes-free, turn-by-turn navigation system that uses speech to provide directions to its user, like for instance the Google’s Navigation application.Footnote 5 Research efforts have started to focus on combining haptics and sound to augment reality at specific locations, providing context-awareness of additional information. These techniques have previously been explored in gaming apps (e.g., treasure hunts) and for commercial or advertising applications (e.g., play a jingle/song when close to a certain shop). Other approaches also consider body movement recognition and gestures as a way to interact with augmented reality experiences [30]; we speculate that, with the advent of in-home technologies like the Xbox KinectFootnote 6 from Microsoft, users are becoming increasingly familiar with this kind of interaction so we can expect to see a growth in this area in the upcoming years.

4 Design Approaches for Mobile Augmented Reality

Although some instances of user-centered design methodologies [48] and techniques have been applied to the design of augmented reality systems [31], reports suggest that user involvement and user studies very rarely take place for the design of augmented reality as a field in general [32]. Design methods for MAR, being a substantially younger field, are even less explored. Table 6.1 summarizes the most relevant papers in this regard from our perspective. These case studies were selected because of their exposition of a specific formative or summative design perspective and/or process. The papers also represent contributions that span a range of different domains (e.g., games, visualizations, interaction paradigms, health care, shopping). While not all these papers are explicitly concerned with design methods and processed per se, they do provide some insight on design issues when it comes to MAR applications and services. The papers can be divided into two main areas: (1) MAR design experiments that describe some of the procedures used to conceive, design and test the concepts and (2) field studies where prototypes or systems have been evaluated in context, out-of-the-lab.

Table 6.1 Summary of used methods and techniques for the case studies discussed in this chapter
Full size table

In the two following sections we summarise the content of these papers and tease out what the authors have to say about the design methods and processes they used. In each case we highlight the tacit or explicit design questions the authors pose, the design options under consideration [and where possible the final decision/outcome of the research in terms of design choice(s) made], and the methods and techniques used.

4.1 Evaluating the Usability and Effectiveness of Mobile Augmented Reality

The significant differences we find between traditional user interfaces and augmented reality have already motivated several studies trying to assess the adequacy of traditional techniques to this domain or even experimenting with new ones. In their work, Nilsson and Johanson [33] conducted a usability study on the use of augmented reality within hospital settings, using a cognitive sciences engineering (CSE) perspective. Rather than studying the system as a combination of different parts, this approach focuses on the system as a whole, including both users and the system itself. This approach was used to assess the user experience and user acceptance of a system for hospital workers who are required to read instructions during their activities. Although the system does not use a smart phone or traditional mobile device, it is still somewhat mobile as it is composed of a head mounted, video, see-through display attached to a laptop. Users had to perform an everyday task for their work context—the assembly of a medical device, following the instructions provided by the AR system. The study took place within the working place itself, as the CSE approach suggests that conclusions about the use of a new technology should be drawn in its intended use-setting. Arguably, in situ testing is even more relevant for assessing the effectiveness of applications intended for specialists in high-risk settings. Twelve participants interacted with the system and data were collected during the evaluation sessions through observation and questionnaires. The results showed that by applying a CSE perspective and conducting the evaluation in situ, at the hospital with end-users, issues emerged that wouldn’t have been noticed otherwise. This real-world setting was crucial in revealing the pros and cons of the design, considering not only the usability facet of the experience but also dimensions such as the social impact that it might have on workers and the working environment. In particular, one of the most noticeable issues for this particular case was the physical appearance and size of the system, which posed some difficulties in the intended use environment.

Xu et al. [34] describe a system that supports in-store shopping using a vision-based mobile AR application. Their study focuses particular attention on the visual attention required to interact with a mobile device while shopping. As a starting point for their design process, the authors conducted an ethnographic study, divided into diary studies and interviews with 12 participants, over the duration of 1 month, to understand the role of mobile devices (a phone in particular) during the shopping experience. Results from this study were divided into four main categories (1) communication—for instance, taking pictures and sending them to friends, calling someone or chatting while waiting in line; (2) organizational—remembering product requirements, location or number of a store and reminding the time of purchase; (3) informative—monitoring biddings, searching for prices and promotions; and (4) transactional—purchasing using the phone. Based on the results from this initial study, a set of design principles and a prototype for a vision-based shopping tool were created. The tool, Point&Find, allows users to point at objects and retrieve information related to those objects. Transactions were not contemplated in this prototype, which focused on the three first categories, as these pertain more to the user experience and interaction process that takes place during the shopping activity. The prototype consisted of a working system composed of three main components: an object recognition function, the user interface that shows recognition results with the viewfinder and connection to various Internet services. To evaluate the prototype a formative field evaluation was conducted, using actual devices. The authors’ intention was to create a more realistic understanding of usability issues, with a particular focus on cognitive and interaction concerns. The authors also conducted the study in a real shopping setting with 17 shoppers who were recruited on the spot. There were significant challenges to conducting the study, with issues arising with instrumentation that affected data collection, issues controlling the circumstances of the testing environment and recruiting participants for longer periods of time. However, results from the study clearly demonstrated that user attention constantly switched between the physical and digital world. In addition, this experiment allowed the authors to detect several patterns of attention switch (i.e., browsers, frequent switchers and immersed researchers). Clearly, these patterns showed that the application interfered with or changed their shopping flow, especially at a physical level, restricting their manual interaction with objects. The kinds of results the researchers observed underscore that evaluation techniques that abstract the use of a device away from real world settings (either empirical methods like lab studies or analytic techniques like task analysis), may not be the most suitable techniques for evaluating this kind of MAR applications and prototypes; the issues observed would simply not have arisen and not been documented but for the occurrence of the environmental factors and natural behaviour in the real world setting. As such, the authors highlight the need to, in addition to conducting studies within real world locations and settings, define tasks that mimic common behaviour for the task being analyzed during the design process.

In a similar experiment, Nigay et al. discuss the use of scenarios for the design of mobile and collaborative augmented reality [35]. The motivation behind their work arises from the object oriented and real world-based approach that characterizes some MAR applications, where physical objects and constraints of the real-world play an increasing role in the design process and resulting experience. The authors argue that field studies and scenarios are especially important for MAR design processes. Field studies force us as designers and evaluators to account for the context of use, involving consideration of physical, technical and social dimensions that are seldom predictable or articulated in initial design specifications and envisionments. Scenarios offer a discursive common ground for the collaboration between the design team and users. To illustrate their points, the authors applied a scenario-based design approach that has two stages. The first stage consists of the design of scenarios based on reports from users on their work practices while the second stage involves iteration and refinements of the scenarios based on an analysis of users’ activities by the researchers—users are observed and recorded at their work site. The data that result from these two stages is then used to derive a set of functional requirements, which serve as the basis for the system specification. Once these requirements are defined, they are evaluated once again against the defined scenarios, now integrated with the specified functions, which the authors call “projected scenarios”. Results from this evaluation stage are then used to create the interaction techniques, which are based on the resulting functional specifications. Once the system is developed it is later evaluated in location, assessing both functionality and usability. Highly iterative, this process is reportedly also most effective: the authors report a successful application of the method in the development of the MAGIC platform (mobile, augmented reality, group interaction, in context), a component of an archaeological prospecting system. In particular, this approach highlighted some inherent limitations to the domain and common practices that had to be addressed, especially for collaborative activities (e.g., collection of data in the field, contextual evaluation of elements and remote discussion between archaeologists). The projected scenarios that emerged addressed collaboration and data gathering, and propelled the design of an interactive system that offers mixed-reality features, allowing users to move objects between the physical and digital world. This was found to greatly facilitate collaboration and information sharing between local and remote archaeologists.

Lee et al. also requested end-users to validate their own work through a series of comparative studies [14]. Instead of focusing on new services or applications, Lee et al. propose a new interaction approach for MAR. Their approach, called Freeze-set-Go addresses some of the problems that result from manipulating objects and interactive items on mobile displays. Not only does the size of the displays affect how users interact with the interface, but also the new usage paradigms that require users to interact while walking or while making use of both hands, decrease accuracy and sometimes produce poor usability results. By allowing users to freeze a scene of the mobile augmented context, creating an image of the real worldview, the system allows users to interact with the items and with content that is overlaid on top of the image more accurately/effectively. To evaluate their system’s performance the authors conducted a study with end-users assessing task performance when making annotations on a MAR environment. Although the experiment took place within a lab setting, the study involved people conducting tasks while simulating real world situations and poses. To do so, participants were requested to complete the tasks under four combinations of difficult and easy tasks, defined by the height in which the objects to be manipulated were located and the use of their interaction approach (FSG—Freeze-Set-Go) or a traditional MAR approach (without freezing the scene). In their study, and although not utilizing an open approach to the evaluation, or taking it to the field, the inclusion of difficult poses, simulating real-world settings and scenarios showed that their design worked better under difficult poses for users, without sacrificing time for accuracy. In sum, given the impossibility of taking the design out of the lab, the authors simulated some aspects of the real world in order to create a life like experience. This yielded better results, allowing for the detection of issues regarding accuracy and how different behaviours were restricted or motivated by different settings.

In 2008, Damala et al. described the design process of a MAR museum guide, offering insights into their approach and the value added by taking a user-centred, scenario-based approach [20]. Their process started with the definition of a set of scenarios and a list of possible functionalities that were presented to possible stakeholders, including museum professionals and technology specialists. Developers and stakeholders were involved in a participatory, collaborative process of potential system requirements specification. This stage was followed by the creation of the actual content that would be presented to users and associated with the works of art displayed at the museum. This process took place at the lab. The final prototype consisted of a mobile device that, when pointed at the paintings, displays 2D and 3D virtual objects. These objects can be interacted with and additional information and digital documents can be easily accessed. To evaluate the system, a set of evaluation sessions took place inside the museum itself, using field observation methods to assess system effectiveness, usability and utility. Users were requested to wear a belt that included a set of additional media recording devices to capture sound and video through a head mounted camera, used to record the users’ interactions with the devices and museum art. The evaluation sessions took place during the course of 2 weeks. Throughout this period 12 users visited the museum using the system and were observed, filmed and interviewed. During each test, participants were required to locate the paintings that were augmented with additional information and to freely navigate in the content according to their preferences. Each test lasted between 25 and 60 min, followed by a 15-min interview. The authors highlight some of the major findings from their user study. In particular, the sheer number of visitors to the museum during some of the sessions posed some limitations to what participants could do; clearly lone visitors would have had a very different experience. The field trial therefore exposed assumption about people’s physical space allotment and the quality of their line of sight to objects of interest. The large number of visitors also made it difficult for the participants to understand the audio that was being provided by the system. Further findings demonstrated that more playful content appealed more to users and that the overall experience was considered to had benefited from the digital guide. Overall, the environmental constraints and the real-world setting in which the study took place provided insight into issues that would not have been found had the study been conducted under a more controlled, lab setting. It is also notable that, while highly valuable, the technical set-up of the study had its challenges. The authors also comment on the inherent challenges posed by the novelty of the used technology. Qualitative approaches were needed, in addition to quantitative ones (e.g., questionnaires with Likert-scales), in order to gain a deeper understanding of the pros and cons of the system in use—only with a qualitative analysis of rich interaction data could analysts distinguish between usability and utility issues that were likely to persist from issues that arose specifically from the unfamiliarity of the users with the technologies themselves. The latter would likely be extinguished through experience while the former would not; this is a crucial distinction when documenting usage difficulties in user trials.

4.2 Field Studies

The work of Morrison et al. illustrates nicely how field studies, combined with some role-playing, have been used in the evaluation of AR experiences [36, 37]. While developing an application called MapLens, a MAR map using lens over traditional paper-based maps, the authors conducted field studies with end-users in a city centre. A very high-fidelity functioning prototype of the application was used for these studies. Twenty-six participants interacted with their application in a game-like scenario while 11 participants performed the same tasks using a 2D traditional map application. Video recordings, logs and field notes, questionnaires and interviews were collected. The authors studied how users held the devices, how they used their hands to interact with the device and maps, nuances of and shifts in their body posture, the kinds of manipulations that were applied to objects, how users walked while using the devices, and forms of collaborative use. In particular they highlight how the most common behaviour was for users to stop walking and to gather around the device and map to explore the area and review detailed information. Moreover, users’ interactions with the environment were documented during the field trial. The possibility that MapLens could be used effectively in conjunction with billboards or other maps was revealed during the study—evidence that field trials can lead to creative invention as well as summative evaluation. In their paper, the authors emphasize that without taking the trial into a real-world setting, the study would not have offered such rich results.

Schoning et al. also utilize a field-based approach to evaluate a map-based AR application [40]. The authors’ goal with their project is to overcome one of the main issues with magic lens approaches: the attention switching between the device and the physical map. With magic lenses, dynamic information is presented on the device’s display when pointed at traditional maps. Here, as a solution for this attention switching issue and the relative small screen and resolution from mobile devices, the authors present a system that utilizes mobile projectors to display additional information directly on the map—i.e., the content is projected onto a map instead of being rendered on the device’s screen. To test their concept the authors created a simple prototype using a small projector connected to a smart phone. Based on the information captured by the phone’s camera, the system projects the augmented content on top of the map. The major benefit from such approach is that users no longer need to switch between the physical map to gain context and the device’s screen to access the extra information. Additionally, the amount of information that can be displayed because of the larger projection is substantially greater when compared to the mobile device’s screen. This larger display area also affords for easier collaborative use of the information that is overlaid on top of the map. A user study was conducted to validate the initial prototype. A set of 12 participants interacted with the prototype and completed a task using the system (i.e., find five parking lots on a map and identify the cheapest one). Results from this study showed the potential that this approach had in terms of collaboration and use by small groups, overcoming some of the limitations that magic lens interfaces pose. Moreover, using the map TorchLight users completed the task 15% faster than using the magic lens approach. However, the low light intensity from the projectors pose some issues to this system as they become difficult to read in outdoor settings, requiring alternative technologies (e.g., laser projectors). Again, these details can only be fully understood and detected when conducting the evaluation process in realistic settings.

In their work, Avery et al. present a similar study and results [38]. To test a see-through vision MAR prototype, they conducted a set of field studies and indoor studies with different users groups, with a total of 37 participants. The system aims to allow users to see through objects and buildings by overlaying videos of what is behind on the screen. Two groups of participants took part in the user study. One group of 20 participants was assigned to an indoor setting and the other group of 17 participants was assigned to an outdoor location. The indoor test was completed on a desktop computer while the outdoor tests used a mobile system. To simulate the working system, the authors created a set of videos that replaced the real-time video stream that would ordinarily be received by the system from the cameras located at the required spots. These videos were used during the evaluation sessions at the locations they were filmed. One of the hypotheses was that users would be able to understand a video more quickly and comprehend its contents if seen in situ with a see-through vision system compared to watching it remotely through a LCD display. In addition to two basic tasks that each participant had to complete (e.g., identify locations based on the videos and AR content shown through the system), a scenario-based approach was also used. This latter task was designed to simulate an emergency rescue situation. To complete this task, participants had to locate three injured people and chart the best route to reach them from an adjacent building. The application enables participant to see through walls and buildings that occluded where the injured people were located. Results indicated that outdoor participants were more efficient at completing the tasks compared to indoor participants and that the videos were significantly easier to understand on the see-through system when compared to the desktop counterpart. Moreover, the learning curve appeared to be small, as most participants completed the second task in shorter times. The success of this study again points to the value of doing evaluations “in the wild”—in more realistic settings, but also that scenario-based approaches that re-enact real life situations, provide a valuable tool to validate mobile AR systems.

A similar project 2010 project by Schinke, Henze and Boll also tackled the challenge of providing information that is concealed or beyond the screen (besides or behind the user) [22]. The goal is to replace the traditionally used 2D mini-map that is often combined with the AR view and replace it with objects and points of interest that are displayed even if off the screen. The system works by displaying arrows pointing in several directions that indicate the existence of additional points of interest, even if these are located behind the user. Such information is not usually displayed on traditional augmented reality apps. To evaluate their approach, the authors conducted a user study. The study took place at a city centre and the 26 users who participated were recruited on location. After having the concepts involved in the system explained (i.e., points of interest and augmented reality), users completed a set of two tasks (e.g., identifying and locating points of interest). A questionnaire and observations were administered. The study results showed clear differences between the two approaches, suggest that MAR interfaces could be improved by use of 3D arrows. It is also noteworthy that although the study took place in the field, the data were simulated. Still, this approached afforded a realistic situation that allowed for field studies and that yielded positive results. Nevertheless, despite the positive outcome, the authors point out that further studies are required to assess whether the quantity and quality of the simulated information being displayed did not affect the realism of the experiment.

Another interesting example of how field studies play a relevant role and how they can be used for the evaluation of MAR can be found in 2008 work by You et al. [39]. The authors conducted a field study with end users to evaluate a mixed-reality mobile game, using both virtual and real world cards placed at different locations using a mobile phone. The study was divided into three stages. The first, a pilot test, was conducted to verify that the system was working properly. This portion was conducted without end users’ participation. The second stage consisted of material preparation; interview scripts, questionnaires, storyboards and additional evaluation material were prepared during this stage. Five users were requested to assist during this stage conducting a meta-evaluation, testing the procedure and the evaluation material and allowing for the adjustment of the process for the final evaluation stage. Quantitative data were collected through logs (e.g., GPS data, trail and time) and qualitative data through interviews and questionnaires. In addition to these two data collection techniques, users were also followed and observed throughout the game/evaluation session and, at points, interviewed during the game, following a method like that advocated contextual inquiry. The final stage replicated the previous one with 30 players and some minor study design adjustments. One of the adjustments was the composition of teams following suggestions offered by earlier participants; users from the second stage commented that the game would be more entertaining if played with friends. The authors report results throughout the process. Users’ reflections on distances, on set up difficulties and on the game’s ability to sustain engagement are reported. The field study also allowed for the assessment and impact of contextual details such as terrain difficulty and safety. Such factors have a direct impact on the experience and can shape the design of the game, but often are overlooked in more device/technology-focused studies.

4.3 Summary

Although mobile augmented reality is a relatively recent field of research and development, there is already a significant body of published work. However, much of this published work is very technology-oriented. Very little of the published work presents authors’ design philosophy, design perspective or design methodologies deployed during the design and development of these systems. Discussions of user participation during early stage design and design processes to elaborate user needs are rare and use of low and high fidelity prototypes infrequent.

However as Table 6.1 illustrates, user centred design approaches and field studies are gradually becoming more popular; a more user-oriented perspective is evident from 2007 onwards. Out of the nine case studies summarized in Table 6.1, eight included users and some sort of user study while designing their system. In particular, it is noteworthy that the majority of these user studies were conducted in realistic settings, mostly in the field.

In the following sections, we present our own approach to the design of MAR applications where we have been focusing on prototyping and evaluation techniques that provide a sufficiently rich experience to enable the gathering of relevant feedback to drive design insights, inspire interaction methods and select between design alternatives.

5 Prototyping Mobile Augmented Reality

In this section we discuss the design of a social media, MAR application we have been developing in our group. A major challenge we have faced in the design and development of these applications is the creation of high-enough fidelity prototypes to conduct meaningful and effective evaluation of design concepts. By “high-enough fidelity” we mean creating prototypes whose embodiment (form factor plus appearance plus interactivity) is suggestive enough for users to be able to give an accurate estimate of their likely utility and usability. Our goal is to, with the least effort/time/design commitment, create effective props to simulate the kind of interactivity that the final application will support. Too little interaction and users are left unable to imagine usage scenarios effectively, yet truly high fidelity prototypes require too much development time—in some instances a high fidelity prototype can lead to infrastructure design decisions, once developed, remain in place simply because they are too costly effort-wise to dismantle and rebuild—irrespective of whether they are (or are not) interactionally effective and elegant. This, therefore, defeats the purpose of the formative evaluation.

Our aim in this chapter is draw on insights from previous work, outlined above, by following field trials but to offer a cost–benefit analysis of different type of prototype from low- to high-fidelity for effective user-driven design elaboration and evaluation in this space. Below, we describe the design probes we developed in the early stages of developing our Friend Radar application.

5.1 The Design of Friend Radar

The Friend Radar application merges social networks, messaging tools and location-based services, and makes use of AR. Information is presented in a way that allows users access without losing the current, local, physical context around them. The Friend Radar Application draws data from existing technologies such as social networks like Facebook and Google+, location based apps/services such as Foursquare and similar check-in tools and previous experiments designed specifically for mobile devices like DodgeBall Social [38]. However, the Friend Radar enhances the experience by providing an enriched visual display of friends overlaid on the users’ surroundings (Fig. 6.1). In particular it provides added affordances that allow users to see where friends are situated in relation to him/her, and their distance from the user. It also presents details such as preferred friends, directions where groups of friends are located and their availability. Unlike map-based radars, this AR based concept does not require users to translate two-dimensional views into their surroundings [41].

Fig. 6.1
figure 1

Initial sketches for the Friend Radar prototype. The top figure shows the main view where the camera feed is displayed and the various friends are identified and their avatars overlaid on top of the real world view. Below, the user profile is shown with information about distance, availability, and recent activity retrieved from different sources and sharing options

Full size image

Our goal with this application, in this initial stage, is to show friends’ positions in relation to the user, including details such as proximity and availability. Friends’ location and status can be retrieved from various sources (e.g., Messenger apps, Social networks) and their avatars personalized with different images or pictures. User avatars are overlaid on the camera view of the location at which the user is pointing his/her camera/device (Figs. 6.1, 6.2 and 6.3). In addition, tapping on a friend’s icon displays his/her profile with information from the service they are using at that moment and allows for some types of communication (e.g., sending a message or alert). The application thus merges social network data (both the contact’s information but also the contact’s relevance—such as, emailed you yesterday—for the viewer) with the real world, enhancing it with additional, contextually relevant, information. This in-formation contains both the user’s location, proximity as well as data such as mutual interests, recent check-ins, multimedia capture within the context of both users’ location and other information that has the potential to trigger a physical connection or some form of collaborative activity.

Fig. 6.2
figure 2

(Left) Low-fidelity mock. The see-through hole allows users to see what is behind the device, simulating the camera. (Right) Low fidelity mock and used icons (right) next to an actual working device with the hi-fidelity prototype (left)

Full size image
Fig. 6.3
figure 3

Video of the mixed-fidelity being played on a smart-phone

Full size image

5.1.1 Design Approach

Our aim is to understand the best way to prototype for MAR. To achieve this goal, we began by utilizing some techniques based on previous experiments for mobile interaction design [1]. As suggested by recent literature, better results can be achieved while designing mobile interaction when immersing users within realistic scenarios, including prototypes and outdoors tests [1, 42, 43]. In order to do so for MAR, our goal was to approach the design process by experimenting with different prototyping techniques and exploring the benefits and drawbacks of each.

5.1.2 Prototypes

Three different mocks/prototypes for the Friend Radar concept were created. To build these we used three significantly different approaches but following the same philosophy: to create a prototype as close to the real experience as possible in terms of form factor and weight. Each is described below.

5.1.2.1 Low-Fidelity Mocks

The lowest fidelity prototype used for this study was built using a dummy/non-functional device (e.g., a product design mockups that was created typically to illustrate a form factor design). The dummy phone mimicked a common Android device with a 3.5-in. screen (Fig. 6.2). In order to simulate the camera feed, the screen was removed and a hole was cut on the back cover of the phone (the inside of the dummy phone is hollow). A transparent screen was placed on top of this hole to simulate the device’s screen (e.g., reflection) and to allow users to easily use it as a touch input device, while maintaining the ability to see through it (see Fig. 6.2). In addition, this screen was also used to allow for an easier use of the Wizard-of-Oz technique, where small icons were glued to the screen to simulate the augmented reality. For the moving avatars longer pieces of paper were used. This facilitated their movement by the “wizard” simulating the actual location-based interaction.

The building process lasted approximately 1 h, including the creation of the attachable icons (e.g., map, avatars).

5.1.2.2 Mixed-Fidelity Videos

For the second prototype, a mixed-fidelity prototyping approach was used, combining aspects from both low and high fidelity prototypes. In particular, for this case, and following the categories defined by McCurdy et al.’s [44], the degree of visual and aesthetic fidelity was high while the interactivity remained low. Our hypothesis was that video would be adequate for simulation of a realistic field-based experience at during early formative design, as movement can be simulated and additional content can be placed over previously captured footage and displayed directly on the device.

To create the prototype, two different locations were selected. Videos were shot at the selected locations. For the first location, a public park with a few people sitting down and walking around was selected (Fig. 6.3). The second set of videos was shot in a busy square with shops, buildings and people walking and standing in different areas. Both locations were selected because they represent places where friends usually meet up, seek encounters and congregate. Each video had an approximate duration of 30 s and included light panning and some jitter to emulate a realistic usage scenario (i.e., scanning the area for friends). Once the videos were captured, they were edited and the friends’ icons and avatars were overlaid using video editing software. This process took around 1 h for each of the videos. The videos were exported to a phone, used during the evaluation sessions.

5.1.2.3 High-Fidelity Prototype

The high-fidelity prototype was developed using the Android Development Kit. The prototype uses the camera feed, displaying it live and showing whatever the camera is capturing. On top of this feed, shown on full screen mode, a set of icons and avatars is also displayed on semi-fixed positions. Using the accelerometer and compass sensors, whenever the device is rotated, the icons and avatars will maintain their position in relation to the surrounding environment. They will be occluded when the device is not facing the icon’s position. This offers an accurate rendering of MAR (Fig. 6.4). In addition to the interactive view, this prototype also supports interaction with some of the icons. Once an icon is tapped, a second screen displaying detailed information about the selected entity (e.g., a person) will be shown (see Fig. 6.1). The working prototype required approximately two working days to be fully developed. We note that the icons had already been designed for the previous prototypes.

Fig. 6.4
figure 4

High-fidelity prototype working on an Android smart-phone

Full size image

5.2 Evaluation and Discussion

These three prototypes were shown to end-users at different locations and interacted with during some outdoor experiments. Three groups of eight users each experimented the prototypes—only one type of prototype per group. A summary of the results organized according to the following design stages and goals can be seen in Table 6.2:

Table 6.2 Summary of results and adequacy of each prototyping approach for design stage
Full size table

  • Probing—triggering users’ imagination in a provocative way and using the prototypes to explore new applications, concepts and usages for the technology being studied [45, 46].

  • Concept validation—addressing the concept in general, the overall idea of the application and its goals, by presenting it to users and requesting their feedback.

  • Feature validation—validating the different features and functionalities that compose the application in more detail.

  • Usability testing—addressing interface usability issues and breakdowns and assessing efficiency and ease of use [47].

  • User experience evaluation—understanding users’ feelings, opinions, expectations, acceptance, pleasure and deeper emotions regarding the experience of use.

As expected, each type of prototype has its benefits and drawbacks. However, while all have positive aspects and are adequate to particular stages of the design process, two types of prototypes stand out for the best and worst reasons. On the one hand, low-fidelity prototypes, which have shown to yield good results from mobile design, were not favoured by participants and provided poor results during the experiments. Major complaints pointed out the distracting process of simulating the movement of the different icons (i.e., Wizard of Oz) and the lack of interactivity of the prototype. Even considering the easy to update and adjust on the fly approach that these prototypes afford, they did not work well for usability testing and functionality validation. More importantly, the cumbersome nature of the experiment and Wizard of Oz approach with this type of prototype also affects they way in which these prototypes can be used to probe users and experiment with different ideas very quickly.

Mixed-fidelity prototypes, on the other hand, were very easy to understand by participants, provided a great way to discuss features and brainstorm over the concept. Moreover, even with limited interactivity, the mixed-fidelity prototypes also allowed for the detection of some usability issues (e.g., avatar size, amount of information displayed at the same time, labels and even the application’s layout). Considering the time spent to create each of the video prototypes, these showed the best trade-off in terms of cost-effectiveness, being very easy to build and providing great results at both early and later stages of design.

Finally, when it came to the high-fidelity prototype, the results fell shorter than we initially expected. The prototype’s interactivity, functionality and the fact that users were actually using a real device raised expectations to such a point that every feature was faced and interpreted as final and working. Although it provided some room for probing and brainstorming, the few glitches and minor bugs distracted users from the concept being tested and detracted from the exploratory nature of the experiment. The observation that prototypes which are too polished can result in user disappointment, more critical assessments and less creative feedback has been observed elsewhere [48]; often more sketch-like prototypes lead testers to creative insights as they “fill in the gaps” [cite Buxton’s book here]. The tension is to support a close-enough experience while allowing room for creative feedback. Of course, higher-fidelity prototypes are likely to provide good results for functionality validation and usability testing later in the design phase. However, considering the time and effort required to build this type of prototype, these are not always adequate for early design stages or as props for ideation and scenario-based experiments.

6 Conclusion and Future Works

MAR is a fast growing and increasingly relevant field; researchers and commercial concerns alike are focused on building the next generation of innovative products in this space. A wide variety of services and applications are taking advantage of the benefits that augmented reality provides. These are especially interesting when used on mobile devices where users are free to interact and see the world augmented by information that would not be available otherwise.

However, despite the appeal and the growing number of services and applications, very few guidelines, design techniques and evaluation methods have been presented in the existing literature. In this chapter, we posed the question:

How can we create a user experience that is sufficiently “realistic” and provocative for users to envisage the final service experience and thus give meaningful and actionable to developers and designers, even at the earliest stages of design.

We provided an overview of methods and techniques that have been reported in the literature for the design of a variety of MAR experiences, ranging from maps, shopping tools, games and even museum guides. Different modalities and interaction paradigms were discussed; our summary focused on design process and in particular the way in which the design process proceeded, the experience was conceived and prototyped and how those prototypes were validated with end users. A common trend observed was the use of field-based evaluations, experimenting mostly with high-fidelity prototypes and end-users within the context where the services and applications will most likely be used.

We noted that, to prototype the envisioned, fully-fledged service is obviously costly. More problematically, development without design evaluation can result in premature, often non-retractable, commitments within the service design; under-examined, premature commitments are precisely what iterative design and evaluation are aimed at circumventing. However, we noted the tension: first, to get useful feedback from users, it is necessary to create an experience that is “realistic” or provocative of the envisioned scenario of use to enable users to give meaningful and actionable feedback, but, second, to make the system realistic enough requires commitment to engineering resources, development costs and design commitments, that, due to limited resources, end up being reified into the system design whether or not they are in fact the most efficient, effective or engaging design options.

Building on this prior work, we evaluated the use of in-context evaluation prototype probes that ranged from low to high fidelity. We presented experiments to assess the prototypes and their potential for revealing design insights at different phases in the design cycle. We highlighted the benefits of different prototyping approaches and discussed the trade-offs in terms of effort and time costs for each of these approaches at various stages of design.

Overall, our results indicate that low-fi prototypes are of little value when used to validate or probe MAR concepts—they do not provide the necessary affordances nor the interactivity required to gather valuable feedback from participants, especially those who are not familiar with AR. At the other end of the scale, high-fidelity prototypes which we expected to yield the best results, were surprisingly ineffective. They provided a relatively realistic experience for users, but raised expectations that led to disappointment and focused negative critique with little creative engagement in dialogue about opportunities for improvement. In the final analysis, the video prototypes proved to be the best option for rapid prototyping. They led to engaged user participation, actionable feedback and creative insights for effective MAR design for location-based social networking. Although interactivity was limited and location/setting/scenario requirements were constrained, from a cost–benefit standpoint, these were the most effective prototypes: rapid generation with low-effort development, coupled with sufficient realism to support scenario engagement whilst retaining the feel of a mutable prototype. Combined, these factors offered participants the best experience of the concept under development while giving them the space to offer constructive critique.