Keywords

1 Introduction

Recent advances in application programming interfaces (APIs) for computer vision, tracking, and scanning 3D geometry on mobile devices provide commonly available platforms for developing augmented reality (AR) applications. However, AR is still in its infancy and few precedents exist for how programmers should design the user interfaces for AR applications. At the Center for Advanced Vehicular Systems (CAVS) at Mississippi State University (MSU), we have developed prototype AR applications for marketing and outreach, education, and industry on iOS devices using Unity and Apple’s ARKit. Each application implements different methods for user interaction with the AR environment. This paper focuses on how interaction with the 3D environment on a 2D touch screen presents challenges for AR application designers.

These applications are used to experiment on different tactics for the program to communicate to the user. One of the prototype applications provided users with multiple methods for interaction, including flat on-screen elements (e.g., ‘Next’ arrow) to supplement 3D interactive objects (e.g., lug nuts in the tire change applications). World-anchored floating 3D text presented challenges in both ensuring an aesthetically pleasing presentation and, more importantly, for ensuring that the information was easily visible and readable. As shown by the Labeled Electrical Box Application, this type of text is often large and gets in the way of the rest of the application. Screen-anchored 2D text, on the other hand, improves readability but fails to fit with the dynamic AR environments.

1.1 Related Work

Augmented reality (AR) is a rapidly developing field characterized by swiftly changing technology. As a field, AR platforms range from television and desktops to mobile systems (e.g., smart phones and tablets) to smart glasses and head-mounted displays (HMD). Application areas for AR systems include industry [1], context-based instruction [2], education [3], entertainment [4], navigation [5], health [6], and more. This paper focuses on mobile augmented reality (MAR) applications developed for a through-the-camera display on a tablet device.

In 2017, standard tools for developing advanced MAR were released by Apple (ARKit for iOS) and Google (ARCore for Android) [7, 8]. In 2018, Unity released AR Foundation tools that provide a common interface to both ARKit and ARCore, simplifying development for both iOS and Android MAR devices [9].

There is a growing literature on guidelines and principles for MAR development [10, 11]. Both Apple and Google provide human interface guidelines for design of MAR applications [12, 13]. A benefit of MAR is that the user’s view of the real world is enhanced, albeit through a limited window [6]. MAR systems can help users to identify and locate key landmarks and features in the environment [1]. MAR can attach and display virtual information to landmarks, e.g., procedures to perform or a display indicating the safe areas near equipment [2]. A challenge for MAR application design is the combination of a 2D input interface (the tablet touchscreen) onto a 3D mixture of virtual and real environment elements. In addition to the 3D nature of the environment, the mobile nature of the device leads to an unpredictable variety of situations and environments in which the application may be used [11]. MAR may also slow user interaction with real world elements due to the pace at which the application provides instruction or its ability to process the environment [6].

2 Augmented Reality Applications

2.1 Tools

Our MAR applications were developed using Unity 2018.3.4f1 and the AR Foundation tools [9]. The applications were deployed to an Apple iPad Pro (Model No. A1893) running multiple versions of iOS over the course of development (11.1.2, 11.3.1, 12.0, and 12.1.4). Unity and the AR Foundation tools therefore provided a platform agnostic interface to Apple’s ARKit (version 1.5) tools [7]. Scene mapping and object recognition was implemented using Vuforia libraries incorporated in Unity. Vuforia provides tools for both 3D object scanning and 3D object recognition. Vuforia was used in the labeled electrical box application to create a virtual version of and to recognize the electrical box.

2.2 Tire Change Application

The tire change application was developed as a continuation of an earlier MAR application designed for the Mississippi State University Formula SAE team. The previous application used MAR to create the appearance of a virtual model of the team’s competition vehicle in the real world. This application was used to educate users on the features and design of the vehicle. The modified application provides interactive instructions for the process of changing a flat tire on the virtual vehicle by walking the user through the procedure step-by-step in MAR (see Fig. 1). In MAR, the user acts on the virtual vehicle with virtual parts and virtual tools presented at a 1:1 scale.

Fig. 1.
figure 1

Screenshot of the tire change application. The user must tap an unattached lug nut or the right arrow at the bottom of the screen to move to the next step.

Full size image

Purpose.

This prototype was developed as a demonstration that MAR can be used for purposes other than entertainment; more specifically, the Tire Change application was intended to serve as a demonstration of an instructional MAR application. The users can learn how to perform the procedures by using MAR on a full-size vehicle without requiring an actual vehicle to be present and minimizing the risks associated with working on an actual vehicle.

Function.

The MAR environment supplements the learning experience by providing the user with the flexibility to view the process as if it were physically occurring in front of them. The mobility of MAR allows the user to review the actions from multiple angles. The interactive features enhance the procedure in a way that maintains the user attention while allowing the user to learn from their mistakes without risk to persons or equipment.

Review.

When testing the application, it became clear that the user interface needed improvement. In some steps, the user is presented with an overwhelming amount of text that obscures the display of the MAR application. The application emphasized the text over the observation of the procedure. In other steps, the instructions were too vague and failed to explain to the user the required actions to move to the next step.

2.3 Labeled Electrical Box Application

The labeled electrical box application was developed to experiment with user interfaces tied to physical objects and to test the reliability and stability of object tracking in MAR. The demonstration box is a power supply for up to eight CCTV cameras. The box also includes a power switch and fuses for surge protection. The MAR application added labeled key features and landmarks on the exterior and interior of the electrical box (see Fig. 2). The application demonstrated how MAR could communicate helpful information about an object without the need of a manual. The application also communicated potential safety hazards of which the user should be aware while working with the electrical box. The application recognized both the inside and outside of the electrical box. It also distinguished whether the box is opened or closed and then labeled the box accordingly.

Fig. 2.
figure 2

Screenshot of the labeled electrical box application. The user may tap on one of the color-coded sections to obtain additional information.

Full size image

Purpose.

This application was developed to provide an example of object-tethered user interface elements in a MAR application. The application is relatively simple but similar to the application described in [2]. The user is assisted during maintenance of the device using MAR. Key landmarks and features are identified and information about the box is provided without the need for a manual. The intent of the application is to provide ease of use and simplicity that will reduce intimidation and make it safer to do maintenance work on unfamiliar equipment.

Function.

This program uses MAR to replace a traditional manual; it greatly reduces the linear search requirements imposed by a manual, enabling the user to quickly determine key information needed to perform tasks. It also reduces the risk of confusing the specifications of the components.

Review.

Demonstrations of the application revealed flaws due to certain design decisions. For example, as the camera moves and the background changes, the floating text becomes more difficult to read. In addition, the application did not communicate to the user what elements were interactive or how to use them. As a result, users did not recognize that certain elements were interactive. However, once the user selected an object and made changes, the UI responded with alterations that led users to understand the interactive nature of the application. For example, the information and icons on the box changed after the lid was opened, providing the user with reassurance that they were moving in the right direction. This example highlighted another UI issue, however, as the electrical box is difficult to open and often requires two hands. Since the user is holding the device and has only one hand available, this made some users reluctant to put down the device and thus, continued to struggle to open the electrical box with one hand.

3 Lessons Learned

3.1 User Interface Design

A comprehensive analysis and discussion of issues encountered for all applications formed the basis of a refined UI design approach. This approach includes a mixture of visual cues and intuitive animations, paired with reduced and more precise text, all of which will simplify and enhance the user’s experience. While proper balance of text and visuals is a common general user interface issue, this difficulty is heightened in augmented reality (AR) applications. In this environment, the need to maximize the user’s view can lead to misuse of text in and overlaid on the camera view. As of this writing, the Apple guidelines recommend screen space text elements overlaid on the camera view [7].

3.2 Loss of Maneuverability

When using MAR one of the user’s hands is dedicated to holding the device, leaving only one hand free for interacting with the physical world. This lack of mobility has no consequences in some applications but begins to cause issues when the MAR application is tailored to maintenance work. The programmer and designer identify these types of situations and prompt the user to put down the device whenever the issue may arise.

4 Augmented Reality Applications

We developed prototype mobile applications to demonstrate the benefits of MAR for two purposes: instructing users on procedures for a virtual vehicle and projecting technical and safety information onto a physical object. During development and review of the applications, we identified usability issues related to display of text on the mobile device and the need to continue to hold the device. As the focus for developing MAR technology moves from technology-driven to application-driven, user interface designers will continue to expand and apply understanding of how issues specific to MAR affect application usability.