1 Introduction

Augmented Reality (AR) is a variation of virtual environments (VE), or the more commonly called virtual reality. VE technologies plunge users into a synthetic environment altogether, so that they can not see the real world around them whilst being immersed in the VE. In comparison, AR helps the user to see the physical world with imaginary objects superimposed over the real world. Hence AR extends reality rather than entirely removing it. AR is therefore the “middle ground” between the purely synthetic VE and the physical world (Milgram and Kishino, 1994; Milgram et al., 1995).

Some researchers define AR in such a way that head-mounted displays (HMDs) are required. This study considers as AR the technology that makes use of any device which has the following three characteristics: a) combines real with simulated, b) is fully immersive and c) is registered in three dimensions (Azuma, 1997). As a research field, it has found applications in various fields, such as medicine, military applications, entertainment and infotainment, technical support and industrial applications, distance operations and geographical applications, for almost two decades (Azuma, 1997; Azuma et al., 2001).

In a user’s view of the surrounding environment, mixing graphical annotations and objects create a powerful metaphor for conveying information about that environment. The potential of AR systems still exceeds its practical applications and, indeed, several AR devices still remain in experimental phases in laboratories. There are many reasons for this; two of the most important are that researchers require more sophisticated technology than is actually available to them and the AR designers needs to address many and complex issues relating to all sorts of human factors getting involved all along the AR development stages. And given that AR systems are typically interactive; usability has to be tested exhaustively before deciding that they are successful (Livingston, 2005).

Every time a new AR technology is introduced, it induces a whole new set of issues to both its designers and end users. These are both technical (i.e. software-specific) as well as contextual. Sometimes the effects are felt by users upon embarking to explore new versions of already known technologies. Expectedly, different users report different experiences (both negative and positive).

A central question then, is how to identify differences in user experiences and consequently, as concerns the core issue treated here, how these differences relate to the degree of faithful representation of reality by an AR technology set. It has been suggested that organizational culture of the company that develops the AR software may also affect how users report their experiences (Tushman & O’Reilly, 1996). Furthermore, studying such technologies in isolation or away from natural settings does not guarantee that users of these technologies will really opt using them (Nilsson et al., 2010).

In this respect, it is interesting to consider an approach from within the established field of human-machine interaction, the “cognitive systems engineering” (CSE) (Hollnagel & Woods, 2005). CSE adopts a functional approach to examining the use of technologies, by focusing on the simple fact that people do not use technologies just anywhere, but in the context of the natural conditions they normally are in, and therefore their experience of these technologies results from the interplay between the natural setting and the artificial space created by the technologies. Hence, simulating the natural setting holistically is necessary in the analysis of users’ experience (Hollnagel & Woods, 1983). Mixed Reality (MR) partially conforms with this concept, in that it fuses virtual and natural elements within a unified natural-technological setting. Hence, the naturalistic approach is particularly conducive to transforming AR experiences from natural environment-irrelevant to natural environment-inclusive.

Perhaps nowhere is this need for naturalistic approach more evident than in the cases where AR technologies are used for educational purposes. Students are often required to acquaint themselves with multiple data sources and data types, originating from both natural and artificial environments (O’Shea et al., 2009) and to use these mixed data to create knowledge and skills that will enable them to navigate in the physical space while using these technologies.

Such carefully designed artificial environments however, open yet another perspective that is interesting to users and to researchers alike: naturalistic settings (particularly the location-based ones) are not always under the users’ complete control and they are not even under the designer’s complete control either (Reid et al., 2011). Characteristic examples are outdoor noise, insects, cars passing by and other distractions and/or environmental disturbances that are beyond the software desinger’s or the user’s control (Dunleavy et al., 2009).

The successful integration of new technologies into an organization or workplace means the system is actually being used satisfactorily by the people for whom it is intended.

Yet, there are several cases where technology was implemented in organisations but it was not used, for a variety of purposes. Frequently, one significant inhibiting factor is the design of the particular software or device that is being used. Thus, a question emerging here is how well the system works in a social setting along with the users i.e. are the users involved and do they see the same promise in the system as the people (management) who wanted to implement it into the organization in the first place? In fact, two important factors influence organizational acceptance of new technology, or rather information systems (Davis, 1989). The perceived usefulness of a system influences attitudes towards the system and user behavior when interacting with the system. If a system’s perceived usefulness is considered high by its users, they can accept it easier than if the system was not perceived by them to be as useful. For an AR system, this means that even though the system may look awkward or bulky (i.e. head mounted), users will opt to accept it easier if its applications are useful enough (or, also pleasant, fancy, sophisticated). Similarly, even an easy to use AR system would not be readily accepted if it is not considered to be useful also (Nilsson and Johansson, 2006). Usefulness alone is not the defining factor in evaluating user’s experience; other factors relate to the system’s ability to support different learning approaches, tasks, programs, roles.

The learning approach “AR task building blocks” for instance, would be more effective if we defined a series of tasks for the user that make up the roles that any AR user will execute at both the perceptual and cognitive levels. AR has historically been mainly visual but tools are now in use meeting the users’ requirements with respect to the auditory and tactile senses (although olfactory and flavor apps are still not available for the greater part of AR applications that are widely in use).

For visual, auditory, and tactile AR functions, we need to continue to analyze performance requirements that must be met by the system or, more specifically, that the user must accomplish through the app, which in effect might require different system performance standards. Research needs also to be carried out to codify program specifications with the aim that the user achieves specific cognitive tasks.

With the aid of AR devices, users are enabled to view characteristics of certain physical objects with visual signs associated to them. To better understand them, the simulated signals must be matched with the actual environment. The spatial resolution affecting the recognition of objects and the perception of their characteristics by the user requires adequate sharpness and contrast in order to distinguish objects from their surroundings. Thus, on these (and similar such) fundamental aspects of sensation, we need to determine the necessary user performance before hand. Visual acuity tests, such as the standard Snellen eye chart can be implemented, while visual and auditory output can be measured by similar tests. Where tactile signs provide the user with details about the system, analogous measures and estimators can be used, which may result in altered, new specifications for display size and related context (visual, auditory, or tactile).

According to the principles of human-centered architecture for immersive environments, defining interface specifications is an integral part of the design phase. User specifications are often defined on the basis of informed opinion elicited from users who have experimented with the new device or software and may take into account the points of view of both the customer and the system designer. These specifications describe how a future product can help users achieve their goals effectively, efficiently and with a high degree of satisfaction. Active user participation is indispensable to knowledge elicitation about the system, its usability and effectiveness and, equally important, to record also negative user experiences that prevent users from a satisfactory exploitation of the AR system’s capabilities (Kujala, 2002). User requirements can be assessed from collections of past records (customer reviews, desk support papers etc), as well as from questionnaires and interviews by using established methods (i.e. ISO 16982 2002 and ISO 13407 1999). Such records can be used to describe the role division between machines and humans (separating machine tasks by those performed by humans).

Previous literature reviews, such as those by Kim and Cooperstock (2018) and Irshad and Rambli (2014), considered studies addressing only potential future research areas of AR and current research regarding user experience of mobile augmented reality (MAR). In neither of these reviews was any reference made to the naturalistic designs applied to AR technologies. Due to this deficiency, additional research articles debating the user experience of naturalistic approaches applied on AR technologies were identified in the context of this research. After considering these articles, it was found that researchers have noted an (expected) positive impact, but also certain challenges imposed by naturalistic designs applied on AR technology, which included usability issues and frequent technical problems. For instance, Martins et al. (2016) pointed out that naturalistic approaches on AR technology may feel unfamiliar for the inexperienced user and may cause frustration due to the use of head-mounted displays. Similarly, Kerr et al. (2011) claimed that naturalistic designs on AR may give users the impression they haven’t understood properly or fully the technical functionalities of the systems they use.

Furthermore, what is clearly missing from the international literature are studies presenting and sufficiently analysing the potential and affordances of AR technology combined with naturalistic approaches (Nilsson et al. 2010), despite the fact that results show that allowing real end-users to interact in a naturalistic setting can also be useful to provide insights on how to design AR applications (Nilsson et al. 2010). There is therefore, a clear synergy here between AR and naturalistic settings, which is worth exploring with potential applications to different domains.

Drawing upon the aforementioned observations, this study aims to investigate the potential use of AR technology combined with naturalistic settings from research papers available in the relevant literature published in peer-reviewed journals (and conferences) and referring to various domains (e.g. education, medicine, digital arts, cultural heritage, human-computer interaction). Specifically, within this context the research questions addressed by this study are as follows:

  1. a.

    Which one of the three types of AR (marker-based, markeless, location-based) is more often associated with naturalistic approaches?

  2. b.

    What are the potential advantages of implementing naturalistic settings on AR technologies?

2 Methods

The electronic databases which were searched in this review included those identified as relevant to education, digital arts, medicine, user interaction and cultural heritage. These searched databases were from EBSCO-host, Scopus and Google Scholar. The search terms (keywords) that were used for the purposes of this study included terms for naturalistic designs applied on AR in different domains of science. More specific terms were also included such as naturalistic approaches in Augmented Reality, ICT and naturalistic approaches, Human Computer Interaction and Augmented Reality, naturalistic approaches in Advanced technologies. Consequently, 33 articles were identified for further analysis.

Methodologically, the 29 experimental articles that were eventually selected satisfied a set of the following clearly defined inclusion-exclusion criteria:

  1. 1.

    The experimental intervention had to be an Augmented Reality strategy that was the primary intervention and, simultaneously, had to present an analytical description of each one of the three particular types of AR (marker-based AR, markerless AR, location-based AR).

  2. 2.

    The article’s researchers had adopt a naturalistic approach and methodology (as manifested in the organization of the field study, the natural interaction etc).

  3. 3.

    The AR intervention had to be used to target functional life skills, social skills or academic performance.

  4. 4.

    The articles had to have been published in a peer-reviewed journal or peer-reviewed conference in the Enlgish language (selected under the advanced search option on EBSCOhost).

3 Results

The analysis of the naturalistic approach to AR from the research papers examined here is articulated on the basis of three forms of AR as defined by Chen & Tsai (2012): “marker-based AR”, “markerless AR”, and “location-based AR”.

Marker-based AR (also known as “Image Recognition”) uses a camera and a visual marker that activates a set of artificial objects and events, which are sensed by a user (e.g. showing a 3D representation matching spatially with the location of the marker). Typical markers include a QR/2D code or labels with a colored or black-and-white pattern that the AR program easily recognizes.

Markerless AR uses sensors embedded in AR devices to accurately detect the real-world environment, such as wall positions and intersection points, thus allowing users to position virtual objects in a real context without having to view an image. Markerless AR systems are based on the recognition of object shapes and rely on natural features to monitor objects and display appropriate outputs to the user.

Location-based AR uses GPS, velocity meters or accelerometers that are embedded in the AR device so as to provide data based on the user’s geographic location. They may also use compass on board to detect the user’s current position and thus promote user navigation by aligning virtual objects with the camera screen.

The research design and main results of articles referring to marker-based AR are shown in Table 1, those related to markerless AR in Table 2 and for location-based AR in Table 3.

Table 1 Synopsis of the examination of the research design and results of articles with researches adopting marker-based AR
Full size table
Table 2 Synopsis of the examination of the research design and results of articles with researches adopting markerless AR
Full size table
Table 3 Synopsis of the examination of the research design and results of articles with researches adopting location-based AR
Full size table

From the examination of these articles therefore, it follows that naturalistic approaches applied to marker-based AR enhance active participation in virtual environments, motivate learners, promote personal involvement in conquering new information, offer different perspectives of the content and arouse interest for knowledge Table 1). Interestingly also, naturalistic designs based on markerless AR foster participants’ collaboration in games, enhance interactivity, can be used in therapeutic purposes, offer panoramic views, visualization and the possibility to examine the role of visual controls Table 2). Moreover naturalistic approaches applied on location-based AR are suitable for designing multidisciplinary applications, can be used for training (i.e. following some trial-and-error procedures) and have the possibility to be designed so as to account for locality and context (Table 3).

It also follows that different types of AR offer different alternatives and advantages for naturalistic approaches to AR (Table 4). Thus, marker-based AR promotes problem-solving, fosters autonomy and improves collaboration, while markerlss AR emphasizes the utilization of haptic technologies and enhances aesthetic experiences of users by being the most commonly type in wearable AR devices. Finally, location-based AR is more suitable for buliding scenarios and it also promotes collaboration.

Table 4 Summary of impacts of the three types of AR on three fields: personal development, social environment, use/exploitation of technology
Full size table

We can therefore suggest that there are exactly three domains that naturalistic approaches apply to AR with the most important benefits: a) personal development, b) social environment and c) use/exploitation of technology (Fig 1).

Fig. 1
figure 1

The naturalistic approach to AR lies at the heart of three key impacts of AR on users and technologies

Full size image

The most observed limitation of AR-based technology systems reported in the studies reviewed here is the fact that it is a technology relatively new to users and consequently, due to their innovativeness, some adaptation time may be required. To resolve this constraint, the role of assistants is important, for providing correct instructions to users, setting up the display room, describing accurately and briefly the process procedure and correctly responding to the users’ questions. The small sample size was another limitation reported by some authors, and consequently future work should focus on larger samples. This should be followed by more research into the creation of AR authoring tools that are more intuitive and user friendly.

Results unveil (Table 5) that the devices used to create naturalistic settings for AR technologies have primarily been motion-sensing input devices (i.e. “Kinect”), smartphones, tablets and computers with video cameras. Motion sensing seems to be a key issue, as the majority of studies in marker-based and markerless AR appear to have utilized motion sensing input devices (e.g. Coimbra et al. 2015; Gilroy et al. 2008; Cai et al. (2017); Zsolczay et al. 2019). A possible explanation for this might be that the majority of the reviewed studies explored new ways by which users interact with their own environment, augmenting it and involving embodiment and more extensive use of immersion. Another interesting result that emerged from the data is that the majority of the studies were in the domains of education, digital arts and human-computer interaction.

Table 5 Characteristics of application used in naturalistic settings applied to AR: technologies, devices and domains
Full size table

Finally, analyzing how the devices used for naturalistic settings for AR relate to personal development, social environment and use/exploitation of technology, it follows that motion-sensing input devices are the kind of technology which appears more beneficial for these fields (Table 6).

Table 6 Summary of impacts of the devices used on personal development, social environment and use/exploitation of technology
Full size table

4 Discussion

The results of the current study can be used in comparison to the exisiting literature examining the combination of naturalistic approaches applied on AR technology in different domains. Since AR is an emerging technology, it is important to provide an overview of the advances and impacts of its use on educational, medical or artistic settings. The potential of naturalistic settings applied on AR technologies is significant because of the benefits that influence the user engagement in practice-based activities, the increase in their autonomy and the wide use of haptic technologies (e.g. Kinect). Naturalistic designs increased motivation, interest in problem-solving, increased aesthetic experiences and sociability, improved collaboration skills and highlight the use of technology. The responsers from real end-users described AR as a useful and pleasant technology when naturalistic settings were implemented. Specifically, naturalistic settings were shown to be useful for educators to recognize the educational potential in their disciplines, for designers of digital arts to support a representation of affective responses that related to aesthetic impressions, for surgeons to simulate needle insertion, for designers of advanced modelling tools, for constructions of virtual representations of archaelogical sites.

Naturalistic approaches in the reviewed studies varied but often included markerless AR (e.g., Gilroy et al. 2008; Kim & Cooperstock 2018). With regard to naturalistic design applied to prototype implementations of AR systems, it is obvious that users evaluated their overall experience as positive and enjoyable regardless of how proactive they were in their interaction with the installation. Fields such as education, medicine, environmental science learning, archaeology, interactive digital arts can become more engaging if AR is combined with naturalistic approaches in which users manipulate virtual objects as naturally as they would manipulate real physical ones. For example a virtual tree whose growth is influenced by the perceived emotional response from spectators is an activity that can facilitate inquiry/discovery learning and augmented interaction with the real world. Nevertheless, more case studies are required in order to understand how to design naturalistic experiences for different topics. Moreover, the majority of the reviewed articles demonstrated several positive outcomes after naturalistic approaches to AR settings in comparison to traditional learning tools (Cai et al., 2016; Martins et al., 2016; Oswald et al., 2015; Brata & Liang, 2019).

5 Conclusion

Naturalistic approaches applied to AR technology can be useful in different fields of science, technology, education and even digital arts. In the present paper, the potential of naturalistic approaches was evaluated with special emphasis on user experience. While this evaluation of research results contributes to a familiarization with naturalistic desings on AR technology by providing evidence of their potential, it also aims to motivate researchers and educators towards adopting these designs in their practice.

This study intends to offer new insights based on experiences of users of AR endowed with naturalistic settings. Three types of AR for naturalistic design have been evaluated, in accordance with the classification by Chen & Tsai (2012): marker-based AR, markerless AR and location-based AR. The implications of this research are that naturalistic approaches positively affect all three types of AR. Specifically, it was revealed that naturalistic approaches in marker-based AR enhance personal development, in markerless AR technology facilitate to foster social-environmental interactions and in location-based AR they bring a clear beneficial effect on use/exploitation of AR technology. Among all domains of applications of AR with naturalistic approaches (education, medicine, digital arts, cultural heritage), the domain of human-computer interaction has attracted more attention in experimental researches.

Furthermore, of the devices that are used for creating naturalistic settings in AR technology, motion-sensing input devices appeared to improve all the previously mentioned fields.

A limitation of this work is that some of the reviewed studies came to positive findings about naturalistic approaches without always identifying whether the benefits of user experience were due to the adoption of specific naturalistic approaches, or due to the advantages of AR technologies in general.

Future research may focus on exploring methods and technologies by which user experience and knowledge construction processes may improve in naturalistic designs for AR settings by further enriching users’ experience. This is because AR designers need to understand how to create AR naturalistic experiences tailored to the particular kind of application they are developing, while also taking into consideration users’ needs.