1 Introduction

The development of Internet of Things (IoT) technology has made it possible to connect various smart devices together through a range of communication protocols. The number of devices connected to the IoT is predicted to grow at an exponential rate, with forecasts predicting that there will be around 26 billion connected devices by 2020 [1]. These devices will lead to a wide variety of services from multiple sources such as monitoring sensors, surveillance cameras and actuators. The management of the services provided by these devices is recognised as one of the most important areas of future technology and has gained wide attention from research institutes and industry in a number of different domains including transportation, healthcare and emergency response [2, 3].

Both service providers and consumers need to be able to engage with IoT applications in a context-aware manner where they sense and react based on the environment and user context. Augmented reality (AR) provides an intuitive interface to IoT as it allows for context as well as superimposing virtual information in the real world such as in cognitive buildings [4]. Service consumers can interact naturally, in a visual manner with suggested nearby services, which are handled by a context-oriented middleware. Service providers can use AR to debug and repair failing IoT devices, by showing key QoS metrics of the services provided by the device such as the response time and throughput. Device-related metrics such as the temperature, CPU and memory usage of the device can also be shown. The internal components of the device can be connected objects, which will allow the field operator to see a specific visualisation to repair the faulty internal component of the IoT device that is not operating correctly [5]. This allows for easy repair and a return to a suitable operating performance.

IoT applications can be used in a range of domains that have different QoS requirements based on the sensitivity and criticality of the application. IoT application QoS can typically be categorised as best effort (no QoS), differentiated services (soft QoS) and guaranteed services (hard QoS) [6]. In the hard QoS case, there are strict hard real-time QoS guarantees. This is appropriate for safety critical applications such as monitoring patients in a hospital or collision avoidance in a self-driving car system. Soft QoS does not require hard real-time guarantees but needs to be able to reconfigure and replace services that fail. This could be a routing application, which uses air quality, flooding and pedestrian traffic predictions, to provide the best route through the city. If one of the services is about to fail, the application should be recomposed using suitable replacement services. The final case is best effort, where there are no guarantees when a service fails, such as a simple atomic service that measures the temperature in a house. For hard and soft QoS AR could provide an intuitive way to alert users that a service may be about to fail and that they may need to retake control especially in the self-driving car scenario. Forecasting the failure of IoT service can be accomplished through the use of LSTM networks [7].

The remainder of the paper is organised as follows: Sect. 2 outlines some of the recent developments in AR that will reduce the barrier to entry and make these applications much more common in an IoT environment. Section 3 demonstrates a service-oriented middleware that can be used to provide context in an IoT environment and a deep edge architecture that provides a scalable solution for AR applications in IoT. Section 4 demonstrates some of the augmented reality applications that we have developed and discusses how they can be used for both service providers and consumers in IoT. We also create some projections of how users may interact with future context-aware applications. Finally in Sect. 5 we outline the conclusions from this paper and some of the open research questions that need to be explored.

2 Augmented Reality in IoT

AR has quickly evolved in recent years and there is a range of commercial software and hardware that can allow for the creation of AR experiences. Today, there are a variety of APIs at the consumer level that makes it possible to create AR experiences with most of them making use of mobile technologies such as smartphones or tablets. Apple has recently launched ARKit [8] that allows building AR environments with little knowledge of the technology behind it. Another example that we use in our demonstration is the Vuforia developer library for Unity, which includes Google’s ARCore allowing AR applications on all modern android devices [9]. This will allow for much more devices to interact with IoT services through an AR interface and will encourage developers to create AR based IoT applications.

With the wide penetration of smart phones and the new APIs released by Apple and Google we project there will be an increase in mobile augmented reality in the next few years. The use of mobile phones as augmented reality visors will be a necessary step to allow the development of applications and to evaluate the effectiveness of AR in IoT. The transition after this can be to a number of alternative head-mounted displays such as Google Glass, Microsoft Hololens or Magic Leap Lightwear. These head-mounted displays allow for hands free interaction with IoT services and objects, which will be useful for field technicians to interact with devices and view visualisations showing which connected components to replace in order to repair a smart device.

Considering the enormous potential of AR and IoT and the large amount of research that has been invested in each technology, the integration of both components is still at an early stage. Most current approaches show characteristics of objects without taking into account additional context such as the user preference, nearby object/locations, time, weather, etc. Some proposals such as a Sentient Visor infer a high level context of the user using a context interface server to alter the manner in which information is displayed to the user [10]. However, the application of a smart watering plant is simplistic and only needs the local sensor information, also a central server for reasoning don’t take into account the vision of IoT as intelligent objects. In our demonstration we show more advanced examples combining dynamic and heterogeneous services from traditional web services and IoT services and using additional context information to provide intuitive applications. More recent work such as ARIoT provides a scalable AR framework for interacting with IoT devices but focuses on targeting objects rather than derived context through connected objects [11]. Other approaches such as Second Surface, where users are able to create, tag and share data around everyday objects are implemented through a remote server in the cloud [12]. The deep edge architecture that we describe in Sect. 3 allows all the analysis to take place at the edge of the network reducing response time and jitter and creating a more immersive AR experience.

3 Context-Aware Middleware

Figure 1a shows a context-aware service oriented-middleware that can be deployed on the deep edge architecture in Fig. 1b to manage the registration, execution and context of IoT services in the city. The middleware manages the context of users in the environment in a distributed way across each of the gateways. When a user requests an application the middleware first receives the user request at the request handler, which establishes a request/response communication channel with the user and forwards the request to the context manager. The context manager is responsible for acquiring and maintaining metadata about the services and the environment to support smart service management. The metadata includes service attributes (e.g., location, type, domain, etc.), QoS properties (e.g., response time, energy consumption, etc.), urban-context (e.g., places and their meaning in the city, points of interest, etc.), and user preferences (e.g., user feedback, QoS experience, user behaviour, etc.). Service attributes and QoS properties are acquired from providers and the monitoring process. This data is stored as part of the service descriptions. Urban-context is acquired from Open Street Maps (OSM) and formalised using ontologies in OWL format. User preferences are acquired from the discovery and composition process. This data is stored as independent JSON documents in the service registry.

Fig. 1.
figure 1

Context-aware middleware and architecture for deep edges

Full size image

For example, if a user is in a car dealership and searches for Jaguar we can search nearby IoT services to see if there are any Jaguar cars available at this dealership with additional context-aware information such as the price, year, fuel efficiency. If a user is at the zoo and searches for Jaguar, then we look for any nearby IoT beacons that are advertising a Jaguar exhibition. Additional context-aware information can then also be supplied such as the amount of jaguars, where they are from and what food they eat. This component can process traditional user information from a request such as the location and time but the component can also be extended to run a deep convolutional neural network to process the image sent by the AR device in real time at the edge [14]. This can add additional specific information about the contents of the image that the user is currently looking at. This would allow for additional context-aware information not available in previous IoT applications, providing more immersive and user-specific applications with an AR interface.

The other main components of the middleware as shown in Fig. 1a are the Service Registration Engine (SRE), the Service Discovery Engine (SDE), the QoS Monitor and the Service Composition & Execution Engine (SCEE). The SRE registers the available services in the environment according to domain. The SDE uses a backward-planning algorithm to identify concrete services, which can be used to satisfy the non-functional requirements of the request and sends this list of services to the SCEE. The QoS monitor is used to monitor these services and identify possible candidate services to switch to if one of the services begins to degrade, using a matrix factorisation based collaborative filtering algorithm in the prediction engine [15, 16]. The prediction engine also contains an algorithm to forecast future QoS values of currently executing services, to identify that a service may be about to fail using an LSTM based neural network and service recomposition is needed [7]. The SCEE uses an optimisation algorithm to maximise the available QoS in the response.

The “deep edge” architecture, which we define in a previous work, allows for much more processing to take place at the edge of the network, without access to the cloud [13]. These devices can be arranged as a network of gateways as shown in Fig. 1b. Here the embedded GPUs (Jetson Tx2) have a number of services registered on them that would be distributed throughout the city such as traffic and weather data that can be used for a range of IoT applications. The additional processing power in deep edges compared to other traditional IoT gateways (e.g. raspberry pi) allows them to run the prediction engine in the middleware locally to make predictions for IoT services in the environment instead of offloading this to the cloud. This makes the IoT applications in the environment much more reliable. As these devices are one hop away from users they can provide much lower end-to-end latency and jitter. This makes them suitable to handle augmented reality applications that require reduced latency and jitter with the additional benefit of being able to take advantage of the GPU on each device for increased graphical processing for higher resolution augmented reality applications. Application can offload the graphical processing from a users phone to a GPU to make the applications higher resolution, while also keeping the short response time available using edge networks to provide a fully immersive experience.

4 Demonstrations

In this section we give a demonstration of some of the use cases of AR in IoT. We consider both service providers and consumers to show how AR provides an easy interface to the virtual information of connected objects and services.

Fig. 2.
figure 2

AR showing QoS of service on device (Color figure online)

Full size image
Fig. 3.
figure 3

Phone used for AR experience

Full size image

4.1 Provider

Service providers want to be able to interact with devices that they are using to provide services in the environment. Figure 2 shows the augmented information for the sensor on a stones marker. In this case this device provides a simple temperature service that is used by other smart applications. By interacting with the device the field technician can identify that the response time of the service is quite high as shown by the orange background on the text. The other key QoS factors are normal as shown by the green background. A time-stamp is also provided for logging when services have failed or are acting abnormally.

The device can further be interacted with to identify the problem and choose a suitable replacement part. Figure 3 shows how a field technician may currently activate the AR experience by using a mobile phone. In the future with the reduction in cost of headware accessories such as Magic Leap Lightwear this will allow for hands-free operation. Hands-free operation allow for more complicated repair operations on larger devices [17].

4.2 Consumer

For consumers, AR provides a low entry barrier for interacting with IoT services. The management of the services will be handled by a context-aware middleware that can make suggestions to the user of relevant applications that may be available in the area. It will manage the registration, QoS, composition and execution of the services. The user just needs to interact with the augmented suggestions on their headset to add additional virtual information to their environment. In this section, we provide some scenarios of consumers interacting with IoT services in the future and show how AR could be used to augment our experience by adding context-aware information from IoT services. These projections provide a top-down view of the possibility of combination of IoT and AR.

The first scenario is a tourist walking into a smart campus in Trinity College Dublin. Location based AR is used to provide additional context aware information of the buildings that the tourist is looking at, as they walk through the campus in Fig. 4. The contextual information is taken from a web service, which provides some details on the design and superstition of the Campanile on the campus. As the person has never been to the campus before and has visited other tourist locations in the city, the middleware can provide suggested locations to visit next such as the Book of Kells. Additional information such as the waiting time in the queue can accurately be measured by IoT devices located at the attraction and reviews retrieved from a web service.

We can also see some advertised services in the environment that the user has not interacted with yet. These IoT beacons provide further information about two statues in the front square of George Salmon and William E.H. Lecky. These advertised services have predicted response time using the IoTPredict algorithm [16] as that is the QoS metric that the user most interested in. The metrics are coloured green to show the services have acceptable response times. This first example shows the importance of context in the scenario as if a student is walking to class and looks at the same building they may not want additional information about the history and design of the building and nearby tourist attractions but may be interested in their calender application to find the location of their first class and the length of the queue at the coffee shop.

Fig. 4.
figure 4

Tourist interacting with services in AR

Full size image
Fig. 5.
figure 5

Farmer interacting with services in AR

Full size image
Fig. 6.
figure 6

Commuter interacting with services in AR

Full size image

The second scenario that we envisage is a smart agricultural scenario. In this case, a farmer is making hay to use as animal feed during the winter months in Fig. 5. The grass has been cut and the farmer needs information about the weather and moisture content of the grass in order to make hay. The farmer can retrieve a seven day weather forecast from a service provider but can also use additional contextual information provided by dynamic and heterogeneous sensors deployed across the field. These sensors provide additional information about the current moisture content of the grass and the soil temperature. Based on the additional context information provided by the services the farmer has an estimated time when the grass will reach the 18% moisture content needed for large round bales from the current 85% of 6 days. The bailing equipment, which is also a smart device can provide a status update showing that it is ready for use or that some replacement parts must be ordered (Fig. 6).

The final scenario that we consider is a smart city scenario of a commuter planning a trip home. In this case IoT sensors in the surrounding location have detected an extreme weather event and the presence of snow and ice on the roads. The commuter is looking at some city bikes available at a bike stand. Based on the additional information from IoT services in the environment the user is given a weather warning that it is unsafe to travel on the bikes due to the snow and ice on the roads. Initially the application shows the bus timetable but when the buses are cancelled due to the weather the application dynamically adapts and gives the user the option to book a taxi or uber. This example shows the importance of context and managing heterogeneous and dynamic services.

5 Conclusion and Future Work

In this paper we have shown how both users and providers can use AR to interact with IoT devices and services. AR can be used to quickly debug devices as well as providing an easy way to interact with services in a local environment. In our demonstration of some possible future AR experiences we have shown the importance of context and having accurate information about the surrounding environment from nearby IoT devices. Additional information about the user is also beneficial, for example whether they are a student or a tourist to further personalise the experience. These figures demonstrated the possibilities when combing AR and IoT and we hope will encourage further research.

There are a number of open research questions in the combination of IoT and AR such as the architecture to run these applications due to the low latency and jitter requirements. We have described a deep edge architecture in our previous work [13] and plan to conduct experimentation on a campus wide level to show the suitability of this architecture for AR applications in IoT. This will provide an experimental platform that we can use to conduct additional experiments with students and tourists that investigates additional factors, that may effect the quality of experience such as the resolution, suitability of augmented information and response time of the application.