Keywords

1 Introduction

Augmented Reality (AR) is the fast-growing technology that is used to superimpose digital data such as images, videos and 3D objects on the Physical world. It is a major part of industry 4.0 as it improves how data is accessed and used. AR using a camera scans the environment and provides the necessary information. It can be used to educate and guide people while they work on complex tasks. Extended Reality (XR) extends the working of AR to interact with virtual environments. Internet of Things (IoT) refers to a network of computing devices (nodes) that includes actuators, sensors, software, servers and communication technologies used for connecting and communicating data with other nodes over the internet.

Although AR and IoT are two different fields, both depend on the data from the environment and sensors and thus complement each other. So, this combination opens doors to a wide range of interactive and web-based applications. Considering the advantages and the possible wide range of applications, the proposed system focuses on the usage of AR and IoT-based applications to access Smart Lock.

A smart locking system is an electromechanical locking device that authenticates the user and opens wirelessly. It is an extension of the smart home network. It allows users to access their smart home without the traditional key through interconnected devices to make this smart locking system less susceptible to attacks. To increase accessibility, we are integrating IoT and AR into the smart locking system so that the user can use his/her smart glasses/mobile devices to access the lock without any physical contact.

The proposed system was formulated in the recent light of Covid-19 regulations where the emphasis was placed on contactless smart devices. The proposed system has been designed to provide security while being contactless and less prone to physical tampering compared to other smart locks. And the proposed system is developed to support the upcoming Smart devices such as smart glasses, smart lenses and other AR devices.

2 Related Works

Rauschnabel, P. A et al. [1] AR can give precise instructions and feedback for manufacturing to make high-quality products. There are a lot of AR glasses coming into the market. These glasses are used to give training and guidance for the assembly of complex systems. XR is the least developed technology in the world. VR is most developed as it has been used in many games before.

Gong, L., et al. [2] Hardware parameters for XR are Field of View (FOV) and Frames per Sec (FPS). Human eyes have a FOV of 114 horizontally. VR headsets have a FOV of 90–110. Headsets with smaller FOVs have a tunnelling effect. For FPS, it is recommended that 60 frames per second are good, although 90 frames per second should be strived for. The most common platform for AR VR development used is Unity.

Wang, J et al. [3] Collaborative AR environment improves applicability as it can be used. Multi-user AR collaborative system is composed of MUCstudio, MUC view and MUCserver. MUCview is used for AR experience, MUCserver is used for database and collaborative experience and MUCstudio is used for generating 3D content.

Kassem, A et al. [4] proposed that a central control module should be embedded in the door itself which is connected to the local area network (LAN) of the house itself. This provides a robust mechanism for the door and access to the door is limited through the LAN only. He also proposed an offline system to open the door in case of connection loss. The proposed system uses a master key which is stored in both the smart lock system as well as the mobile application to open it.

Kwok, A. O et al. [5] Tourism is highly dependent on location connectivity and destination accessibility. During the covid period, it was affected a lot due to the lockdown. But here XR comes as a saving grace as not only one can have an immersive experience but also stay safe.

Hadis M. S et al. [6] proposed a smart lock system that uses Bluetooth as a medium of communication between the smart Lock system and its mobile application. The smart lock system is made in a manner so that people with disabilities can also access it without contact. The proposed system given is that the smart lock is embedded in the door and the mobile application communicates through Bluetooth. Whenever the mobile is near the door, the smart lock opens. This system is carried out using two different areas concerning the smart lock system 1) Bluetooth area and 2) validation area. For the door to unlock the authorized user has to be in both the Bluetooth area as well as the validation areas. If an unauthorized user is in the validation area, and the authorized person is in the Bluetooth area then access will be denied.

Masood, T et al. [7] AR is an integral part of industry 4.0 because it visualizes and projects digital information over the physical world and supports human interaction making it more accessible. The aggregate market is projected to reach 75 million dollars in 2025. AR allows access to digital information and overlaying of that info over the physical world. Although the efficiency of AR depends on the task at hand, experimentally it is observed that AR-supported tasks are efficient in terms of timely completion and have less error rate. The effectiveness of the AR system also depends on the work experience of the personnel.

Egger, J et al. [8] The vision of Industry 4.0 is to build cyber-physical production systems (CPPS) which connect the physical and the digital world seamlessly. This led to industries moving towards smart factories, which utilize concepts like predictive maintenance or extensive machine-to-machine communication. AR allows us to interact with the digital world of the smart factory. AR makes huge amounts of data generated by CPPS accessible to humans in real-time and hence is a big part of smart manufacturing.

Danielsson, O et al. [9] Implementations of AR can be categorized as head-attached, hand-held and spatial. Head-attached AR, especially smart glasses, are becoming popular due to their lightweight and hand-free operation. These glasses project information on the physical world. AR smart glasses increase the efficiency of the operators in production lines as they don’t have to stop working to refer to the instruction books, but those instructions can be displayed using AR.

Saidin, N. F et al. [10] AR enables users to interact with virtual and real-time applications, making the learning process more active and meaningful. This also helps in better understanding and increases the ability to retain the information for a longer time.

Evangelos Anastasioua et al. [11] XR has a huge potential in the primary sector of society. As it can be used in many sectors such as aquaculture, livestock farming and agriculture. There are many ways and devices that allow human–machine interaction in XR environments for the primary sector. These use cases can be achieved by using various XR controllers and viewers that allow interaction with XR environments and help users in achieving higher performance and engagement compared to traditional methods.

Pringle, J. K et al. [12] Extended reality can play a major role in learning and teaching in forensics as it can be accessed irrespective of time and apparatus available, can be used asynchronously and repeatedly interrogated to reinforce learning, and helps users in exploring different strategies with lesser time and can be to simulate any physical fieldwork environment and practical classes. However, the Generation of XR virtual learning environments are time-consuming and the educational eGame needs significant user-experience design expertise, computer programming skills, scientific input and, evaluating and refining the product. Which makes the design and production of applications complex and expensive.

Takanori Sasaki et al. [13] Usage of Mixed reality (MR) has brought a revolution in the field of Oral Surgery. Which is alveolar bone grafting for cheilognathopalatoschisis, temporomandibular joint mobilization for ankylosis, resection of a mandibular calcifying epithelial odontogenic tumour, genioplasty for jaw deformity, open reduction and internal fixation for a mandibular fracture. The usage of MR in complex operations has helped surgeons in making precise and more accurate actions while operating. The visibility of the three-dimensional images was good, and preoperative image information could be observed in real-time.

Alyousify, A. L et al. [14] An AR-based software scans Target Images on a printed book and renders 3D objects and alphabet models and sound and sound. The sound and 3D representation of the letter are played as soon as a target image is detected. The target image depicts 3D objects of the alphabet and the sound of the letter’s pronunciation, and an example of a 3D item of a word containing the letter, with the word itself shown on the target image. Augmented reality is in high demand in the educational sector, particularly for educating children.

Pinjala, s et al. [15] proposed a remotely accessible smart lock system using raspberry pi with an HD camera. The proposed system works in the following way: The visitor rings the bell at the door. The visitor’s live video is sent to the owner via the raspberry pi to the mobile application. The owner can willingly let the visitor access his house by typing the preset password to the smart lock system. The owner can also leave a voice note to the visitor by typing the message in the app.

Kim, S et al. [16] User-created automation applets, which are used to connect IoT devices and applications, have gained popularity and are widely available. IoT application network with the data of the IFTTT (if this then that) platform is one of the most popular platforms for the self-automation of IoT devices. Triggers and actions are specified using IFTTT applets, which are usually the set of actions to be performed for their corresponding triggering actions.

Zamora-Antuñano, M. A et al. [17] Three phases of AR: first is scanning and recognition of the environment. Second, the virtual information provided is processed and aligned. Finally, the virtual information is augmented onto the physical world. This can be used to integrate with embedded systems for various purposes. Liu and Li presented an AR solution for drone-based building inspection by integrating UAV inspection workflow with the building information model.

Stark, E et al. [18] The following consists of the workflow to be followed while integrating AR and mechatronic devices. The mechatronic system can be recognized by the AR application by scanning the trigger image from the camera. This Application then connects to the server where the mechatronic system’s data is stored and its twin is stored in the cloud. The data from the sensors of the mechatronic device is sent to the server, and the device twin in the cloud is synchronized. The application downloads the definition of the user interface and draws a graphical interface for control and monitoring of the system by obtaining information about the mechatronic device. The graphical interface in augmented reality can be used by the user to interact with the mechatronic device. Control commands are sent to the server, which sends them to the connected mechatronic device.

Zhu, Z et al. [19] The implementation of a smart lock based on openCV and Efficient Altitude tracking mechanism made the door lock easily accessible, easy to set up, and easy to operate, less susceptible to errors and damage, and made the lock more secure than the traditional Locks. It also provides the user with the data of the person accessing the door lock.

Croatti, A et al. [20] proposed the combination of Web of Things and Augmented Reality is referred to as Web of Augmented Things (WoAT). WoAT can be considered an extension of WoT. Data generated by different things in WoT is typically collected, stored and managed in the cloud. Thus, in the WoAT it should also reflect the information on augmented entities making a massive impact on people working and collaborating. He also sought the use of augmented or mixed reality in the management and diagnosis of several complex equipments to which only a select number of people have access. This eliminates the requirement for installing physical control panels, to which even a regular employee may gain access.

Park, S et al. [21] Brain-computer interface is an emerging technology that helps with the interaction of devices with brain waves. Integrating BCI (Brain computer interface) with AR and IoT can give hands-free operation of smart devices in the smart home. Experiments conducted showed that it took an average of 2.6 s to switch a device on/off and a false positive rate of the switch operation was 0.015 times/min.

Michael W. Condry et al. [22] proposed the usage of a framework for employing control system gateways to address the issues in security issues, paying special attention to a method of connecting IoT devices directly to control system gateways. Multifactor authentication and authorization over an encrypted communication channel are added to increase the security, and “Real-Time Identity Monitoring” is also included to continuously ensure a legitimate user identity.” (Fig. 1).

3 Methodology

Fig. 1.
figure 1

Architecture Diagram

Full size image

An AR based application is made using a trained model to detect a specific target image using Harris corner feature points. The smart door has the target image which is recognized by the AR application. The AR application then uses Simultaneous localization and mapping algorithm (SLAM) to augment the keypad on the target image. SLAM also positions the keypad in a real-time scenario to sync the keypad with the movement of the target image. The keys in the keypad are jumbled so no one can guess the password by the pattern. The application uses the camera to capture the input as the user interacts with the augmented virtual keypad using his fingers. The application then calculates the difference in feature points of the actual image and the real-time image to decide the number entered by the user. This Secret code is captured and verified by the AR application. If the Secret code is correct, an HTTP request is sent to the Blynk platform to switch the value of the virtual pin(V0) to 1. The V0 value is then taken by the NodeMCU. The V0 is attached to the D1 GPIO pin of NodeMCU. If the value is of V0 1 the door will be unlocked else it remains locked. If the input pin doesn’t match the actual pin it notifies the user regarding the invalid attempt to access (Fig. 2).

Fig. 2.
figure 2

UML-Activity Diagram

Full size image

4 Hardware Implementation

The proposed system aims to make a robust locking system. We introduce a door that neither has a door-knob nor a keyhole. This door design makes the door secure from the physical tampering of locks. A smart lock module is attached to the door from the inside consisting of NodeMCU which works in hand with Blynk. Blynk is a platform for IoT applications in which the Virtual pin from the previous chapter exists. A dedicated virtual pin(V0) is constantly synchronized by the NodeMCU so that the given value is reflected during the execution of the program. An Output pin of the NodeMCU is connected to the relay as NodeMCU works in the range of 3 V while the solenoid lock requires a 12 V input signal so this relay is used to handle high-power lock circuits with low-power NodeMCU output. The VCC terminal of the relay is connected to 3V3 NodeMCU, the Common terminal of the relay is connected to the positive terminal of the 12 V DC jack the solenoid lock is connected to the NO (Normally Open) terminal of the relay, the NO relay contact is the pin that is open when the input to relay is LOW. That is If the virtual pin value is 0, the output of the NodeMCU pin is LOW thus unlocking the door and if it’s 1 then HIGH is produced by the NodeMCU thus locking the door (Fig. 3).

Fig. 3.
figure 3

Smart Lock Circuit (Model Door)

Full size image

5 Algorithms

5.1 Harris Corner Detection

Harris corner algorithm is used to detect all the corner points that are in the image; These points are used as feature points to detect the target image. These points are also used in the image difference algorithm to detect interactions with virtual buttons.

Algorithm:

Step 1: Take the input Image.

Step 2: Convert the input image from colour to greyscale.

Step 3: This greyscale is converted into a matrix holding the intensity in terms of numbers (0–255) of each pixel

Step 4: A corner in an image is detected when the differentiation of image intensity with respect to the x and y directions is different. I.e rapid change of image intensity in two different directions.

Step 5: Visualize each corner using a marker (Fig. 4).

Fig. 4.
figure 4

Harris corner detection

Full size image

5.2 Simultaneous Localization and Mapping (SLAM)

Algorithm to track feature points and Augment digital information. This algorithm is used to Map Augmented data onto a set of feature points so that the information is localized to that set and does not randomly appear on the image. This algorithm also helps in tracking the image dynamically and augments the data in an efficient manner when there is a change in camera distance, angle etc.

Algorithm:

Step 1: Take real-time input from the camera.

Step 2: Detect the image from the real-time input using harris corner.

Step 3: Map and augment the virtual buttons to a set of distinct localized and predetermined feature points.

Step 4: Update the augmented virtual buttons in each frame with respect to their corresponding feature points (Fig. 5).

Fig. 5.
figure 5

The following figure shows the input captured by the application through AR interactions.

Full size image

5.3 Image Difference Using Feature Points

In this module the differences in the actual image and the real-time image when the finger interacts with a virtual button. The idea behind this module is that while the finger is kept between the camera and the image, it hides some of the feature points. These hidden feature points are mapped to a particular virtual button. From the difference, the hidden feature points can be highlighted and the corresponding virtual button is entered.

Algorithm:

Step 1: Take real-time input from the camera.

Step 2: Extract the Feature points of the image using the Harris corner algorithm.

Step 3: When the user uses his finger, the finger covers a set of feature points.

Step 4: This new image with the finger is captured by the camera and compared with the original image. The comparison results in feature points that are hidden. The results are bordered with rectangular areas.

Step 5: Each set of feature points in the image is mapped to a number. The application then uses this result to decide the corresponding number pressed. Each frame is taken as an input.

Step 6: For example, if the area covered has the number 7 then the application takes the number 7 as input

6 Result and Discussions

6.1 Flaws in Existing Smart Locking Systems

Existing smart lock systems use RFID, Bluetooth, keypad, biometric system etc. These technologies can be eavesdropped on, intercepted or tampered with to create a Denial of Service attack (DOS). Due to intractable input devices and interceptable communication methods, the existing smart locking devices pose multiple security threats.

6.2 Advantages of the Proposed System Over the Existing Systems

A single application to access multiple doors with a smart lock. Unique images on the door are used to identify each door. The intractable AR application is to achieve contactless door access. Applications can be deployed on various platforms such as android, iOS, windows, smart glasses, smart lenses etc. Doesn’t require the user to carry a key or an access card. Usage of the Internet to access the smart lock minimises the threats from eavesdrops and interception of communication. Silent and less access time. Safe from external weather and physical threats. Implementation of Two-factor authentication to provide better security (something you have - user dedicated Mobile application, something you know - passcode).

Given below are images of working prototype of the proposed system (Figs. 6 and 7).

Fig. 6.
figure 6

The following figure shows the input captured by the application through AR interactions.

Full size image
Fig. 7.
figure 7

The following figures show the accepted and rejected status in the text field when correct and wrong codes are entered respectively.

Full size image

7 Conclusion

The incorporation of AR with IoT devices has made the smart locking mechanism more accessible, interactive and less prone to physical attacks. The authorization to access the lock is limited to the authorized users, and the physical interaction has been eliminated to make “A lock with no holes”.The usage of AR made the smart lock contactless, making the lock hidden and limiting exposure to the outer environment This decreases any possible ways of physical intervention hence decreasing the chances of intrusion. The smart lock works on two-factor authentication requiring a mobile application and password for access. Implementation of access using AR and IoT has made the lock easily accessible without a key.

AR is a niche field with huge potential for innovation and improvement. Smart devices like AR smart glasses and AR contact lenses are in development, which gives a boost to our project as wearing these devices the virtual button can only be seen by the authorized user and no one else. The interactiveness of the virtual buttons will also improve in the coming future owing to the technological advances being made. Following the user requirements, many more user-friendly features can be added to the project just by making minor changes such as Remote unlocking, Track of lock usage, and Intrusion detection. Security can be increased by adding advanced authentication methods authentication.