Development of a visual to audio and tactile substitution system for mobility and orientation of visually impaired people: a review

Abstract

According to the World Health Organization (WHO), visual impairment is one of the most common problems affecting approximately one sixth of the world's population. It was found that 39 million of these sufferers have lost their vision completely and require supervision from other people to do their daily chores. Until today, several methodologies have been built to provide effective solutions to visually challenged people, mainly in terms of navigation. The assistive technologies developed for the visually challenged are generally very popular due to the benefits provided, but these technologies are limited in many aspects. One of the main problems with the technologies is that they are non-adaptable and cannot adjust to the changing needs of the participants. The demands of blind people are constantly growing, and there is no powerful technology that can meet all of these demands at the same time. The major constraints with assistive technologies are cost-effectiveness and user-friendliness. This paper presents a complete review of the assistive technologies introduced in the literature to deal with the navigation problems of blind people. Each and every literature work focuses on the requirements of the blind, and several features are considered accordingly. Thus, the presented review provides a detailed description of the evolution of such assistive technologies and the improvements brought in to satisfy the users. Apart from this, the complexities and limitations associated with the technologies are also unwounded to provide a clear perspective on the current scenario for future developers and researchers. Several constraints associated with different assistive technology are interpreted, and this review ends with the insights drawn from the study along with future scopes.

1 Introduction

Vision and sight are essential to human beings because they permit people to interact with their surroundings, keep them safer, and make moving from one place to another easier. People affected by visual disorders and defects require aid to triumph over daily tasks like moving and surviving unknown environments. Visual impairment indicates a loss of vision or blindness, or a significant reduction in vision that occurs as a result of trauma, disease, or a degenerative condition that cannot be reversed with any medical treatment or therapy. Compared to normal people, visually impaired people have great difficulty carrying out their daily activities and face several challenges in their lives [16]. Also, people with blind face several challenges in their daily lives, like difficulty finding their materials, arranging clothes, difficulty walking on the road, difficulty travelling, etc. A recent declaration from World Health Organization reveals that 1.3 billion people in the entire world are visually impaired. Each year, the percentage of visually impaired humans increases by 2 million. A prediction has illustrated that the number of blind people may be doubled in 2021 [63]. The increasing number of visually challenged people around the globe specifies the urging need for a proficient solution that helps them with daily activities and navigation. Visual impairment can be categorized into two levels where in the first level, the visual acuity is referred to as 6/18 or equal to 6/60, with the best correction indicating low vision. The second level, or blindness, provides a visual acuity of 6/120 with the best correction possibility in the best eye. Visual impairment is the imbalance of neurological or physiological components, termed the condition of discernment deficiency.

A recent estimate by the world health organization (WHO) states that approximately 285 million people around the globe are visually impaired, among which 39 million are completely blind without any known cures. The remaining 246 million people are identified to have vision problems, including low vision due to some medical condition. Without any additional interventions or solutions, these numbers are estimated to rise above 75 million blind and 200 million with vision problems that can be either corrected or not. Vision loss includes serious complications in one’s life, including performing activities such as information access and communication, recognizing communication cues, obstacle avoidance, navigation, etc. Although various evolution in innovations is encountered nowadays, but visual impairment remains a critical problem. Recognizing non-verbal communication cues such as hand gestures, facial expressions, body posture, eye contact, etc. and differentiating inanimate and animate objects in the surroundings is a major challenge. Several reports have also confirmed that vision loss increases the risks of individuals in unintentional injuries such as falls, traffic-related injuries, occupation-related injuries, etc. These problems result in significant physical and emotional distress among the individuals leading to substantial health care costs. In order to overcome the daily challenges of blind people, various assistive devices have been developed in this modern world. These assistive devices provide many benefits to visually impaired people in their normal life. The latest version of assistive devices allows blind people to perform several things like browsing the internet without the help of others, improving reading capability, providing safety, preventing accidents and so on. Several researchers have concentrated on this problem to develop assistive tools for blind people [2]. The two essential tools for people with blind deficiency people are white canes and guide dogs. The performance of these tools is restricted by coverage, capacity and speediness, which are mainly approachable to people with eyes for navigation. The walking assistant helped blind people with their daily activities related to location and navigation, which were included with mobility support in the 1960s.

The assistants support people with visual deficiency by finding and placing obstacles in the environment using sensors that can detect the sense from the exterior area [36]. The technology of visual assistance is classified into three types: substitution, vision enhancement and replacement. The enhancement of vision and substitution are similar in functionality. The input image is processed and displayed through a monitoring device to enhance vision [1]. The manufacture of the vision substitution is acquired through vibration or with the assistance of alarming devices which can make a noise. Compared with vision, it has poor information capability. The visual sensory substitution can acquire different forms and may be arbitrated by a device [12]. The classical white cane technology is one example of visual sensory substitution devices [19]. Without the help of vision, the information of the object is sensed using the tactile receptors of the hands. As compared with vision, this approach is highly limited. A cane’s spatial and temporal resolution is the worst, and the possibility for error is respectively large. The research on visual substitution has concentrated on both the auditory and tactile senses [53]. Basically, blind people face several challenges in their daily life, making it more difficult to walk from their homes to nearby regions. Thus, daily activities without eye vision are critical issues to survive in the world. The displays of tactile assist the visually impaired fashion designers and shoppers to feel the materials of cloths through the internet to help with purchasing decisions online [27]. The tactile display system gives improved realism in a medical training simulation, which helps the user categorize the varied types of tissues. Tactile-vision substitution systems (TVSS) are constructed to exhibit visual information to people with visual deficiency [57]. In a common system, the camera receives visual information and is transformed into a tactile representation on a two-dimensional pin array.

In general, travelling in unknown exterior environments has been difficult for several people with visual deficiency. The advancement of intelligence systems, the electronic travelling devices for blind people triggers high attention from the research organization. The increasing technology of assistive devices still requires more developments in several fields all over the world. A global navigation satellite system (GNSS) is more accurate for surpassing the approach to locating. But the reduced cost of portable GNSS devices is adequate for the requirement of people with blindness. Ordinary persons tend to recognize each place and object through eye vision, and visual images could be employed for localization to aid navigation [24]. Using the images to localize is called visual localization, which is to recover the related images of a provided query image from a dataset. Recently visual impairment has been mainly affected by old age people because of several reasons. Training is utilized to aid blind people in travelling individually with advanced mobility and orientation skills [47]. Orientation is recognizing the individual’s position in affinity to the environment. Also, mobility is said as the capability to move efficiently and safely [17]. The training in orientation and mobility directed blind people to utilize their low vision and another sense to move around the world.

There are numerous literary works to present a better and more effective solution for blind people to deal with their daily chores. The problem arises with the identification of the best solution that can provide effective remedy to individuals from different aspects. It is always a problem to choose between multiple options as one can provide a better solution but may include several other constraints and limitations that are not satisfactory. Since a solution for visually challenged persons in terms of navigation is important, this work reviews the existing assistive technologies available for visually challenged people to help them with navigation. The main motivation behind this review paper is to provide a clear insight into the technologies available and the changing needs of blind people that require much attention. Also, the paper deliberately reviews the pros and cons of the existing methodologies to provide a clear idea for future developers and researchers in this field regarding the improvements that can be made. The major constraints with the existing devices and the places where improvements can be brought in must be elaborated. Thus, this review paper reviews the most effective assistive technologies available to show the requirements that must be followed soon for further improvements.

This review paper is organized as follows: Section 2 describes the review objectives, Section 3 deals with the assistive technologies of blind people, Section 4 discusses the visual to audio substitution systems of the visually impaired, and Section 5 presents the tactile substitution system of blind people, Section 6 deals with computer vision-based assistive system for blind people, Section 7 deals with wearable assistive systems, Section 8 discussed the mobility and orientation of visual disability people and Section 9 provides the conclusion of the entire paper.

2 Review objectives

The visually impaired and blind people are our community’s important characteristics. Due to the growth of eye deficiency, many people worldwide are visually impaired, and it has become a challenge to overcome. Blind people face several challenges while performing their tasks of daily life like detection of currencies, reading labels on each product, finding the appearances of an object, operating electronic devices etc. In recent decades, various assistive technologies have been implemented to reduce the struggles of blind people. The assistive methodologies of the visually impaired are mainly concentrated on orientation, identification, mobility, detection of the path and object recognition. Considering the eye deficiency, many wearable devices, such as smart canes, retinal implants, tactical displays and smartphone devices, are developed. To make the assistive technologies comfortable, pervasive, non-obtrusive and useful for people, the user experience understanding of the technologies is highly critical. Deep user study and evaluations are required to build an assistive technology for blind people to iteratively evaluate user satisfaction and improve the technology accordingly. Also, to assess the technical performance of the systems, the offline exams and evaluations of computer vision techniques introduced by assistive technologies are of great significance. In general, the assistive technologies introduced in the literature lack usability assessment. These assessments under complex scenarios are required to prove the effectiveness of the technology in assisting visually impaired people. Thus, before reaching a final solution in terms of user adoption and technical accuracy, it is suggested to follow several developmental loops along with user evaluations. Also, each component associated with the technology is required to be evaluated with different combinations of these components to identify the improvements required to be made. Through evaluations and analysis, the new and crucial requirements in the systems can be identified from both user and technical perspectives. This research intends to review the recent technologies which are assistive to visually impaired people. In this paper, the visual to audio and tactile substitution systems of the visually impaired are discussed in detail.

To present a wholesome review of the topic, this review methodology is partitioned into three stages: planning, identifying, and reporting. At the planning stage, the researcher is responsible for figuring out the need for the review and the research queries to be addressed with better explanations of the review. The identifying stage predominantly includes the searching strategy followed, articles and keywords utilized in the search and the analysis of several methods. In the final reporting stage, the researcher should elucidate the complete concept of the research domain with the methodology and outcomes obtained. The rest of the paper includes a complete and clarified review of the considered sub-headings. Initially, the queries related to the tactile substitution systems for the mobility of visually impaired people are identified. The exploration strategies followed, and the papers analysed are presented in the next step. Lastly, the overall substitution systems for visual impairment, the aim behind the development of such systems, analysis with other systems, techniques followed, datasets used, and the parameters assessed in other papers are elaborated. The methodology followed behind this review is displayed in Fig. 1.

Fig. 1

Methodology behind the review

2.1 Research queries

The presented review focuses on addressing or answering the following research queries:

• 1) What are the topics that have been researched in the context of assistive technologies for the mobility of visually impaired people?

• 2) What are the mechanisms and approaches followed in the considered area of research?

• 3) What are the devices that have been developed in the sense of visual to audio substitution systems to help with the mobility of blind people?

• 4) What evaluation scenarios followed to identify the actual requirements in the technologies from both the user and technical perspectives?

2.2 Searching strategy

Based on the queries formulated, this work follows three types of search strategies: word-based search, creeping-based search and inclusion/exclusion-based search. In the first type, words related to the research topic are utilized in an inbuilt source, namely google scholar, to identify relevant papers. The previous research works established in the considered domain are explored in the second search strategy. In the final searching strategy, evaluation is done by checking whether the papers read are relevant to the domain.

2.2.1 Word-based search

Specific words related to the considered topic are used as keywords to search for the related papers. Some examples include the following:

• i Assistive technologies for the visually impaired

• ii Different substitution systems for the visually impaired

• iii Object detection system for blind people

• iv Maintaining mobility and orientation of visually impaired

2.2.2 Referred resources

The existing works presented in recent decades are discussed in this review paper. The papers referred for writing this review are mostly journals and conferences that are accepted by the literature search engines and databases. The sites referred to are listed below:

• i Springer

• ii Elsevier

• iii Research gate

• iv Science Direct

• v IEEE Xplore

• vi Applied sciences

The articles are selected based on specific headings, and inappropriate headings are rejected. After that, the most needed papers are selected and reviewed properly.

2.3 Study preference

Thousands of literature papers are downloaded and evaluated using the mentioned search strategy. Some irrelevant papers are discarded, and the remaining papers are considered for further evaluation. Although some papers are discarded, an exact filtering methodology is required to filter out only the most relevant papers from the available publications. To this end, this work follows some inclusion and exclusion rules as mentioned below:

• 1) Inclusion rules

  • a Discard the publications that are out of range and include only the required publications within the year range 2015–2023.

  • b Include only the publications related to assistive technologies for visually impaired people and discard the irrelevant works.

  • c Include only the publications that are published in the English language and discard the works from other languages.

• 2) Exclusion rules

  • a Exclude the works without any details regarding a prototype or proposed methodology.

  • b Exclude publications that are completely based on only literary works and surveys.

  • c Exclude the publications without any clear information about the methodology followed.

2.4 Quality validation

The downloaded papers included in this review paper are validated before reference. Certain quality validation tools ensure the paper is relevant to the research domain. These tools are presented below:

• i Does the selected paper follow the correct flow of writing?

• ii Does the paper identify the exact contribution of the considered research domain?

• iii Does the paper include sufficient information regarding the assistive technologies for visually impaired people?

• iv Does the paper provide a brief overview of the abbreviations and terminologies utilized?

• v Are the experimental validations and outcomes explored well?

• vi Does the methodology result in optimal performance under different experimental scenarios?

3 Assistive technologies for visually impaired people

Assistive technologies were established in the 1960s, intended to resolve daily troubles faced by people who are blind. The daily difficulties of the visually impaired are related to navigation, the transmission of information and orientation assistance corresponding to mobility assistance. The assistive technologies are classified into three types such as electronic travel aid (ETAs), position locator devices (PLDs) and electronic orientation aid (EOAs).

3.1 ETAs

ETAs are devices which can capture information about environmental surroundings. The captured information is transmitted to the user via sonar, sensor cameras and laser scanners. Based on the National Research Council, ETAs have several rules which are followed as,

• i Finding barriers around the user's body from top to bottom.

• ii Providing guidelines to the user about surfaces with textures or gaps.

• iii Finding obstacles to various items.

• iv Presenting information on the distance between the user and the barriers with important direction guidelines.

• v Giving information to provide the capability of self-orientation of the surroundings.

3.2 EOAs

The EOA devices provide the pedestrian with directions in unknown environments, and the instructions of EOAs are given as,

• i The routes are represented to choose the suitable path.

• ii Detecting the path to accurately compute the user’s current place.

• iii Affording mobility guidelines and signs about the path to instruct the user and maintain their knowledge about the surroundings.

3.3 PLDs

The PLD device determines the precise location of its user like device, which utilizes GPS technology.

Xiao et al. [71] presented assistive technologies to instruct them to travel individually by considering visually impaired people. The author established an approach for context-aware navigation services for blind people. The intelligence system is developed by establishing the operational procedure to aid blind people. The system’s ability is demonstrated with the help of pilot evaluation. Assistive technology is generally considered a methodology constructed for elderly people or the visually impaired. The assistive technologies for visually impaired people are highly concerned with equipment, apparatus, technologies, systems, and services performed to avoid several social and physical obstacles to individual lives as same people of the community. In recent years, innovations in wearable technology, computer vision, medical interventions and multisensory studies have supported the growth of several assistive methodology solutions for non-invasive and invasive.

Various studies based on assistive technologies for blind people have mainly concentrated on object recognition, navigation and mobility. To obtain a robust computer vision-based localization, an assistive navigation system was developed for blind people [72]. In this, a dual channel human–machine communication scheme is introduced, which can provide accurate guidelines with the assistance of macro long term planning through audio. This framework helps the visually impaired in troubled environments and prevents barriers with frequent instructions. A mobile wearable context aware indoor assistive navigation system was introduced in [45] to avoid obstacles for the visually impaired. This model traces the objects presented in front of blind people by using a depth sensor. A speech-to-audio interface provides realistic instructions and alert signals using a priority-based scheme to minimize the user’s cognitive load. The fundamental robotic assistive technology is shown in Fig. 2.

Fig. 2

Principle of basic robotic assistive technology

3.4 Essential electronic devices for visual disabilities people

Various electronic devices are present at several services for people with eye deficiency based on the gathered data from the environmental surroundings. The information is collected through the sonar, camera sensors, laser scanner, etc., and forwarded to the individual via audio or tactile format. The system needs some of the common features to provide improved performance. These features are important to compute the reliability and efficiency of the electronic device, which gives orientation and navigation services for visual disabilities people.

3.4.1 Smart cane

A smart cane is one of the electronic devices for travel purposes that fit on the white cane’s handle. This smart cane is an advanced version of the white cane which rejects the drawback of the white cane by tracing the hanging barriers. The smart canes capture the information of the presented barriers and deliver it to the user. The mentioned device uses ultrasonic sensors for tracking, and a microcontroller controls the system. The information of several locations is stored in the memory card inserted in the global positioning system (GPS). It is operated on rechargeable batteries that can be recharged using an AC adaptor or USB. The smart cane devices contain vibration motors that aid in providing feedback and are also user-friendly. A microcontroller based smart cane was developed in [34] in which several difficulties faced by blind people are reduced by limiting the obstacles. The developed smart cane has a water sensor, ultrasonic sensor, GSM modules and GPS modules. The cane is structured to be foldable; thus, it is easier to handle the user. A smart cane outdoor navigation system was established in [15] to assist visually impaired people. The quality function deployment (QFD) approach transforms the user’s demands into system features. The features of the developed navigation system are increased cane, pulsing magnetic apparatuses and frequent metallic trials. People with eye disabilities can capture the metallic trial via the enlarged smart cane. This method helps the visually impaired to travel individually. At the point of interest (POI) on sidewalks, like turning and another determination, the pulsing magnet apparatuses would be established to instruct the users via vibrational data. The outcome of the smart cane is based on collecting information to generate audio information via the speaker to the blind one. Moreover, hearing disabled people have particular vibrator gloves afforded with the canes. It has a particular vibration for every finger, and all the fingers have a specific meaning.

3.4.2 Eye Substitution

Various devices are developed to behave as an eye substitution for visual disabilities people, which aids in navigation and direction. The aspect of Android application is to utilize advanced GSM, GPS and GPRS to obtain the blind people’s location and make perfect instructions. The embedded device has dual HC-SR04 ultrasonic sensors and three kinds of vibrator motors. Figure 3 illustrates the eye substitution device for visually disabled people.

Fig. 3

An example of an eye substitution device [23]

The ultrasonic sensors transmit a set of ultrasonic pulses. If the barrier is traced, the sound will be forwarded to the receiver. The motors are started by processing the measurements of the ultrasonic sensors by the microcontroller, which also gives reduced power consumption. Figure 4 shows the sequence of ultra-sonic pulse reflection between the transmitter and receiver.

Fig. 4

Reflection of ultrasonic pulse sequence between transmitter and receiver

The constructed device is light weight and more comfortable to use also, the system utilizes dual sensors to avoid the narrow cone angle problem, as illustrated in Fig. 5. Thus, the three ranges are covered by dual ultra-sonic sensors. Moreover, thus the system has reduced reliability and was restricted to Android devices.

Fig. 5

Dual ultra-sonic devices covering three ranges

The appropriate auditory sensitivity is determined by minimizing the transmission latency in visual to auditory sensory substitution, which corresponds to the sense of visual information. To analyze the suitable auditory sensitivity, a cross model generative adversarial network based approach is evaluated in [39]. Based on the three groups of people, such as congenitally blind (CB), sighted users (SU) and late blinds (LB), the experiment is performed. This existing work uses CNN to process the audio embedding and uses created images to compute auditory sensitivity.

3.4.3 Combination of artificial vision and GPS

The assistive technologies related to GPS could give blind people specific autonomy. But the date of geographical information system (GIS), the accuracy of GPS, and map-matching approaches are mainly suitable for vehicle navigation and fail to aid pedestrian navigation for the visually impaired. Thus, the combination of GPS and artificial vision is designed [46] for blind people. This developed device is highly suitable for the visually impaired, even in an urban location where the signal GPS is weakened. A solution is established concerning the realistic combination of artificial vision positioning signals, GPS, and SIG. The established device gives accurate positioning and instruction to blind people. The positioning module corresponding to fusion is combined in the NAVIG EOA prototype. It will be verified in several environments to analyze the accuracy gained by utilizing geolocated landmarks. The designed device is highly comfortable for people with eye disabilities. It is adaptable to many situations like car navigation, robotic control etc., when the blind individual is walking along a road. A SpikeNet is employed to locate effortlessly recognizable requirements. An example of visually tracked landmarks of several locations is shown in Fig. 6.

Fig. 6

Visually tracked landmarks of different locations: an example [29]

Navigating complicated routes, exploring ways in an unfamiliar location, and object determination are challenging tasks for visual disabilities people. To make the effective instruction directives, a computer-vision based assistive technology is designed in [64]. This constructed device aims to improve mobility and autonomy in object location, pedestrian navigation and grasping. An enhanced geolocalization is attained by integrating images, satellites and other sensor information. The bling individual achieves the micro navigational target by determining the spatial audio rendering using a quick image recognition platform and enlarged directions. An improved version of the dialogue controller is designed to ease utilization and maximize the performance for visually disability persons. However, this assistive device cannot replace assistive approaches like guide dogs and smart canes. Advanced assistive devices should provide blind people with essential information about landmarks, spatial descriptions and proper instructions to follow. The developed NAVIG device attained improved autonomy in familiar and unfamiliar locations for the visually impaired. An effective navigation aid with reduced cost for the visually impaired was presented in [25] with a smart electronic travel aid device. This developed model provides better information for the blind ones regarding the environmental surroundings. This model focuses to limits the difficulties in several approaches and affords user-friendly devices with reduced costs for navigation of the visually impaired. The device is designed using ultrasonic sensors, power supply, GPS receiver, speech integrated circuit (IC), GSM module and headphones. The established device provides minimized cost solutions to various visual deficiencies people.

3.4.4 Bank note recognition for blind people

Blind people have an enhanced challenge in detecting and recognizing several currencies worldwide. Several studies have been developed to recognize the currencies of the visually impaired. By considering blind people, an automatic system is designed to recognize ringgit bank notes depending on the images of banknotes [65]. With the help of the MATLAB tool, the feature extraction was performed in six classes of bank notes. Here, the K-nearest neighbors (KNN) and decision tree classification (DTC) approach are used to recognize different bank notes. The optimized KNN and DTC were chosen using the Tenfold cross validation technique, which depends upon the minimized cross validation loss. The developed models attain an improved accuracy of 99.7%, and the developed system effectively recognizes the bank note with the assistance of DTC and KNN. Computer vision technology was developed to recognize Egyptian bank notes automatically to aid blind people [60]. This developed system was more robust in accuracy, recognition rate and efficiency. This system helps visually impaired users obtain the destination for the captured image. To generate the system as more robust to several conditions like scaling, viewpoint variation, rotation, illumination change and wrinkled bills, a speeded up robust features (SURF) approach is utilized. A bill in the camera’s view is detected using the spatial equivalence of corresponding SURF features. This mechanism reduces the false recognition rate and accurately instructs the blind user to correctly recognize the bill. The established system provides a true recognition rate of 100% and is verified by visual disability users. An example of a money reader for blind people is shown in Fig. 7.

Fig. 7

Money reader for visually impaired users [66]

A device is designed for the application of mobile phones to automatically recognize credit cards and money bills [43]. This designed system aimed to reduce the daily challenges faced by blind people. Several general problems are aroused in the daily lives of blind people, which can be minimized by developing an automatically recognizing system. In this system, the money bills presented in Argentina are recognized by the blind user. The database size is extended to recognize the currency of different countries. The currency conversion option is included, which allows the person to scan a bill from different countries to attain a similar value in the appropriate currency. In this manner, the number of various kinds of people who would utilize the application would be extended since if a blind user travel to a foreign country or a blind foreigner visits our country, they could utilize this application for currency recognition. The blind user can personalize the database by including images; thus, they can recognize them later. The application was tested for different types of visual disability people. Depending on this, the interface is enhanced; thus, many blind users can utilize the device. This existing work provides a more positive experience by recognizing the currencies in the mobile phone application.

3.4.5 Limitations with assistive technologies

Based on the survey conducted with assistive technologies, it has been observed that the sensor-based technologies were initially developed to help blind people in obstacle detection and navigation. These devices are coupled to the cane or other wearable devices to make them comfortable. Following these devices, camera integrated devices were developed that increased the weight of the equipment due to the additional weight of cameras. For the past decade, deep learning methodologies have been highly followed for detecting obstacles, but these methodologies require high processing power to function well. Although assistive technologies help assist visually impaired people, it is limited in use due to the following reasons:

• i Certain high-tech assistive technologies require a lot of learning that is out of the endurance abilities of the pupil.

• ii Some of the solutions provided by these technologies can be acquired by other inexpensive and simple technologies using low-tech devices.

• iii The pupil cannot follow normal communication flow when it gets engaged with electronic communication devices.

• iv Constant re-appraisal of the technologies is highly necessary as the pupil's requirements are always changing, and new products may soon become outdated to complete the needs.

• v Some of these assistive technologies are highly expensive and cannot benefit poor people with visual impairment.

• vi Experts and specialists may identify the requirements and provide better solutions, but there is no final solution for the problem as the requirements are dynamic.

• vii These technologies are only one among the wide range of options available to solve the visual impairment problem in the pupil.

• viii When the assistive technology is more accurate with multiple features, then the technology includes additional costs and weight due to the increased hardware requirements of the device. Apart from this, a lightweight and cost-effective technology cannot include multiple features, a major drawback of these technologies.

4 Visual to audio substitution system for visually impaired

The visual to audio sensory substitution devices (SSDs) provides several benefits to visually impaired users by transforming the images into auditory data. The SSDs are equivalent to reading as an image in a printed word form is transformed into audio. Hemispheric lateralization is tested for processing visual to audio information in blind people [55]. The audio information generated by vOICe and SSD creates a novel and standard dichotic listening test. The blind users were tested in the online or lab with similar stimuli. A hemispheric bias was not analyzed in visual to audio information processing in blind users or experienced vOICe users. In this, no differences were analyzed between the vOICe users and blind users. Because of the high effect of processing in the right hemisphere, the images processed at the auditory in SSDs are bilateral. This developed system provides higher advantages for visually disabled people. A gamification approach is designed for visual to audio substitution for blind people [50]. The main objective of this approach is to enhance the navigation skills and sound localization of eye disabled people. In training, blind people used an improved version of SSDs, which provides environmental information. Here, the training is concentrated on real time activities, and the developed strategy provides motivation, user engagement and immersion.

Hearing and touching are the two major basic senses utilized to substitute for a visual disability utilizing SSDs. Using an audio–video substitution system, the vOICe is tested for visual object recognition by substituting audio for the device [9]. The blind users heard the sonification of bi-dimensional images, equivalent to changes presented in tactile or visual modalities. The factors for training protocols are discussed in this existing work and construct a useful device for visually impaired people. In another study, the text information from the presented image is extracted with the help of OCR and a text-to-speech synthesizer [6]. The object presented in the images is analyzed. These selected features are fed to the input of the classification process. The classification uses the KNN classifier technique, and suitable labels are allocated to the test data. In the surface realization, the created contents are transformed into audio. The developed approaches perfectly recognize the object from the images. The processing is expanded further for analyzing several objects. Caption generation can be optimized by using natural language processing (NLP). The camera taken images are also utilized to afford realistic test data. The mentioned approaches provide an effective visual-to-audio substitution for visually impaired users.

The objects are identified using the canny edge detector and corresponding pixel size thresholding. The appropriate features are extracted by introducing the HOG approach, and the extracted features are the input of the classification stage. The developed model uses the KNN algorithm for the classification stage. Figure 8 illustrates the caption generation using the input object like an aeroplane.

Fig. 8

An example of caption generation of the aeroplane (a) RGB input (b) Edge detection using canny edge detector (c) Extracted features using HOG (d) Created caption

The basic structure of converting input text to audio is shown in Fig. 9.

Fig. 9

Conversion of text from the input image to audio

4.1 Advantages of visual to audio substitution devices

• i The entire navigation performance was improved in the visual to audio substitution devices.

• ii This substitution device can be utilized flexibly in several scenarios because of the conversion of numerous visual information into another format that the blind individual can sense.

• iii The sensory substitution devices can detect objects by hearing the sound with improved quality.

• iv The visual to audio substitution devices can forward the visual information in real time scenarios with higher efficiency.

4.2 Disadvantages of visual to audio substitution devices

Visual to audio substitution devices have offered enormous benefits to visually impaired people but also suffer from certain disadvantages that must be sought out. These disadvantages include the following:

• i Some developed devices require more time to sense the information, which disappoints blind users.

• ii In most of the analyses conducted in the existing systems, the accuracy of the visual to audio substitution devices is lower than in other tactile-based systems.

• iii The response time of these devices is also very high compared to other tactile substitution systems, as these devices require more time to complete the process.

• iv Some of these technologies exhibit processing capacity limitations in object recognition, and the most important information cannot be separately identified as all the viewed information is considered useful.

5 Tactile substitution system for visually impaired people

A tactile display is a human-to-computer interface system that uses lactation to provide information. Tactile displays can generate the tactile parameters of roughness, shapes, objects, surface texture and temperature. The tactile displays have been developed as an interface in virtual environment (VE) applications and as a substitution for the visual exhibition of information via tactile channels for blind people. Also, the tactile displays can be utilized for various applications like presenting touching feedback for virtual reality, tactile interaction in mobile phones and teleoperations [70]. The existing systems were designed with the aid of conventional methods. But recently, micromachining and microelectronics have been used to design large density tactile stimulators by reducing costs. Various applications are presented for tactile displays, both software and hardware. In the window environment, the software-based applications were designed. Some of the latest application areas are given as follows,

• a) Graphics and text

• b) Medical applications

• c) Military applications

• d) Virtual environment applications

• e) Educational and entertainment applications

• f) Consumer electronics and wearable devices

• g) Engineering applications

5.1 Graphics and text

The tactile devices help the blind user to read the computer’s screen and gather text based information through refreshable Braille displays (RBD), tactile mice and braille readers in direct manipulation devices [54]. An RBD is one of the electro-mechanical devices which continuously exhibits braille characters by increasing and minimizing pins via holes in a flat surface. This procedure is accomplished in a rotating wheel based refreshable braille system. In several models, the cursor position is analyzed by vibrating the dots, and a few models have a switch related to every cell to turn the cursor directly to the cell. The tactile mice are structured to assist blind users in navigating via computer screen affording information to the fingertips of blind users. Also, it can recognize text, graphic shapes, pictures, maps and art via touch [41]. In a smart touch, the visual images collected by a sensor are converted into tactile information and exhibited with electrical stimulation. When the system facilitates the printed material recognition via the tactile sense, it could be enforced as a Braille display for blind people.

For providing multimedia information and braille translation for a 2D multiarray braille display to avert the drawbacks of a 1D conventional braille system, a 2D multiarray braille display system was designed for blind people [40]. Blind people can read eBooks speedily and competently by expressing the text in various lines. This existing study improves the expression of the image on the braille display by edge extraction and image conversion for the braille. This approach aids in sensing visual media via touch. The audio and diverse multimedia contents were precisely translated to braille with the display of multiarray braille. Also, it could play audio files on smart mobile devices and tablets, which replaced the braille systems. The developed 2D multiarray braille display mechanism is helpful for blind users because of the high accessibility of information to navigate via multimedia content. The 2D mobile braille display is highly beneficial for attaining information like auto-based learning, scientific figures and literature. The devices like smartphones and tablets utilize the applications like braille pad simulators and braille OS. The braille pad simulator application is utilized to express the display contents. The Braille OS has numerous built-in applications like a calculator, image viewer, web viewer and games to an eBook reader application. The architecture of the eBook reader application is shown in Fig. 10.

Fig. 10

Structure of eBook reader application for 2D braille display

One effective braille display is HIMS braille sense U2, which can deliver digital information, office work and education to the visually impaired. Recently, the new braillesense Polaris is attaining more famous because of its enhanced performance around blind people. Depending on the relationship between other devices, the information is transmitted via wired or wireless systems to devices such as smartphones, tablets or PCs [30].

The Braille display devices are shown in Fig. 11. The Polaris uses gestures with the assistance of braille cells, and it can convert printed materials with fundamental text into braille. It allows several content of braille than the other established braille displays. Moreover, the mentioned devices were included with single-line braille cells, which cannot provide a convenient reading process for enlarged texts, and the braille figures cannot print.

Fig. 11

Braille display devices

5.2 Medical applications

Sensing through tactile is an essential source of information and helps the surgeon scientist to percept fixed tissue. Essential ducts and vessels were mainly concealed in connective tissues, and when the ducts and vessels were presented, they generated several damage. The tumours presented in the colon or liver must be abolished without turning into cancerous cells. The teletaction helps display and sense tactile information to the surgeon in the medical field. The contact aspects were sensed remotely through a tactile sensor array. Pressure distribution is generated on a fingertip by the force generator array in order to deliver location shape information. Several amounts of devices are established, and few of them are commercially presented for applications in the medical field. The friction forces on sliding human fingertips are controlled to generate synchronous vibrations in an enlarged broad range of ultrasonic frequencies, audible and tactile [62]. The fingertip can sense the skin vibrations, and the vibration present in the air can be heard from the finger’s closeness. This existing mechanism can record the friction forces up to 6 kHz and represents the fundamental electro adhesive amplifier. In addition, haptic assistive technology was designed to assist visually impaired people [51]. Tactile graphics, braille reading, mobility and orientation are the main areas in this work. The developed model is easier to use the visual disabilities people. Thus, it was concluded that tactile displays are highly beneficial for medical applications.

5.3 Military applications

The tactile displays help minimize recognized workload by providing beneficial aspects like intuitive nature, simple interpretation and allowing information transmission without changing the attention of blind users from the processing task. The tactile displays in military applications were considered for navigation, simulation and training, orientation and communication. Navigation through space is a challenging task and must be processed in several environments, and the tactile devices hold many hints for this application. The tactile display designed for navigation applications provides course error information and guidance on course correction [13]. In various environments, the waypoint navigation utilizing tactile concepts was illustrated, which has divers processing underwater at MANRL, automobiles at TNO, large speed boats at QinetiQ and aircraft at TNO or NAMRL. This tactile display is mainly beneficial as a navigation display for blind people, drivers and fire fighters performing in dust, smoke or darkness.

5.4 Virtual environment applications

Various applications in the tele-operation region and tele-presence are on the market and are willing to buy the products. The most determining are cyber touch, touch master and TacTool system. For CyberGlove, the cyber touch gives a tactile feedback option. At the end of the finger, the tactile actuators were placed, and the palm of the hand renders the vibrations and impulsion. At the tips of the fingertips, the touch master affords a tactile display, and for voice coil actuators, the thumb was utilized. In the TacTool system, the tactile display was placed on a cable mounted on the fingertips. The improved grain of the surface was continuously alternated due to the tactile interface’s excitation of surface acoustic water.

5.5 Educational and entertainment applications

The sensations were simulated using tactile stimulators, from bee stings and electrical shock to bullet impacts. The tactile interactive multimedia (TIM) [4] uses a movement detector and tactile boards as a component of a flexible multimodal game interface for visually disabled children. This existing work aims to provide varied levels of psychomotor development. It also provides the feasibility of automatically operating computer-related games for blind people without the help of normal people. The VTPlayer software was designed for blind children, and a mouse like the game console was established to play with the VTPlayer. This VTPlayer has functionality similar to an ordinary mouse but has extra tactile displays. With dual fingertips on dual fields, the user can sense with 4*4 dynamic Braille pins the software’s animations, characters and images. The VTPlayer software contains cognitive games, learning and plays software applications, tactile tools and geographical maps. An advanced synthesis of tactile music code and sound instruction was described in [37]. A graphical notation of music provides a high amount of information that the reader can sense in the same fashion. The Weasel system utilizes PVC tactile layered on an Intellikeys touchpad in fusion with audio and speech output. This system has been tested with several blind people and can be utilized along with previous psychological knowledge on communication with textures, tactile symbols and raised lines to generate a fundamental group of design concepts for tactile communication.

5.6 Consumer electronics and wearable devices

The tactile interfaces were utilized as a channel for miniature handheld communication for blind people, wearable devices afford convenient communication. The utilizations are alarms through touch, navigation, notify the process status to communicate with the device. The tactation provides several advantages to the users by interacting and communicating in a presented device. A visually dominated interface is not as effective for processing a provided task. In several cases, the graphical user interface easily ignores blind users. When blind users want to control and notify, a process previously affording visual notifications like radio-controlled components, slide projector, stage lighting, etc. Thus, tactile displays provide various advantages to blind people in wearable devices and electronic applications. 3D printed tactile graphics were presented to provide audio to tactile graphics with general tablet based computers or smartphones [59]. Every blind user can use a tablet computer to communicate with the developed 3D printed tactile maps. The users with minimum residual vision could accurately process the tasks with visual augmentation.

5.7 Engineering applications aiding the visually impaired

The blind people mainly confide in touch feedback, while the persons with deaf and blind were highly based on their percept of touch. The capabilities of blind people were improved with the aid of tactile technology by enhanced navigation autonomy. An Android application is designed to make tasks easier for visually impaired people through smartphone devices [38]. Appropriate information is attained by multi-tracking the signal flow between system elements. Initially, the visually impaired individual utters a command from the catalogue of presented voice commands. Due to the increased sensitivity of electret microphones, placing the telephone closer to the mouth is not required. An earphone was set with a microphone containing wireless devices like Bluetooth standard. The vocalized command crosses the recognition of speech concept via the GSM network using the Google speech recognition service. The character streams are given back by the speech recognition module to the recognized words and transmitted to the smartphone. Then, the program determines whether the delivered commands are in the presented commands. The blind user will hear “there is no such command” if not in the presented commands. The allocated function will suddenly be performed when the correct command is determined.

The fundamental concept of blind phone operation is shown in Fig. 12. The command flow extended since geographical coordinates, evaluated on the telephone based on the signal of GPS, are transmitted again via GSM to Google Maps. This service was termed reverse geocoding, in which the coordinates are projected onto the digital map and determine an occurred related address. The address is in a character chain form and forwarded to the smart mobile device, which is read out by the Android speech synthesizer [58]. The volume of sound can be limited by the standard buttons presented on the telephone. The designed application in this existing work offers several benefits to the visual deficiency people and makes their daily lives easier.

Fig. 12

Concept of blind phone operation

5.8 Advantages of tactile substitution devices

• i Easy to convey spatial information to blind people and effective in providing the information through tactile input.

• ii It is inexpensive to construct, consumes reduced power and allows a broad range of tactile sensations.

5.9 Disadvantages of tactile substitution devices

• i Some tactile devices are more costly and have several limitations in quantity and quality.

• ii Some of the devices cannot provide real information to blind users.

6 Computer vision-based assistance system for the visually impaired using different techniques

In recent decades, there has been a large growth in the requirement for assistive technologies, which are highly important to avoid the troubles of blind people and enhance their life quality. Various research works focus on a group of cross-application computer vision activities as the centre to develop the assistive technology categories utilized to help blind users [33]. Nowadays, computer vision-based approaches are more effective in developing assistive technologies for the visually impaired. In various existing research, the possible impacts on the evaluation of assistive technology users, economic and medical, are also reported.

6.1 Requirement of user and computer vision-based tasks

The number of people required to be addressed by an assistive technology has recently increased. Depending on the Flagship programme, the Global Cooperation on assistive health technology (GATE) evolved by the world health organization (WHO). Several tasks are classified as personal mobility, mental functions, daily life activities, skills and communication training and sensory functions.

• 1) Personal Mobility

Personal mobility considers the ability of a person to move in environmental surroundings and the ability to handle objects [48]. Moreover, visually impaired people find it difficult to do their daily tasks, so their mobility function is compromised. Several research works have been developed to support mobility problems and rehabilitate the human body’s functions.

• 2) Mental functions

Mental functions affect the thinking capacity of humans as well as concentration, emotional reaction, remembrance etc. The mental functions of visually disabled people are different from normal propels. Various assistive technologies support mental functions involving personal digital assistance (PDA), voice instructions, smart watches and electronic calendars [23]. The general internet accessible mobile phones provide specific user interfaces that support the mental functions of the visually impaired. The rising assistive technologies for mental function support extend mental health care services.

• 3) Activities of daily life

In recent years, various research works have been concentrated on devices for medical services by developing ICT technologies to improve the awareness of the blind user. Also, increasing their comfort, safety, and independence ease their daily tasks. The major characteristics affect the sense of life quality and consider the feasible deterioration of health issues, security and community life. The recent technologies provide several services in domestic environmental sensing, common health condition monitoring, localizing indoor and outdoor and interfacing remote control [26]. Wearable systems mainly operate safety and health condition observations with electronic modules and sensors. The computer vision-based solutions were utilized for health observation and sensors, which reports object localization, tracking and detection of an object, activity recognition of blind people and environmental mapping to assist the visually impaired.

• 4) Skills and communication training

Several technologies have been established to improve the skills and communication process of the visually impaired [52]. A Psycho-education Program was conducted to effectively enhance the communication skills of blind people. Social communication is a fundamental term which supports maintaining an individual’s society. 24 blind users explored a traditional raised line and interactive map, and the memorization time was monitored in [8]. This existing work reveals that the interactive map was better than a tactile paper map with braille legend considering different dimensions.

• 5) Sensory functions

The sensory impaired people have minimized ability in hearing, sense and vision perspectives. The sensory impairment effect is ranged from minimal to entire ability loss to utilize the perspectives with a high or reduced effect on daily tasks. The assistive technologies aid sensory functions, mainly converting the aspects of one sensory modality into the provocation of other perspective modalities. The sensitivity loss in fingers or hands can make it hard to utilize a device like a keyboard or a mouse, and replacement interfaces are required. Computer vision technologies highly assist visually affected people. Effective object detection and recognition support blind people to individually access unknown environmental surroundings [68].

Many computer-based approaches have been established to reduce the difficulties of visually impaired people. Object recognition is one of the common problems in computer vision. The human visual system effectively performs object recognition tasks, which was difficult in computer vision. For object recognition, numerous techniques were presented in computer vision-based systems. But they cannot attain improved results because of the unlimited conditions in environmental surroundings like high variation in illumination, background, luminance, object orientation based on camera, occlusions, size, scale etc. Also, indoor environmental conditions like luminance and illuminations are comparatively stable. Many people work in outdoor environments, and the daily tasks in indoor locations like workplaces, banks, homes, stores, groceries etc. [18]. For recognizing the object, computer vision-based techniques for blind people in indoor surroundings can be classified into tag-based and non-tag-based [44].

The system needs separate visual tags in the tag-based approaches to be positioned on all the objects that need to be detected. Depending on the information on the tag, the objects get recognized. In the non-tag based techniques, the system has no necessity for tags on the objects. Without using tags, the object was recognized by using the information on the shape of the object, color, size and other physical characteristics. The non-tag based techniques can be classified into two groups such as 3D modelling techniques and 2D modelling techniques. The 3D modelling techniques use stereo vision to construct the objects in 3D models, and the recognition is processed depending on the matching of the 3D model. The 2D modelling techniques mainly use only one camera to collect the image data. The recognition is processed based on the extracted features from the given data. The 2D modelling techniques can be categorized based on different types of extracted features from the visual information. The following features are SURF, SIFT, and edge/colour-based features. It is observed that common object recognition techniques use different features. Figure 13 describes the classification of computer vision-based approaches for aiding blind users in recognizing objects in different indoor surroundings.

Fig. 13

Classification of computer vision-based techniques for the assistance of blind people in recognizing objects in indoor locations

Indoor navigation was proposed for blind and elderly people based on radio frequency identification (RFID) tags [69]. RFID is the utilization of a wireless non-contact system that uses radio-frequency electromagnetic fields to transmit the information from a tag placed on the object for automatic tracking and detection. The RFID tags were categorized into two types such as passive RFID tags and active RFID tags. In the passive RFID tags, the battery is not needed and are get powered with the help of electromagnetic fields. The active RFID tag employs local power sources and is not cheaper than the passive RFID tags. The passive and active tags involve electronically stored data transmitted through radio waves when the tags are initiated. Concerning the visual deficiency of people, an RFID based indoor navigation system brings numerous benefits.

In this, a mapping process by representing the building blueprints was established. Also, advanced localization and barrier prevention method was presented. An antenna circuit is constructed to enhance the aspects of the developed system. The developed localization server and wearable module offer effective voice instructions, perfect navigation and indoor localization. The blind folded volunteers verify the mentioned features in different schemes. A computer vision-based approach is developed to identify the components automatically by utilizing 2D digital images [28]. The developed approach mainly confides on four kinds of combined shape and color based modules which identify electrical outlets, studs, insulation and different conditions for drywall sheets. The images were categorized into five states depending on the four modules’ results. The developed method offers improved component detection performance and enhanced recall and precision.

Computer-vision based techniques should be capable of accurately detecting moving objects and obstacles in any complex, cluttered scene. Specifically, the urban scenes are highly cluttered due to the high number of moving objects and the presence of a large count of obstacles. To present a reliable solution, Tapu et al. [67] introduced a new computed vision-based perception system that helped visually impaired people in autonomous navigation. The moving obstacles and objects in urban cluttered scenes were detected in real-time, and those detected objects were classified. To overcome the problems identified in traditional GPS-based systems, a building/landmark recognition mechanism was also introduced. The method also gave importance to the objects of interest specified by the user in certain indoor environmental scenes. Such objects were localized using the multi-object identification and tracking approach. The feedback from the system was presented through warnings, and bone conduction headphones were administered to allow the people to hear those warnings. To make the system user-friendly and cost effective, the system was integrated on an Android smartphone. Though the method was effective in obstacle detection, some of the complex obstacles important in navigation cannot be identified. Those obstacles include stairs and crossings detectors, familiar persons, navigation information, etc.

Compared to outdoor navigation, indoor navigation seems problematic as the GPS is now gaining an advantage in helping the visually impaired with outdoor navigation. To deal with the problem, an assistive technology based solution was provided by Elgendy et al. [21]. The approach provided a better solution for visually impaired people to navigate indoors using markers. A navigation system prototype was implemented in that work that helped the visually impaired people use computer vision tags to identify the locations. The solution initially asked the people to choose a map and then stored information regarding the marker places. Again, it asked to choose the appropriate destination point for navigation. Then, a smartphone camera was utilized to locate the marker places from the source point to navigate the person to the destination. The person reached the marker point of the destination with the help of navigation commands provided by the system. The evaluations of the approach under complex scenarios with QR codes and ArUco markers proved the significance of the approach. The prototype helped the visually impaired people better navigate, but more knowledge and understanding were required in the presence of occlusions. The markers gathered by the system were not always accurately detectable in the presence of occlusions and sometimes when some objects hide part of the marker.

Another system based on the use of markers for navigation was developed by Elgendy et al. [20], where the markers were identified using a classifier. A prototype with the use of markers was introduced to help in the navigation of blind people. The method included four main contributions: 1) to introduce a marker-based system to assist visually impaired people in navigation. 2) A comparison was performed between the QR codes and ArUco markers to prove that the markers work better than the codes. 3) A simplified form of CNN architecture was implemented in the detection phase to detect the candidate markers under diverse challenging conditions and to enhance the response time. 4) To enumerate the performance of the proposed model in terms of training and testing accuracy by comparing it with the existing model. The shortest path between the source and destination was identified with the help of the generated map, and then commands were provided for the users to accurately reach the destination point. The methodology worked under diverse complex conditions, and the markers were accurately detected in different scenes. Though the model was effective and efficient, the map generation part remains a burden, and more advanced techniques can be developed to automatize this part.

Shopping and indoor navigation are identified to be complex activities that cannot be easily handled by people with visual impairments. To help visually impaired people enhance their lifestyle, a new approach integrating deep learning and markers was introduced by Elgendy et al. [22]. Initially, the markers were installed at interest points in a building. Then, using a graph, the map was constructed to store the interest points and to identify the relationship between those points. Further, the smartphone was utilized to provide commands to the person to reach the destination with the help of markers. Since the system failed to accurately detect the markers under challenging conditions, a deep learning based TinyYOLOv3 model was introduced that accurately predicted the markers under challenging conditions. The feature extraction and detection modules were changed, and the model was modified into three versions to obtain a better and more accurate output. The methodology was implemented and evaluated under diverse complex scenarios using far and full datasets to prove its effectiveness. The method proved to be more adaptive and efficient than the other models, but the wrong directions taken by the person during navigation require more control and knowledge regarding the surroundings.

6.2 Advantages of computer vision-based assistive devices

• i Delivers more information about the object and provides superior guidance to the visually impaired.

• ii Better for the outer surroundings, real time processing, and offers quality instructions.

6.3 Disadvantages of computer vision-based devices

• i Some of the devices require RFID tags to detect the object.

• ii Some of the developed devices are not suitable for deaf people.

• iii Large processing units make the system bulkiest.

7 Wearable assistive system for blind people

The requirement for assistive devices for the visually impaired has improved in recent decades. White canes and training dogs are popular tools for the visually impaired but cannot offer all the information for safe mobility. A cost-effective wearable device is designed for blind people [61]. An ultrasonic sensor and Arduino Nano microcontroller are the two major components used in this existing work. A unicorn gyroscope is utilized, which generates a critical element of the system and makes a prototype for the barrier identification mechanism. The major aim of this existing model was to make the alternative of white cane with a prototype considered an advanced technology at an affordable price. The ultrasonic cane improved the object detection range in the white cane with the aid of an Arduino board, ultrasonic sensor and vibration motor. Thus, the difficulties faced by blind users are minimized, and the developed model provides more advantages for the blind in navigation.

The prototype was simple to carry and cannot require additional space to store. A wearable assistive system is designed by using CNN to assist visually impaired people [31]. This system was the fusion of binocular depth camera ZED 2 and Jetson AGX Xavier. Depending on the developed neural network and the depth map, the environmental image in front of the blind user is separated into several similar divisions. A confidence value for walking in every division is evaluated, and a voice is suddenly initiated to instruct the blind user. This provides the blinded user can survive a secure path on the sidewalk, prevent any barriers and securely walk on the crosswalk. The YOLOv5s network, in some research, detected the barriers in front of the user. The experimentation of the visually impaired in each of the views is demonstrated in Fig. 14.

Fig. 14

Experimentation of blind users in both front and back views

The experiment was carried out to instruct a blind user walking on both crosswalks and sidewalks. In the back view of the figure, the developed prototype was carried by the visual deficiency user. This experiment was executed by the MRT station closest to Taoyuan High-Speed Railway in Taiwan. The destination was set only by the blind user, and Google Maps will plan the route from the user’s leaving point to the destination. In this experiment, the walking path of the blind user was planned and demonstrated by the small blue color dots in Fig. 15, and the route length was 280 m. The results reveal that if the time consumption of the voice instant was avoided, then the device’s processing speed was 6 FPS with segmentation of 0.99 s, depth measurement and object detection was 0.06 s, and selection of division and confidence computation was 0.0145 s.

Fig. 15

A walked the route of a visually impaired person

The time taken for this overall experiment was three to four hours and seven to eight rounds in an experiment. The experiment’s walking path length was 300 to 1200 m. Some of the voice instants are "right", "slightly right", "left", "slightly left", "poly", and "go straight". When the blind user arrived at a certain point in the planned route, “poly” was played. Also, the prompt “poly” always emerged at the location of the curve.

In this existing work, the experiment is not only performed for people with visual impairments but also for some of the students in their lab by wearing blindfolds, as shown in Fig. 16. As described, a wearable assistive system is designed for smooth navigation and environmental awareness for the visually impaired [3]. The designed wearable device helps the blind user navigate securely and quickly in an unknown location. It also supports them in tracking objects in indoor and outdoor surroundings. The developed device contains a red, green, blue and depth (RGB-D) camera and an inertial measurement unit covered on a smartphone and eyeglasses. A lightweight CNN-related object recognition system is established and placed on the smartphone device to improve the sensation ability of blind users and enhance the navigation system. It can offer information about the objects in different environments like locations, categories and object orientation. Human–machine communication was processed via audio modules such as beeping audio for barrier alert, recognition of speech for understanding the commands of the user and synthesis of speech for exhibiting the information of environments.

Fig. 16

Experiment by some of the students with blindfolds

Another existing work is presented an indoor-based navigation system for blind users utilizing a path detecting approach and a wearable cap [35]. This developed system has two modules: a wearable part and an area schematic in which the navigation system is processed by instructing the user. The wearable segment has a cap constructed with IR receivers, headphones, an Arduino Nano processor and an ultrasonic sensor. The schematic segment previously plans the direction of movements in the room by splitting the area of the room into cells with a predefined matrix, including environmental information. The position of the blind user was monitored continuously by 16 IR transmitters and was positioned at a similar interval in the XY directions of the indoor location. The blind user uses a Braille keypad to provide the cell number for analyzing the destination region. A path detecting approach was designed to analyze the location of the blind user and instruct the user to attain the destination.

The presented approach tracks the blind user’s location by attaining continuous transmitter information and instructs the blind user to achieve the destination region by voice command. The ultrasonic sensor was covered on the cap to track the barriers in the path of the blind person. The developed system cannot need any of the complicated infrastructure design or the need to include additional devices for the blind user. Based on the destination option of the blind user, the pre-recorded voice command will offer movement instructions to each edge of the indoor location. This developed system generates the navigation system as effortless and user-friendly for users unfamiliar with the innovative technology and blind people. The established system has no need for GPS or telecommunication networks, which makes it feasible for use in rural areas without the coverage of a telecommunication network. Figure 17 resembles the indoor navigation system’s designed wearable cap for blind users.

Fig. 17

Wearable cap for the indoor navigation system for blind people

In this wearable cap, an ultrasonic sensor is placed, which helps to detect the barriers in the pathway of blind users. If the designed wearable cap tracks any objects, it will give a “Barrier” noise to the blind user as an alert. The blind user must detect their path by rotating the head to reach the destination point. Another research presents a wearable assistive device with a fuzzy decision support system for visually impaired users [7]. The designed navigation system confides mainly on realistic operating boards, sensors, user interfaces and a fuzzy logic-based decision support system. In this, the sensor information is considered the input and offers the secured orientation to blind people. The blind user was informed about the choice based on a combined voice-haptic interface. This existing system has two wearable barrier tracking systems controlled by an embedded controller.

The control system uses the structure of a robot operating system aided by the beagle bone black master board, which reaches the realistic constraints. The data aggregation and barrier ignorance are performed by various nodes maintained by the ROS to provide combined haptic voice information for the instruction of blind people. A fuzzy logic-based system was developed to support blind users in selecting a secured path. The implemented system was performed on both blind people and blindfolded persons. The designed system was constructed using three components: an embedded controller, an eyeglass frame with 3 ultrasonic sensors for barrier identification and the same accessory, which can be placed in the hand for ramp and up-down obstacle tracking. The vibrator modules are utilized for tactile interfaces, which excite the plantar region with vibrations. The developed approach provides a better instruction system for blind users walking in different environments. The three major components of fuzzy-based wearable assistive device for blind users is shown in Fig. 18.

Fig. 18

Components of fuzzy logic-based decision support system for visually impaired

Three major steps are carried out for constructing the device with the help of 3D printing approaches. i) The 3D geometry of the wearable device was modelled by CAD software; ii) the collected information was processed via standard software; and iii) the developed device system was realized using a 3D printer. The developed fuzzy-based system provides better guidance for visually impaired users. Deep learning techniques generally offer higher outcomes for generating assistive devices for the visually impaired. A low-cost wearable sensor with improved sensation for blind users is developed using deep learning approaches [5]. This existing work can detect barriers in outdoor environments. The developed deep learning approach is based on a non-intrusive wearable device with reduced cost. The colour image estimated a depth map of the location, which offers 3D data about the scenario. After that, an urban object detector tracks the information of the object in the location. The obtained information was forwarded to the blind user via haptic feedback. The implemented system can top processed at 3.8 fps and attained an accuracy of 87.99% in barrier presence identification. A healthy person and visually impaired users analyzed the efficacy of the developed system. In this existing work, a small size wearable device was designed to capture information about the unknown environment. Figure 19 illustrates the small-sized wearable camera for blind people.

Fig. 19

Small size wearable camera fixed to the blind user’s pocket

In this, the blind user wears a minimal size wearable wireless camera, which can monitor information about the environment. The smartphone device processes a custom application which will attain the images gathered by the camera in realistic through WiFi. The image size was reduced to 608*608 and moved from the smartphone to the remote deep learning server. This designed system can make percept and learns the location of the visually impaired user. The presented devices provide a better description of the detection of barriers within the scenarios. The deep learning techniques give better performance in terms of accuracy and are highly beneficial for visually impaired users. In another research, a reduced cost solar powered wearable assistive device was designed to ease the detection of objects for blind people [10]. This existing system was constructed using three major components: a small-sized low-cost camera, ultrasonic sensor and system on module (SOM) computing unit. The low-cost camera was placed on the eyeglasses of the blind user and obtained a realistic video of the closest environment. The aSOM computing unit was placed on the belt and processed by deep learning approaches, and the video captured by the camera detects the objects with the help of spatial approaches. An ultrasonic sensor helps to find the objects presented in the environmental surroundings.

The designed device instructs the user and supports them in finding the object accurately. After analyzing power consumption, a wearable solar harvesting system was combined with the implemented assistive device. It was tested to improve energy autonomy in several processing modes and locations. The experimental results attained with the designed reduced-cost assistive device have illustrated an appropriate, realistic object tracking with an enhanced recognition rate of 86% with a 215 s time interval. This existing system can recognize 91 objects provided by the Microsoft Common Objects in Context (COCO) database. For training the dataset, the CNN is applied to include the objects in the system. The designed device offers improved results in detecting objects in urban scenarios like buses and cars. Another work was developed using deep learning methods concerning visually impaired users [14]. Here, a wearable smart glass was named MedGlasses system for blind people to enhance their medication utilization. The developed system has dual wearable smart glasses, a mobile app, and a cloud-based data management system. The MedGlasses system forwards the medication information to the cloud-based management to design medication use records in which the family members can capture the medication condition of the blind patients by utilizing the mobile phone app.

This existing device attains a 95.1% recognition rate and supports blind people in reducing the issues of drug interactions affected by taking the wrong medications. This work minimized the medical treatment cost and offered blind patients a secure medication scenario. A wearable guide device was designed for blind users based on deep learning approaches and video streaming [32]. This project aims to offer good assistive to white canes utilized by blind persons and provide them enhanced freedom of movement utilizing the developed wearable system. Computer vision in the developed device utilizes an RGB camera without a basic RGBD camera in computer vision. The RGB images are transformed into depth images with the aid of deep learning approaches and evaluate the plane for indoor object identification and secure routes for walking. Using the CNN structure, the developed system can understand numerous feature routes and then the model was created from the learning outcome. Thus, this established system provides the visual deficiency of people to accurately detect the flat and find secure routes for walking. Based on the CNN model, the indoor depth value was predicted, and it is given as,

$${\mathop {D}\limits^{\wedge }}_{k,m,n} = w_{lq}^{Z} H_{k,m,n} + B_{l}$$

(1)

Where, ${\mathop {D}\limits^{\wedge }}_{k,m,n}$  is represented as the depth value, $H_{k,m,n}$ is considered as a feature vector, and the baseline is denoted as $B_{l}$. The feature vectors and baseline was utilized to evaluate the pooling parameter $P_{l}$, and it is given as,

$$H_{k,m,n} = h\,(E_{k,m,n} ,\,\theta_{h} ) = \,W_{l} P_{l - 1}$$

(2)

$$P_{l} = pool\,(nonl\,(W_{l} P_{l - 1} + B_{l} ))$$

(3)

The loss function is computed as,

$$L\,(\theta_{h} ,\,\theta_{lq} ) = \frac{1}{M}\sum\limits_{k,m,n} {(D_{k,m,n} - \mathop {D_{k,m,n} }\limits^{ \wedge } )^{2} }$$

(4)

This existing study focuses on designing a wearable device for blind users by reducing their walking efforts in various scenarios. Multimedia tools and application wearable systems also analyzed the plane length. The blind user with the developed device can walk to a target place under the secure instruction of the established device. The developed system device with a white cane allows blind users to attain secure and individual movements like guide dog techniques. Because of the restriction of the hardware process, extra efficient approaches will be needed.

The developed device was tested with several users, illustrated in Fig. 20. Thus, this existing study provides improved outcomes and reduces the effort of visually disabled users. Table 1 mentions the devices developed in the previous studies and their significance.

Fig. 20

User wearing the guided device

Table 1 Devices developed in existing studies and their benefits

7.1 Advantages of wearable assistive devices

• i Allows the user to carry the device anywhere and anytime. Ability to detect the shortest path and instruct the user to travel in the object less path.

• ii More suitable for indoor and outdoor environments.

• iii Provides improved detection rate in identifying static obstacles, making interface learning easier.

7.2 Disadvantages of wearable assistive devices

• i Some of the devices were designed for heavy weight and uncomfortable carrying.

• ii RGB-D sensors are needed in some wearable devices to detect obstacles, making the system more complex.

8 Mobility and orientation of visually impaired people

The mobility and orientation training offers important skills and approaches for independent and secure mobility for visually disabled people. The requirement for mobility and orientation training was enhanced as the number of blind people enhanced. Also, increasing studies on mobility and orientation assistive technologies have investigated the experience of blind users during orientation and mobility training. The mobility and orientation training was mainly provided to blind people to limit mobility challenges and aid independent life skill learning. The mobility and orientation training was performed with the eligible mobility and orientation trainers (MOT). Mobility and orientation are highly concentrated on obtaining strategies and essential skills to perform daily activities. Several research exhibits that employment, mobility, and orientation training have an influential association. This was mainly important for frequent mobility and orientation training and support. The classical assistive technologies utilized in mobility and orientation training are presented in mobile applications, which aid excellent spatial awareness and route planning.

Many methodologies focus on solving the problems faced by blind people by using various assistive technologies like wearable sensors, smart canes, etc., which also help ease the mobility of visually impaired users. An advanced intelligent system is implemented for instructing blind people with the automated orientation and mobility system [56]. This system allowed blind users to move effortlessly in unknown locations like sighted individuals. The established system was attached with a Global System for Mobile Communication (GSM)—Global Positioning System (GPS) module to pinpoint the blind user’s position and to make a dual-way communication path in a wireless concept. Also, the developed system offers information about the direct path so that the barrier is detected with the assist of ultrasonic sensors. The presented system also contains a vibrator, beeper and accelerometer sensor. The overall design system was small, lightweight, and attached to the white cane.

The experimental validation examined the accelerometer sensor and GSM-GPS module to pinpoint the blind user’s position by developing a dual-way communication path. It also tests if there is an alternative to blind stability. Figure 21 demonstrates the verification of the accelerometer sensor and GSM-GPS module. The above figure automatically transmitted information from the system to a pre-arranged telephone number mentioning the user’s location. The system shared this information as an outcome of stability swapping in the z-direction.

Fig. 21

Examining accelerometer sensor and GSM-GPS module

A volunteer user also performed the experiment to analyze the smart white cane’s functionality. Figure 22(a) illustrates that the volunteer user was provided with a training course on utilizing the white smart cane and was asked to move towards a wall from a random space. The volunteer user was instructed to raise the left hand, and he experienced a vibration below the right hand, and the walking continued until the sound was heard. A threshold value was set to the vibration, and half of this value was provided to make a sound representing the difficulties of barriers. In this case, the user can predict the distance of the barrier with the help of sound and vibration. Figure 22(b) exhibits the experiment done for the volunteer user who was requested to walk in front of the car, which lies underneath the horizontal level of the smart cane. The user can identify the car before the one-metre distance. At the last testing stage, the user was requested to turn the white cane right and left to detect barriers while walking. Figure 22(c) resembles that the volunteer user can identify the tree using the presented approach from a one-metre distance. This experiment aimed to train the user to identify the barrier size and decide on an appropriate direction. The advantages of this developed system were discussed in this existing work; it allows blind users to move from one place to another without facing any difficulties.

Fig. 22

Experimenting with the smart cane by a volunteer user (a) Identifying a wall by utilizing vibration and sound (b) Detecting barriers below the horizontal level of the smart cane (c) identifying barriers by turning the smart white cane right and left

A rehabilitation program was conducted to enhance the mobility and orientation of visually impaired people [42]. This existing study presents the combination of a blind aid in a mobility and orientation rehabilitation program as training support for blind people. The BlindAid system allows users to communicate with several virtual objects and structures via haptic feedback and audio instructions. This study analyses how the rehabilitation program with BlindAid can support people with visual disabilities to train themselves in known and unknown locations. In this existing research, nine participants are considered in the mobility and orientation rehabilitation program at the Carroll Center for the visually impaired. The research was conducted in different environmental surroundings using virtual environment analysis and orientation activities. The developed system has quantitative and qualitative approaches with a questionnaire, interviews, user computer logs and videotape recording. The experimental results show that the BlindAid training provides participants extra time to analyze the virtual scenario. The computation of mobility and walkability demands of blind people in an urban environment was presented in [11].

This existing research intended to compute the walkability of the visually impaired in urban scenarios by maintaining a qualitative investigation based on commented paths method (CPM) to define subjective computations from the realistic experience of certain paths. The physical aspects of the investigated path’s infrastructural elements were accessed using the Likert scale. A single enlarged judgement evaluates separated and global walkability instructions and the enlargement of thematic maps as an advanced combination of the approach to exhibit the risk characteristics to all stretches in which the path was separated. The emotional sensations, physical interaction and moving habits of blind people are analyzed in this existing work. The CPM was developed for blind users to do their tasks without facing any challenges. The integrated application of CPM mentioned its advanced features, and the answers were gathered using the Likert scale based on judgements. Utilizing these outcomes for evaluating global walkability and separated instructions are mentioned on thematic maps. The value of instructions enhances, and darker colors were related and supported to detect the main difficulties related to every stretch. Based on this, the developed approach performs the qualitative analysis of the estimation for the single path sections.

In [49], a device for mobility and recognition of an object in the interior scenario for the blind was developed. The autonomous orientation and navigation of blind people were offered by developing an advanced assistive device named NavCane. This developed system supported the users in detecting the barriers in indoor and outdoor locations. The designed NavCane helps to recognize objects in indoor locations. This NavCane device gives information about the barriers presented in the path. The gathered information by the device is forwarded to the blind user with the assist of audio and tactile communication approaches. The established device was low-cost and a reduced power embedded device for barrier identification and barrier tracking. The device contains ultrasonic sensors, a radio-frequency detection reader, vibration motors, a battery, a global positioning system module, a wet floor sensor and a gyroscope. Eighty blind people from different scenarios were experimented with, and the designed device was tested for object recognition. Thus, it analyzes the benefits of the NavCane device.

Figure 23 illustrates the experimental analysis of the designed NavCane device in indoor scenarios with several obstacles. This indoor location contains 15 obstacles involving five-foot, waist, and knee levels. In the rectangular-shaped box, the obstacles were organized at 777 cm in length and 296 cm in width. The blind user had to initiate from one side of the path and obtain the other end across all presented obstacles. The testing results demonstrate that the designed device was more effective for obstacle detection, wet floor navigation and recognition of an object in familiar and unfamiliar environments. Also, it reveals that the NavCane device attains improved performance than the previous devices. A comparative analysis of the existing substitution systems is presented in Table 2.

Fig. 23

Experimenting with the NavCane device in an indoor location with several obstacles

Table 2 Comparative analysis of the reviewed substitution systems

The outcomes presented in the table suggest that most technologies focus on enhancing the accuracy and response time needed for a system to complete the task. The beneficial result in terms of accuracy is provided by the visual system introduced by Uematsu et al. [70]. This system is also efficient, with a response time of 12.4 s which is very low compared to the other two techniques. The visual system introduced by Bhat et al. [6] resulted in an accuracy rate of 88%. Still, the methodology is not evaluated regarding response time as the system is not very efficient. The system introduced by Brown and Proulx [9] resulted in an overall accuracy value of 54.61, and the tactile system achieved 64.36, which is less than other compared techniques. Also, the response time taken by the method is very high than the acceptable rate. This is because of the complexities involved in the completion of the task. The analysis identified that the system developed for visually impaired people should be accurate and efficient enough to satisfy the users' requirements.

9 Conclusion and future scope

The review conducted in this paper provides a clear insight into the difficulties faced by visually challenged people in their day-to-day activities and the urging need for effective technology to provide better guidance in navigation. Blind people generally face several challenges in navigation in both indoor and outdoor environments. The scenes and information gathered from the indoor and outdoor environments include several complexities along with clutters and occlusions that must be dealt with in a proficient way to provide accurate navigation commands. Several assistive technologies that have been developed to date are reviewed in a detailed manner to guide researchers in this field. The research topic chosen is the most popular and hot topic under discussion for several decades. One primary challenge in identifying an effective solution for blind people is the changing requirements that differ at different times. The technologies must be equipped with effective features based on their changing needs. Apart from this, it is also required for assistive technology to be cost-effective and user-friendly to promote better navigation of blind people. Different authors have published different methodologies and prototypes to successfully promote the safe navigation of blind people even without the assistance of other persons. Though a vast amount of literature is available to deal with the navigation issue of blind people, more focus and care are required to be taken in terms of cost effectiveness. This is because blind people who are underprivileged and poor generally hesitate treatments and solutions due to increased expenses. Among the reviewed systems, the tactile substitution systems have played a role in helping blind people in navigation and are also advancing in terms of several factors.

Currently, there are several devices available in the market to promote navigation and guide blind people in both indoor and outdoor environments. But, as the review suggests, more proficient devices must be developed to smoothly deal with the clutter and occlusion problems in indoor and outdoor environments. Some of the works have presented a prototype to detect traffic signals to help blind people in crossing the road by themselves. This could be an advanced and interesting research area where more focus is required as the commands provided for blind people to cross the road are of utmost importance. Mainly, the directions taken by blind people with the help of assistive technology should be more accurate to reach the destination. Another major concern is the weight of the technology, as these devices must be carried by blind people wherever they go. Therefore, a lightweight technology would be of high preference. While reviewing the literature, it has been identified that several methods were developed to reduce the weight of the device, but most of them are limited to only preliminary studies. Therefore, a real-time implementation of such a device is crucial to attaining beneficial outcomes in the present scenario. A single device that can integrate all the above-mentioned requirements will greatly benefit visually impaired people in the long run. Thus, this review suggests that future researchers and developers should concentrate more on multi-featured assistive technology without any technical constraint to benefit visually impaired people in the near future.