Keywords

1 Introduction

In day-to-day life, visual impaired people find several difficulties that they try to manage and overcome the best way they can. Presently, society is being endowed with social, psychological and technological capacity to assist citizens with disabilities. Technology has brought new insights in solving, at least, some of these difficulties. Nevertheless, there are still some gaps to fulfill. Aesthetics is one example. In some professions, the way we dress, for example, has a great impact in the context we are inserted. This task can be a huge problem especially for the visually impaired. They dependent on their relatives, friends, and/or support center professionals for the purchase or for choosing the clothes to wear.

This project follows a previous work, “MyEyes” [1,2,3]. With the support of the Association of the Blind and Amblyopes of Portugal (ACAPO) [4] and the Association of Support for the Visually Impaired of Braga (AADVDB) [5].

The goal of this project is to assist the visually impaired in selecting the color combination of the clothes to wear through the development of a mechatronic system endowed with artificial intelligence to support the choice.

This paper in divided in 5 sections. Section 2 describe the previous and related work, in Sect. 3 mentions system goals, in Sect. 4 is described the prototype development and finally in Sect. 5 concludes with final remarks.

Technological developments have reached important proportions in terms of support for visually impaired. In this way, this paper presents some solutions to identify the highlights and the gaps that can be overcome.

There are various versions of smart virtual closets that have been applied in the current market. However, despite the different versions of virtual wardrobes there was no study on the consumers’ attitudes [6].

In order to overcome this gap, Perry [6] performed and published a study on the intention of adoption of consumers of smart virtual cabinets. In this study a positive attitude towards smart virtual cabinets was led by two beliefs about the product, utility and ease of use. Utility and ease of use accounted for 84% of the variation in consumer attitudes towards the smart virtual closet. The results confirmed that the high utility of the product is one of the decisive factors of people’s attitude towards intelligent virtual cabinets, and the low complexity of the operating system is a significant secondary determinant of consumer attitudes.

Recently, Bhowmick and Hazarik [7] published a survey, where 3010 papers were collected focused on the vision of assisted technology for visually impaired and future research trends. In conclusion, the authors emphasize the importance of the technological advance dedicated to blind people, referring to the increase in the functionality of the mobile technologies, advances in computational vision processing algorithms, miniaturization of electronic devices and the new cutting-edge medical interventions that should boost this field for the challenges and reality of successful technology creation.

Some fashion apps have been developed as STYLEBOOK, an application to manage the clothes, create outfits, and plan what to wear during the month [8]. Mode-Relier application [9] is another example that takes into account the skin-tone, hair color and even make up products. ShopStyle [10], allows to plan purchases based on the user favorite’s stores, searching items across the web. Tailor [11] is a closet that learns the user’s preferences and choices and suggests the combinations to wear. This system uses sensors called ‘TailorTags’ which are embedded in the clothes in order to detect the items and keep tracking of daily choses, learning and make suggestions [11].

A recommendation system based on semantic content to advise clothing items for recycling was proposed in [12]. In order to demonstrate the validity of its approach, an evaluation was performed on a set of data taken from the Web.

Goh et al. [13] present a smart wardrobe system, based on Radio Frequency Identification (RFID) technology where tags are placed on clothes in order to be identified by the mobile device.

Finally, another study proposes a system for the recognition of patterns in clothes and the dominant color in each image. The system is finger-based camera that allows users to query clothing colors and patterns by touch [14].

With regard to electronic devices, the Colorino, has come to fill the difficulties of the blind in the distinction of colors for the most varied tasks, since it helps in the choice of clothes, the washing procedure and the color combination [15].

There is also the ColorTest 2000, which is a device similar to Colorino that, like this one, does the identification of the colors, but also can read the dates and hours and detects if a light of the house is on or off [16].

Although there has been a great effort to develop systems to aid visual impaired, there is yet no solution capable of covering all the questions proposed in the scope of this project, namely an automatic system for combining and identifying the state of pieces of clothing for the visually impaired.

Thus, the scope of research presented in this paper has two modules: the first one is focused in algorithm of image processing and machine learning that will complement this second, presented in this paper, the mechatronic system. The algorithm will be the base of all system and has been developed in sense to be flexible for be adapted to other future situations. There are some projects that has been taking in account to develop our model [17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36].

In this context, the goal is to develop a physical prototype able to identify garments, colors, patterns having the ability to make autonomous fabric suggestions of clothing combinations for the user.

2 Related Work

There are various versions of smart virtual closets that have been applied in the current market. However, despite the different versions of virtual wardrobes there was no study on the consumers’ attitudes [6].

In order to overcome this gap, Perry [6] performed and published a study on the intention of adoption of consumers of smart virtual cabinets. In this study a positive attitude towards smart virtual cabinets was led by two beliefs about the product, utility and ease of use. Utility and ease of use accounted for 84% of the variation in consumer attitudes towards the smart virtual closet. The results confirmed that the high utility of the product is one of the decisive factors of people’s attitude towards intelligent virtual cabinets, and the low complexity of the operating system is a significant secondary determinant of consumer attitudes.

Recently, Bhowmick and Hazarik [7] published a survey, where 3010 papers were collected focused on the vision of assisted technology for visually impaired and future research trends. In conclusion, the authors emphasize the importance of the technological advance dedicated to blind people, referring to the increase in the functionality of the mobile technologies, advances in computational vision processing algorithms, miniaturization of electronic devices and the new cutting-edge medical interventions that should boost this field for the challenges and reality of successful technology creation.

Some fashion apps have been developed as STYLEBOOK, an application to manage the clothes, create outfits, and plan what to wear during the month [8]. Mode-Relier application [9] is another example that takes into account the skin-tone, hair color and even make up products. ShopStyle [10], allows to plan purchases based on the user favorite’s stores, searching items across the web. Tailor [11] is a closet that learns the user’s preferences and choices and suggests the combinations to wear. This system uses sensors called ‘TailorTags’ which are embedded in the clothes in order to detect the items and keep tracking of daily choses, learning and make suggestions [11].

A recommendation system based on semantic content to advise clothing items for recycling was proposed in [12]. In order to demonstrate the validity of its approach, an evaluation was performed on a set of data taken from the Web.

Goh et al. [13] present a smart wardrobe system, based on Radio Frequency Identification (RFID) technology where tags are placed on clothes in order to be identified by the mobile device.

Finally, another study proposes a system for the recognition of patterns in clothes and the dominant color in each image. The system is finger-based camera that allows users to query clothing colors and patterns by touch [14].

With regard to electronic devices, the Colorino, has come to fill the difficulties of the blind in the distinction of colors for the most varied tasks, since it helps in the choice of clothes, the washing procedure and the color combination [15].

There is also the ColorTest 2000, which is a device similar to Colorino that, like this one, does the identification of the colors, but also can read the dates and hours and detects if a light of the house is on or off [16].

Although there has been a great effort to develop systems to aid visual impaired, there is yet no solution capable of covering all the questions proposed in the scope of this project, namely an automatic system for combining and identifying the state of pieces of clothing for the visually impaired.

Thus, the scope of research presented in this paper has two modules: the first one is focused in algorithm of image processing and machine learning that will complement this second, presented in this paper, the mechatronic system. The algorithm will be the base of all system and has been developed in sense to be flexible for be adapted to other future situations. There are some projects that has been taking in account to develop our model [17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36].

In this context, the goal is to develop a physical prototype able to identify garments, colors, patterns having the ability to make autonomous fabric suggestions of clothing combinations for the user.

3 System Goals

This work has as main objective which is the development of a low cost system, capable of supporting the visually impaired in performing the tests of inspection, identification and combination of garments.

In order to achieve this goal, specific objectives were set, in particular:

  • Development of an algorithm for image acquisition, processing and analysis;

  • Based on the previous algorithm, it is intended to develop software that is intuitive, easy to use, respecting accessibility rules;

  • Development of artificial intelligence algorithms for combinations, suggestions and learning of the user’s preferences;

  • Building a database for storing all the information that will support artificial intelligence algorithms;

  • Development of a mechatronic system for garment inspection;

  • Construction of a physical prototype to prove the methodology and project developed.

In pursuit of the objectives, the following research question arises:

  • How can a mechatronic device with artificial intelligence make the inspection, identification, combination and management of clothing for a blind person?

4 Prototype Development

The development of this prototype should consider parameters, as dimensions, response time, illumination and interface, among others.

4.1 Assumptions

For the scope of this work, it is assumed that clothing manufacturers embedded Near Field Communication (NFC) tags in their garments, at the time they are produced, where their characteristics such as: type, color, size, season, washing process and pattern are saved.

The prototype uses a Near Field Communication (NFC) reader to recognize it automatically (see Fig. 1).

Fig. 1.
figure 1

NFC tag embedded in clothing able to be read.

Full size image

The system was designed in sense that any device could be in touch with the smart closet. In this sense, the smart closet and all the devices are connected via the Internet.

4.2 Illumination

In an artificial vision application, its viability depends on the lighting factor.

The dark field is an annular illumination system perpendicular to the camera’s capture axis. This technique is used to highlight surface defects or codes recorded on a surface.

The backlight are panels of homogeneous and diffuse light usually rectangular in shape. They are usually used by placing the part to be examined between the illumination and the camera, allowing to recognize the silhouette of an opaque object by contrast. There are many applications that require diffused lighting, which eliminates any kind of reflection; depending on the applications it can be used: coaxial lighting system, dome or diffuse low-relief lighting systems.

In order to achieve a homogeneous light field a light diffusion plate is used (see Fig. 2). A diffused illumination is used to reflect the properties of the clothing on the camera.

Fig. 2.
figure 2

Diffused illumination.

Full size image

4.3 Smart Closet Features

In order to minimize hardware costs, the image processing and artificial intelligence algorithm will be processed in a centralized unit, cloud server.

Focused only on a smart closet, there are features that will be available and present in prototype projection like:

  • Identify the garment and check the condition, whether it is wrinkled, dirty or ready for use;

  • Save user feedback of clothing, meaning the rating of clothe combination;

  • Compare its initial state with actual moment in order to recognize some degradation;

  • Identify color for possible mistakes occurred during wash that have leads its changed.

4.4 Human - Machine Interface (HMI) Accessibility

This system will interact with the user and the Web application is the most usable way, since it could be access in anywhere. However, as the users are blind we need to provide mechanisms to assure proper accessibility requirements as users need to access the data by smartphone, or other device, even a tactile screen which is only possible through a combination of software and hardware.

The accessibility is an important factor with regard to the use of devices by people with visual impairments.

A study was performed [37] to help systematize research on visual impairments and mobile touchscreen interaction by providing an overview of the major causes of visual impairments that affect the effectiveness of touchscreen interaction and efficiency in smart mobile devices.

Other research refers to the opposing differences regarding the usability and aesthetics of Web content. Web aesthetics are found to increase purchase intent and search activation. Although web usability has a stronger effect on the purchase intention than the aesthetics of the web [38].

The W3C (World Wide Web Consortium) mentions the requirements to be fulfilled in the realization of graphic interfaces on the web [39].

In this way, our system will assure all usability rules, in interaction with a user in order to provide a total user-friendly system.

4.5 Hardware

The hardware, and its functions, is based on the prototype requirements and includes:

  • Camera - color, state and dirty detection;

  • Speaker - report all output for user by sound;

  • Display - could be used for informative user and replacement of smartphone;

  • Microphone - allows interaction and voice commands by the user;

  • NFC Reader - readout all the information about the garment;

  • DC Motor - allows to rotate the garment inside the closet;

  • Braille Buttons - essential for inputs, like confirmation, feedback and start interaction by user.

In this way, the prototype besides being possible to pair with the smartphone will contain features that can be used separately. Thus, the prototype is accessible to all users, even those who do not want to use smartphones as an interface.

The unit controller chosen was Raspberry Pi (see Fig. 3). It is a small computer integrated in a single plate. Its versatility and low price allow it to be used in countless ways.

Fig. 3.
figure 3

Raspberry Pi and camera module [40].

Full size image

Its characteristics include [40]:

  • Broadcom BCM2837B0, Cortex-A53 (ARMv8) 64-bit SoC @ 1.4 GHz;

  • 1 GB LPDDR2 SDRAM;

  • 2.4 GHz and 5 GHz IEEE 802.11.b/g/n/ac wireless LAN, Bluetooth 4.2, BLE;

  • Gigabit Ethernet over USB 2.0 (maximum throughput 300 Mbps);

  • Extended 40-pin GPIO header;

  • Full-size HDMI;

  • 4 USB 2.0 ports;

  • CSI camera port for connecting a Raspberry Pi camera;

  • DSI display port for connecting a Raspberry Pi touchscreen display;

  • 4-pole stereo output and composite video port;

  • Micro SD port for loading your operating system and storing data;

  • 5 V/2.5 A DC power input;

  • Power-over-Ethernet (PoE) support (requires separate PoE HAT).

In addition, it is possible to incorporate one of its two cameras, which provide sufficient features for a first exploration of image acquisition. The characteristics of camera v2 can be found in Table 1.

Table 1. Camera module v2.
Full size table

4.6 Prototype Dimensions

The prototype should have, at least, the following dimensions:

  • Height: 180 cm;

  • Width: 130 cm;

These dimensions took into account the Horizontal Field of View (HFOV) of the camera as well the vertical. By this way Eq. (1) show the trigonometry equation to calculate the dimensions based in FOV of camera.

$$ {\text{HFOV}}\left[ {\text{mm}} \right] = 2\times {\text{OD}}\left[ {\text{mm}} \right] \times { \tan }\left( {{\text{AFOV}}\left[ {\text{degrees}} \right] / 2} \right) $$
(1)

Where the distance to object (OD) from the camera as well the Angle of Field of View is necessary to obtain the horizontal distance, as well as, vertical distance.

The first version of the proposed prototype is presented below (see Fig. 4).

Fig. 4.
figure 4

Proposal prototype – smart closet. a - Speaker; b - Display; c - Microphone; d - Braille Buttons; e - NFC Reader; f - DC Motor; h - Camera; i - Led panel for illumination.

Full size image

In summary, the prototype of the smart closet has a back panel with led diffusing light that allows homogeneous light, where in the middle of the panel is placed a camera that will take photos in order to inspect the garment.

In the side of light panel, the user is able to interact with the smart closet by voice and receive output by the speaker. The NFC reader can collect all information from clothing tag and associate to the database its characteristics.

In order to inspect the two sides of the garment a motor is placed in order to rotate it.

The buttons in braille allows the user to interact with the system. The smartphone can also be connected with the closet allowing to control the system.

The system is connected to a cloud server that is responsible for processing the information. The system is connected to a cloud server that is responsible for processing the information relative to image processing and machine learning. By this way, the data processing is centralized and consequently the costs associated to the hardware integration with high capacity to handling the data are avoided.

This design takes into account the fact that we want to control the clothes in a controlled environment.

5 Final Remarks

It is important to recognize that clothing aesthetics are of central importance in the life of blind people, who are happily increasingly inserted in the labor market, have an active social life, and as with most people in the same circumstances, they need to be accepted and recognized.

A duly qualified or rehabilitated blind person has all the conditions to, if he/she wishes, be able to live alone or even to have a marital relationship with another blind person. However, blind people experience some difficulties with regard to clothing. Some are overcome with the help of family or friends, others with a great organizational capacity.

In order to select and match the clothes the person wears daily, it is crucial to have a faultless organization and internal discipline. During the purchase of the garment, the blind person asks the other person for help in describing the garment with as much information as possible, in order to distinguish from the other garments, basically looking for something different, such as a tag, a different area, a cutout, deep down the idea is to find a distinct brand in the clothe.

The next step is to organize the different types and colors of clothes in drawers and drawers, in order to become easier to pick up a particular piece that it is intended to wear that day. However, such a thorough process can be prone to errors, particularly when washing clothes and having to replace them where they are removed, or when using luggage, where the organization is extraordinarily more difficult.

Another difficulty is to detect dirt or stains on clothing. Except for situations where stains can be detected by touch, for the most parts this is objectively impossible.

Against this backdrop of some difficulties, technology can be a very interesting ally in which a smart wardrobe would be welcomed.

Thus, the mechatronic system proposed aims to help blind people to overcome their difficulty in choosing clothes. So, this system proposes a new concept for significantly improve the daily of the blind, allowing the blind to combine garments, identifying dirt and garments that are not in good condition. Moreover, the user could benefit from the automatic suggestions resulting from the machine learning algorithms.

This first prototype also intends to evaluate the interaction of the user with the system and verify the effects in terms of well-being and self-esteem, resulting from greater autonomy of the blind.