Keywords

1 Introduction

Over the last years, there has been an effort to develop technologies that provide handicapped people access to information and knowledge. In the specific case of blind people, there are plenty of technological solutions that help in activities of daily routine. However, there are several challenges related to daily basic tasks, especially in the choice of personal garments. These individuals cannot choose autonomously the desired clothing and, consequently, the identification of features on clothing becomes a slow and difficult task. This paper presents an automatic wardrobe that allows an improvement in the quality of life and well-being of blind people. The main goal of the work under development is the design, development, and validation in ACAPO (Portuguese Association of Blind and Amblyope People) of a prototype of a mechatronic system for the choice and management of garments that includes the automatic wardrobe. In the future, this physical system will be integrated with a mobile application for iOS that is part of the system MyEyes [1,2,3] and will allow a virtual wardrobe replication in the smartphone, making management easier and intuitive.

This paper has seven sections. After the introduction given in Sect. 1, Sect. 2 describes the state of the art; Sect. 3 presents the system overview; the hardware architecture and the software architecture are presented in Sect. 4 and Sect. 5, respectively; the preliminary results are presented in Sect. 6; Sect. 7 presents the conclusions and some ideas regarding future work.

2 Previous Work

The literature review shows that there are several solutions to help people in the choice of personal garments. These solutions are based on virtual wardrobes, a concept with a great increase in the last years. It is important to know the consumer’s attitudes about this new approach. In [4], Perry et al. present a study that focuses on consumers’ acceptance of smart virtual wardrobes. The results show that there is a great acceptance of the concept of smart wardrobes. The utility and ease of use are key factors for the choice.

A survey carried out encompassing all the delegations of ACAPO (Portuguese Association of Blind and Amblyope People) allowed the identification of several problems regarding garments identification by blind people [5]. The main goal of the survey was to recognize both the importance of garment identification and several technological solutions existing for this specific community. It was possible to identify that there are several solutions, but they do not focus on the aesthetic aspect of managing garments. MyEyes, presented in [1,2,3] was developed to overcome this gap. The solution is based on a system that integrates a mobile application with an Arduino board allowing users to have a virtual wardrobe with their garments. Near Field Communication (NFC) tags attached to the garments allow the addition of clothing items to the virtual wardrobe. A general overview of a possible prototype implementation for this solution was presented in [6].

In [7] is presented a solution to help blind people match garments. The system integrates NFC technology with a smartphone, allowing visually impaired people to choose the desired clothes. The solution presented in [8] explores NFC technology combined with Quick Response (QR) technology with the main goal of developing a clothing matching system with audio description.

Solutions whose main target is people without any or partial visual disability are in a crescent rising with more implementations emerging. Goh et al. [9] propose a system that integrates tags with Radio Frequency Identification (RFID) technology, allowing unique identification of clothing items. The system is based on RFID tag reading and the user obtains the features data that were saved on each tag. This system is controlled by an application that allows garment management and suggests clothing items based on several criteria, such as style, color, material, and user’s mood. Due to the increasing demand for implementations that aid in garments management, some fashion brands present different solutions developed to help people plan what to wear. In 2017, Amazon presented the Echo Look [10] based on a kit with a camera that allows garments photo capture and cataloguing of outfits. This method also suggests combinations based on meteorology and users’ preferences. The mobile application Fashion API [11] is a closet that plans what to purchase and adds garments based on QR code reading. Another mobile application is the Smart Closet [12], which plans combinations to wear and allows the addition of clothing items based on photo capture. TailorTags [13] is a system that uses smart tags to detect garments automatically. Also suggests combinations based on users’ preferences or meteorology. Table 1 summarizes the advantages and limitations of the systems previously described.

Table 1. Advantages and limitations of several smart wardrobe solutions.
Full size table

Besides all efforts to develop systems to help blind people, except for solutions presented in [1,2,3, 7], and [8], the systems in Table 1 focus on solutions that help blind people. So, based on the gaps and limitations of the solutions presented in Table 1, an automatic wardrobe capable of improving the quality of life of blind people is presented in this paper.

3 System Overview

The developed system has two main parts: the physical prototype and the system software. The system integrates an NFC reader whose responsibility is to read the tags attached to each garment. The circular movement of garments is the responsibility of a stepper motor screwed to the wardrobe’s roof. In each hanger is included a servo motor that allows a 180 degrees rotation when the photo capture takes place. This rotation allows the capture of a front and a back photo. On photo capture, on each capture moment, two photos are taken (a top and a bottom) and submitted to a stitching algorithm. This system is based on a Raspberry Pi 4, which is responsible for data processing and sending the data to the user when requested. Regarding to lightning, the wardrobe contains an inside illumination to evidence garments features and eliminate any kind of reflection with the main goal of providing a controlled system for garment photo capture allowing the avoidance of dark fields and shadows (Fig. 1).

Fig. 1.
figure 1

System overview

Full size image

4 Hardware Architecture

The system encompasses four phases: NFC reading, photo capturing, hanger rotating, and circular movement. The NFC module reads the tags attached to each garment allowing the obtainment of the UID code associated with each tag. The photo capture is the responsibility of a vision system with one camera. Due to the impossibility of capturing the full of view of a garment with only one shoot, a small servo motor is connected to the camera to rotate 180 degrees allowing two positions on photo capture. With this solution, a top photo and a bottom photo are taken and submitted to a stitching algorithm that joins the photos, originating a complete garment photo. In each hanger, there is attached a servo motor providing a 180 degrees rotation which allows the capture of a front and a back photo. Regarding the circular movement inside the wardrobe, a stepper motor was screwed to the roof whose shaft had connected to a circular platform where the servo motors attached to the hangers were connected.

4.1 Illumination

Illumination is a key factor in photo capture. The garments should be illuminated avoiding any dark fields so, the colour should be white and diffused. The type of illumination chosen was a LED strip that allows great flexibility for a low price. This strip is constituted by hundreds of LEDs that emit light with a temperature of about 6000K being located on the wall where the cameras are allowing a higher focus on each garment.

4.2 Prototype Design

The goal of the project presented in this paper is to develop an automatic wardrobe able to improve the quality of life and well-being of blind people. For this proof of concept, it was considered a small wardrobe from IKEA [14] with the following dimensions: 50x30x80 cm.

As mentioned before, on the wardrobe roof, there is a stepper motor responsible for the circular movement. A circular platform is coupled to the motor shaft that supports the weight of all servo motors attached to each hanger. The servo motors responsible for a 180 degrees rotation of each garment are attached to the circular platform inferior surface. The NFC reader is screwed to a side wall to allow the collision and detection of User Identification (UID) associated with each tag. The camera is connected to a small servo responsible for a 180 degrees rotation allowing the capture of two photos (Fig. 2).

Fig. 2.
figure 2

Hardware overview

Full size image

The microcontroller used as a base for the system implementation was the Raspberry Pi 3B + board, which is used in complex projects, offering great versatility for a low price. The 40 pins on board can be used for several functions, such as Inter-Integrated Circuit (I2C), Serial Peripheral Interface (SPI), and Universal Asynchronous Receiver-Transmitter (UART) communication. This minicomputer has 1GB of RAM and has inserted a Secure Digital (SD) Card with 16GB of storage that allows the installation of the Raspbian Operative System. For the tags reading was used a PN532 module that communicates via NFC. This module allows I2C, SPI, and UART connection to the Raspberry Pi board and can detect tags up to 4 cm of distance.

Two types of actuators were used: servo motors and a stepper motor. Each servo motor was connected to the hangers to warrant a 180 degrees rotation of each garment when the photo capture takes place. A small servo was connected to the camera to allow two positions on photo capture. The stepper motor was screwed to the roof allowing garments movement inside the wardrobe. This stepper is connected to ULN2003 driver that controls the motor rotation.

For the photo capture, it was used one OV5647 camera. This module integrates the camera with a small board responsible for the connection to the BCM2835 processor via Channel State Information (CSI) communication. The camera has 5 MP and delivers images with a resolution up to 2592 x 1944. This module is connected to the Raspberry Pi over the Channel State Information (CSI) bus.

5 Software Architecture

A control software was developed in order to test and validate the wardrobe automatism. The state machine presented in Fig. 3 describes the functioning of the system, which has six states: READY, STEPPER, DETECTION, CAPTURE, SERVO, and STITCH. When the user requests a garment, the initial state READY is changed to STEPPER. In this state, the stepper motor that controls the garments movement inside the wardrobe is triggered and moves until the tag with the requested UID is detected and stops. The state changes to DETECTION and the UID tag is displayed. Then, the system changes its state to CAPTURE, and a top and bottom front photos of the garment are taken. The system state changes to SERVO and the servo motor is triggered, performing a 180 degrees rotation. After this rotation, the system returns to CAPTURE state, and a top and bottom back photos are taken. The system then changes to the STITCH state, where a stitching algorithm combines the photos. The stitching algorithm allows solving a problem related to the impossibility of capturing the total full of view of each garment with only one camera shoot. So, this algorithm, which is part of the OpenCV library, combines images based on keypoint detection and overlaps common points on each picture.

Fig. 3.
figure 3

System State machine

Full size image
Fig. 4.
figure 4

System Flowchart

Full size image

6 Preliminary Results

Some preliminary results allow the validation of the different modules previously presented. The automatisms responsible for the movement are already developed and implemented. The stepper motor has a circular platform connected to the shaft and is connected to ULN2003 which is a driver that controls the motor rotation. This is a module that integrates the motor and the driver on the same board. The circular platform is where the servo motors, responsible for each hanger, are connected. Each motor has a hanger connected to the shaft to warrant the rotation on the photo capture. The control software developed allows the test and validation of the wardrobe automatism, i.e., the integration between the hardware and the software. The flowchart presented in Fig. 4 describes the flow of the automatism implemented.

The system flow starts with the stepper motor being triggered until the tag detection. When the requested tag is detected, a top and a bottom front photos are taken, and the servo motor rotates 180 degrees in order to a top and bottom back photos be captured. Next, the photos are submitted to a stitching algorithm originating two photos (a front and a back). This stitching algorithm was implemented to solve a gap related to the garment photo full of view. As can be seen in Fig. 5, this method was capable of effectively detecting several common points in each photo and matching them in the final photo.

Fig. 5.
figure 5

a) Garment Top photo Algorithm Test. b) – Garment Bottom photo Algorithm Test. c) – Garment Complete Photo

Full size image

The developed automatic wardrobe has the main capability of guaranteeing a controlled system on photo capture avoiding dark fields and shadows. A garment was tested on the automatic wardrobe to show the accuracy of the controlled system on photo capture. Figures 6a) and 6b) show that in capture outside the controlled system is not guaranteed a photo without shadows or dark fields. On the contrary, inside the automatic wardrobe, the photo is taken, and the illumination evidences the garment features and eliminates any kind of reflection (Fig. 6c).

Fig. 6.
figure 6

a) Garment photo taken outside the con-trolled system (Day). b) Garment photo taken outside the con-trolled system (Night). c) Garment photo taken inside auto-matic wardrobe

Full size image

Comparing the proposed system with the solutions described in Sect. 2, only in [9] is presented a physical system. However, that system has only the capability of storing the garments, contrasting with the system proposed in this paper, that is capable of not only storing the garments but also capturing garments photos inside the physical system, as well as integrating an automatism that guarantees the pieces movement when requested by the user. The other solutions mainly focus on management apps with wireless technology. This feature is also present in the system proposed in this paper, which uses NFC communication by attaching a tag to each garment in order to ease the addition and removal of pieces.

7 Conclusions and Future Work

In this paper the general overview of an automatic wardrobe for aiding blind people was presented. The hardware that controls the system was already implemented and tested. The next step is the connection with a mobile application with an implementation of a server in Raspberry Pi to manage the requests from the users. With this integration, a background removal and a color detection algorithm will be implemented. These methods will be part of the image processing techniques that will be applied to the garments photos captured by the physical prototype camera or by the smartphone camera. These algorithms will help classifying the type of garments and identifying the predominant colours of each piece. These approaches will be implemented on the mobile application, limiting the Raspberry Pi to the implementation of the stitching algorithm.

As this project is a proof of concept, the system presents some limitations, such as the wardrobe small size. This hindrance allows only the use of small garments and a reduced number of clothing items inside at the same time. After testing the system with the small wardrobe, the goal is to validate the complete system with a higher wardrobe capable of allocating larger garments.

Based on the automatic wardrobe developed and the integration with the mobile application MyEyes, the final mechatronic system will propose a concept of wardrobe which allows an improvement of blind people’s daily routine, specifically in garments identification and management.