Currency Recognition App for Visually Impaired Users in India

Abstract

A system for the recognition of currency notes is one kind of intelligent system which is a very important need for visually impaired and blind users in the modern world of today. In this paper, we present a currency recognition app applied to Indian currency notes. Our proposed system is based on interesting features and correlations between images. It uses the Convolutional Neural Network for classification. The method takes Indian rupee paper currencies as a model. The method is quite reasonable in terms of accuracy. The system deals with the images of all the currency note denominations, some of which are tilted to an angle less than 150. The rest of the currency images consist of mixed, noisy, and normal images. It uses the current series (1996–2020) of currency issued by the Reserve Bank of India (RBI) as a model currency under consideration. The system produces an accuracy of recognition of 94.38% and gives an audio output to the users. The proposed technique produces quite satisfactory results in terms of recognition and efficiency. In the future, this app can be improved by adding a dataset of other currency notes of the world.

1 Introduction

Modern automation systems in the real world may require a system for currency recognition. It has various potential applications including banknote counting machines, money exchange machines, electronic banking systems, currency monitoring systems, and/or assistance to visually impaired people. The recognition of currency notes is a vital need for visually impaired or blind people. They are not able to differentiate among currency notes correctly. It is very easy for them to be cheated by other fraudulent people. Therefore, there is a genuine need to design a system to recognize the value of currencies easily regardless of their orientation, wear and tear, illumination, scaling, and other factors that may reduce the quality of the currencies. The World Health Organization (WHO) has estimated the number of visually impaired people all over the world to be about 285 million. Out of which 39 million people are completely blind, and the rest have low vision [12]. Several studies have been conducted for them and a few techniques for currency recognition have been proposed. Indian currency notes contain several features which enable the visually impaired (color-blind, partially sighted, and entirely blind people) to identify them, viz. intaglio printing, tactile mark, variable currency note size, large numerals, variable color, monochromatic hues and patterns.

Fig. 1.

Sample notes of Indian national rupees (INR) 20 and INR 100 with identification marks

There is a feature providing identifiable and differentiable shapes for various denominations i.e., INR 20 in Vertical Rectangle, INR 100 in Triangle, as depicted in Fig. 1, and INR 500 in Circle. But these identification marks may fade after some years. Other techniques have focused on textures, colors, and/or sizes of currency notes. They have some limitations like if the currency note is worn out by excessive usage or if the image is not clear, then it gives an unpredictable output. Most of these techniques are sensitive to illumination conditions and may rely on taking images at fixed environment settings with camera location and image background.

The computational power and camera availability of current mobile phones make them a suitable candidate for currency recognition. In this paper, a mobile system for currency recognition that can recognize the Indian currency notes is proposed. A mobile application is developed to identify currencies that are partially visible, folded, wrinkled, or even worn by usage. The proposed system can recognize the current Indian currency notes of all types and that includes currency notes with INR values of 5, 10, 20, 50, 100, 200, 500, and 2000. This system employs voice communication to inform users about the identified value of the currency through the mobile speaker. This research paper has mainly described the design and development process of a currency recognition app for visually impaired users in India.

This currency recognition application for visually impaired users has been developed based on the generalized Human Work Interaction Design (HWID) framework, proposed by Orngreen et al. [13], as seen in Fig. 2. Human work aspects such as cognitive abilities and limitations of visually-impaired users, along with orientation, wear and tear, and scaling of currency notes are considered in the design of currency recognition application. A couple of visually-impaired users were involved, and their user experience has been considered in interaction design of this system. The environmental as per general HWID frame-work such as environmental illumination has been a vital factor in recognition of Indian currency notes. The contextual factors such as note orientation, texture, size wear and tear are also required to be considered in HWID context. Thus, the effect of HWID perspective [13, 14] can be observed prominently in system design of proposed currency recognition app.

Fig. 2.

Generalized HWID framework [13]

Visually impaired users have challenging life in developing countries like India. They have lesser education as well as employment opportunities. Only about 29% of visually impaired people go to school and are part of the education system. The employment rate among them is about 30% and they are getting jobs in customer care, hospitality and teaching fields [15]. These visually impaired users, including unemployed among them, have challenging socio-economic life and may face financial difficulties. Most of these users have to deal with money – the currency notes during their everyday life. The feature providing identifiable and differentiable shapes for various currency notes may not be useful as these shapes fade away after initial years of usage. The visually impaired users fail to differentiate among currency notes, even with tactile marks, variable note sizes, large numerals or variable colors, and it makes them easy targets for cheating by other dishonest people. Most of them use economical Android mobile phones. Therefore, there is a dire need for designing a mobile application, which will recognize the value of currency notes for these visually impaired users.

The real challenge for this currency recognition app is dealing with older currency notes, which are worn out or faded due to excessive usage. The illumination conditions and low-resolution cameras may lead to unclear images, making the currency detection even more challenging. The audio output, conveying note value to the visually impaired users, also face challenge of noisy surroundings. The proposed currency recognition app is an innovative and successful attempt to deal with these challenges and help visually impaired users to detect all currency notes in India.

2 Related Work

A related work of related research papers has been conducted to study image processing and image acquisition techniques. The research papers focusing on the recognition of Indian currency notes were studied. There are various techniques for currency recognition that involve currency texture, pattern, and/or color.

In the first paper, the authors - Vishnu et al. have proposed an interesting recognition method [11]. The currency images are normalized by using histogram equalization. They extracted five features (Shape, Center, RBI Seal, Micro Letter, and Latent Image) from images of currency notes by placing a rectangular box of specific dimensions, which discovers the Region of Interest (ROI). The extracted ROI can be used with Pattern Recognition and Neural Networks Matching techniques. First, they acquired the image using a simple flat scanner with fixed dpi, and with a particular size. A few filters were applied to extract the denomination values of currency notes. They used different pixel levels in different denomination notes. The Pattern Recognition and Neural Networks matcher techniques were used to match or find currency value/denomination of paper currency.

In another paper, the authors have chosen Faster RCNN (Region-Based Convolutional Neural Network) to train models and recognize Indian currency very well [1]. Faster RCNN has improved the running time of the network. They used Regions Proposal Network (RPN), which shared convolutional features of images with the network, and it simultaneously predicted the object and bounds with the score at each position. RCNN was trained to generate high-quality region proposals which were used by RCNN for faster detection.

In the next paper, six characteristic features were extracted [6]. The approach used includes several components such as image processing, edge detection, image segmentation, features extraction, and comparing images. The features extraction was performed on the image of the currency and it was compared with the features of the genuine currency. The Sobel operator with gradient magnitude was used for feature extraction. Paper currency recognition with good accuracy and high processing speed was of great importance for the banking system [4]. This study shows that by using the Local Binary Pattern (LBP) algorithm, 100% accuracy can be achieved in the case of good quality images and low computational cost is needed to meet the high-speed requirement in practical applications. The algorithm is highly used for texture analysis and feature extraction.

3 Peer App Review

A systematic review has been conducted for currently available Android apps related to Indian currency recognition. The selected apps include Mobile Aided Note Identifier (MANI) [7], IndicAI Vision [5], Roshni [8], Cash Reader [2], and Third Eye [10]. The app review is focused on five identified Android apps and aimed to provide interesting insights into functionalities and other design aspects. To examine such aspects of selected Android apps, an expert-based peer app review has been conducted by authors.

Initially, a review of five selected apps has been conducted based on app features and functionalities. These features/functionalities include App accuracy, Screen Reader, Voice- based User Interface (UI), Text-to-Speech, Voice-based Instructions, Easy Navigation, Fake Note Recognition, Active Feedback, and Multi-lingual Support. App Accuracy is calculated based on their accurate recognition of denomination of currencies for a certain number of currency scans for different denominations. Their comparative details are depicted in Table1.

The formula used for calculating accuracy (%) is given as –

$$ {\text{Accuracy = }}\frac{{{\text{Number of True Positives}}\, \times \,100}}{{\text{Total Number of Currency Scans}}} $$

Here, true positive denotes correct recognition of each denomination. The total number of currency notes scans includes 20 scans for each denomination.

Table 1. Comparison of selected currency recognition apps based on features and functionalities

The following observations are made after carefully reviewing the selected apps as seen in Table 1, based on the app features and functionalities:

• a. Only one App provides Fake Note Recognition functionality.

• b. Few of the Apps (2/5) provide Easy navigation using Voice-based UI.

• c. Most of the Apps (3/5) provide Multilingual support.

• d. Few of the Apps (2/5) provide Voice instructions for first-timers or naive users.

• e. In the current scenario, MANI is the most suitable app for visually-impaired users to recognize Indian currency with features like Active Feedback and Easy Navigation along with better accuracy.

4 Study of Indian Currency Notes

The government of India has introduced a currency note of INR 2000 in 2018. Its security features are depicted in Fig. 3. These features include watermark, micro- lettering, optical variable link, security thread, and latent image. Following are some common features which can be useful to blind users or visually impaired users in recognition of the currency notes.

Fig. 3.

Sample note of INR 2000 with its security features

Braille Marking or Bleed Lines:

The Reserve bank of India launched braille-like markings on currency notes of INR 100, INR 500, and INR 2000. This is done to identify the currency value by visually impaired people. Depending on the number of lines printed on the note, the value of the note can be identified. The INR 100 notes have four parallel angular lines printed along the border, just beside the Mahatma Gandhi watermark. The INR 200 notes have four angular bleed lines with two circles in between the lines, whereas the INR 500 notes have five parallel lines. The INR 2000 notes have seven parallel lines as seen on the upper left side in Fig. 3.

Intaglio Printing:

The Ashoka Pillar Emblem on the left, portrait of Mahatma Gandhi, the Reserve Bank seal, guarantee-promise clause, Reserve Bank of India (RBI), Governor’s signature is printed in intaglio printing, i.e., in raised print which can be felt by soft touch, especially for blind users.

Identification Mark:

The identification mark is on the left-hand side of each note on the front side. These marks have raised intaglio print and have different shapes for different denominations like a vertical rectangle for INR 20, square for INR 50, triangle for INR 100 and circle for INR 500. A horizontal rectangle as an identification mark can be seen for a currency note of INR 2000, on the central right side in Fig. 3.

5 Research Methodology

The research methodology is used for studying, evaluating, and analyzing the designs and design aspects of similar types of apps as well as related research work. Initially, related work and peer apps reviews have been conducted. The inferences gathered from related work and peer app review are further used in dataset preparation, model training and design of the proposed system.

5.1 Dataset Preparation

A dataset containing images of new Indian currency notes with INR values of 50, 100,200, 500, 2000 was not readily available. So, the dataset has been prepared by clicking the images of these notes and adding them into the Dataset. This dataset contains various Indian currency notes with INR of values 10, 20, 50, 100, 200,500, and 2000 with different backgrounds. This dataset had about 4000 images of notes. These images have variations in illumination, background, and rotation as seen in Fig. 4.

Fig. 4.

Sample images of Indian currency notes

5.2 Currency Recognition Model Based on Machine Learning

Fig. 5.

Process of training proposed model

A dataset containing images of new Indian currency notes with INR values of 50, 100,200, 500, 2000 was not readily available. So, the dataset has been prepared by clicking the images of these notes and adding them into the dataset. This dataset contains various Indian currency notes with INR of values 10, 20, 50, 100, 200,500, and 2000 with different backgrounds. This dataset had about 4000 images of notes. These images have variations in illumination, background, and rotation as seen in Fig. 4. The training process of proposed currency recognition model based on machine learning approach is represented in Fig. 5, along with related CNN layers and dataset.

6 Design of Proposed Currency Recognition System

According to Peer App Review, the Voice-Based UIs have been vital requirements for visually impaired users. Such users can’t select a camera and take a picture of a currency note under consideration. Therefore, this Android app has been designed in such a way that the camera will automatically open with the app.

6.1 Model Training

Convolution Neural Network [3] is the most popular deep learning neural network that is used to detect and classify objects in images. The CNN has several layers as shown in Fig. 6, which carry out feature extraction by performing different calculations and manipulations. There are multiple hidden layers like the Convolution layer, ReLU layer, and Pooling layer that perform feature extraction from an image. Finally, the output of all hidden layers is passed as input to the output layer which identifies the object in the image.

Fig. 6.

Layers of convolution neural network (CNN)

The convolutional layer has several filters that perform the convolution operation. This layer considers every image as a matrix of pixel values. The dot product of this matrix and filter is computed to get the convolved feature matrix. Extracted features are moved to the ReLU layer which performs an element-wise operation and sets all the negative pixels to 0. It introduces non-linearity to the network, and a rectified feature map is generated. Pooling reduces the dimensionality of the feature map. Various filters are used by this layer to identify different parts of images like edges, corners, color in the image. The next flattening layer converts all resultant 2-D arrays from pooled features into single long continuous linear vectors which are fed as input to the fully connected layer to detect the image as shown in Fig. 7.

Fig. 7.

Sequence of outputs from layers processed by CNN

6.2 Feature Extraction

To extract the features of the image, CNN uses a kernel over each image and adjusts the kernel as per the propagation in the network. A kernel is then convolved over the entire image to produce feature maps. This feature mapping of the input signifies the locations where a feature is present as shown in Fig. 8 for INR 100. As the layers become deeper and deeper, the network acquires the knowledge of larger and larger feature extraction. The initial layers take care of the smaller details of the image and deeper layers can identify the bigger picture.

Fig. 8.

Feature extraction of currency note of INR 100

6.3 Transfer Learning

The transfer learning technique was used to increase the accuracy of the model. Mobile Net V2 was used which could be directly used to train the dataset as it is already learned on a large dataset. The model was retrained to get better accuracy and to reduce the computational cost of the model. This learning approach is called Transfer Learning. MobileNetV2 is a convolutional neural network architecture that seeks to perform well on mobile devices. It is based on an inverted residual structure where the residual connections are between the bottleneck layers. The intermediate expansion layer uses light-weight depth-wise convolutions to filter features as a source of non-linearity. As a whole, the architecture of MobileNetV2 contains the initial fully convolution layer with 32 filters, followed by 19 residual bottleneck layers.

6.4 Model Deployment with TensorFlow Lite

TensorFlow Lite [9] is an open-source, product-ready, cross-platform Deep Learning framework that converts a pre-trained model in TensorFlow to a special format that can be optimized for speed or storage.

After the model was trained, it was converted to the TensorFlow Lite version. TensorFlow Lite model is a special format model that is efficient in terms of accuracy and is a light-weight version that will occupy less space. These features make TensorFlow Lite models the right fit to work on Mobile and Embedded Devices. TensorFlow Lite Converter converts a TensorFlow model to TensorFlow Lite flat buffer file (.tflite).

6.5 Model Testing

Once the model was trained, test images were fed to classify them into the correct classes. The results of the model are shown in Table 2. From the table, we can see that as the number of epochs increases, the accuracy and validation accuracy increase, and loss and validation loss decreases.

Table 2. Accuracy, loss, validation accuracy, and validation loss at each epoch

7 Mobile App

A mobile app was built using android studio. The user can directly open an application by using Google Assistant or users can use Talkback which is a built-in screen reader for visually impaired users. As shown in Fig. 9. the app consists of an instruction button at the top of the screen which on clicking will speak out the instruction. It also consists of the bottom sheet layout that shows the denomination of the note being predicted. When the app is opened, it will directly use the back camera of the phone to detect the note. Once the note is detected, it will tell the denomination of the note and show it on the bottom sheet layout as shown in Fig. 10.

An app can give an unpredicted output in case there is not enough light while detecting the currency note or if there is a continuous movement of the phone or the currency note that is to be detected.

Fig. 9.

System workflow diagram

Fig. 10.

Layout of currency recognition app

7.1 Additional Features

Following are additional features incorporated in this mobile app:

Optical Character Reader (OCR):

It is the electronic or mechanical conversion of images of typed, handwritten or printed text into machine-encoded text, whether from a scanned document or a photograph. It is used to detect text and numbers written on currency notes as shown in Fig. 11. It also shows the captured details of currency in light green color as seen in Fig. 11. In Settings Menu, there is an option of selecting OCR language from multiple languages as shown in Fig. 12.

Audio Help:

It reads out related information or instructions for the user, in an audio format. As shown in Fig. 12, this information contains the use of various features of app like OCR and Currency Recognition.

Settings:

It has different options such as flashlight, auto focus and multicolumn text as shown in Fig. 12. These options are discussed ahead.

Flashlight:

If the flashlight option is enabled, the flash of the mobile will be used at the time of capturing currency. It helps to detect the currency notes under low or poor lighting conditions and provide result with better accuracy.

Auto Focus:

This option helps to focus on the currency note for its accurate detection even though if currency is not stable while app is capturing currency for recognition. As shown in Fig. 12. the Auto Focus option is automatically enabled for better accuracy by default.

Zoom:

While using the OCR feature, the zoom feature helps to read the textual words more accurately.

Fig. 11.

Overall layout with inclusion of additional features

Fig. 12.

Help screen and settings menu

8 Conclusion and Future Work

In this paper, a simple Indian currency recognition system has been proposed for the blind. Visually impaired people are unable to recognize the currency notes due to the similarity of paper texture and size between different categories. This Indian Currency Recognition app helps visually impaired people to recognize and detect Indian currency notes. The current currency recognition apps face the gap of providing audio instructions to the user and of accuracy while detecting the notes. The proposed system starts with capturing still images from the front camera. For currency recognition, this application uses TensorFlow lite and uses a Machine learning classification technique to detect currency based on images or paper using a mobile camera. This app has been designed using android studio. It also has an instruction button on the top that reads out the instructions for the user when clicked.

The Mobile Net V2 model was used to train our image dataset similar to this paper [1]. This author used Faster RCNN as a pre-trained model. As we built an android app, Mobile Net V2 is a more suitable model as compared to Faster RCNN as Mobile Net V2 reduces the complexity cost and model size which is suitable for devices with low computational power like mobile. TensorFlow model is converted into TensorFlow lite model to deploy it on the android app.

However, the dataset used for this app contains pictures of only Indian currency notes. So, with this system, the user cannot detect the denomination of other currencies. This can be done by including the dataset of other currency notes and training them, respectively. Thus, we expect our system to easily adapt to other currencies of the world while keeping at a similar level of accuracy and speed. This app also needs modification in terms of the user interface to be able to prompt the user in case the note has not been placed properly under the camera or is not detected.

In future, the proposed app will be added with several functionalities with initial focus on improving accuracy and other currencies recognition. App will also get the modification of prompting the user if currency is held too much close or out of camera reach to capture it properly in subsequent versions. There is also a plan of providing user with the guide through various functionalities of app using audio input and output so that it will be convenient for a blind user to go through all available functions. The recognition of fake currency is also one of the functionalities that will be incorporated in app.