Keywords

1 Introduction

Major technological advancements gravitate towards automation of systems and processes that would need minimum to no human intervention. Automation in retail can be achieved through smart shopping trolley whose aim is mechanization of user shopping experience by CV using Internet of Things and cloud-based web technologies. Rupanagudi et al. [1] have developed a cost-effective smart trolley prototype using RFID, Wi-Fi, Zigbee. The smart trolley developed is enabled with a web camera along with video processing capability. Sainath et al. [2] have proposed an automated Shopping Trolley which integrates a Raspberry Pie Embedded Chip with two Bar code Scanners and a Battery kit to allow users for self-checkouts at Super Markets. Wankhede et al. [3] have proposed an Electronic Shopping Trolley for getting products scanned using a barcode scanner. Lekhaa et al. [4] have designed an intelligent shopping cart using IoT consisting of barcode scanner, Liquid Crystal Display (LCD) display, Bolt ESP8266. Sutagundar et al. [5] have designed IoT-based smart shopping mall system that uses ESP8266 Wi-Fi module along with web application that enables customer login and data entry to the cloud for automatic payment. Lestari et al. [6] have proposed a method for automatic trolley control that follows a user based on the clothes color; coordinate locations, speed of movement, and size of the object captured by the camera. The proposed system uses Python application Camshift algorithm to develop the model. Athauda et al. [7] have proposed a low-cost, robust, passive Ultra High Frequency (UHF) RFID-based shopping trolley that allows tracing and processing shopping data in real time. Sarala et al. [8] have developed a prototype using an Arduino ATmega2560, Infrared (IR) sensor, barcode acceptance button, power supply, LCD display, Global System for Mobile communications (GSM) modem for automatic identification of shopping items. Viswanadha et al. [9] have proposed a smart shopping cart is with barcode scanner and touchscreen display to scan the products and display the product information, cost and total bill. Awati et al. [10] have designed an AtmelAT89V51 microcontroller-based Trolley by using optical sensor, barcode scanner, RF Transmitter-Receiver to design a smart trolley for malls. Sanap et al. [11] have developed a prototype of Smart Mobile Autonomous Robotic Trolley (SMART) using ATmega 16 microcontroller interfaced with EM18 RFID reader module, buzzer and Light Emitting Diode (LED) indicator. Data transfer between the trolley billing system and billing station system is done via an XBee ZigBee. Dhianeswar et al. [12] have proposed a Smart Shopping Trolley to follow the customer automatically using a Kinect Sensor.

2 Proposed System

In the era of Industry 4.0 and Industrial Internet of Things (IIoT), modern sensors and actuator control have enhanced the conventional machines to smart machines. In this paper, we are proposing a smart mechanized shopping cart using embedded system and motor drive control circuitry based on computer vision algorithm-based real-time shopper tracking. Figure 1 shows proposed Smart automated shopping cart for automation in retail using Computer Vision (CV) and IIoT in the era of industry 4.0.

Fig. 1
figure 1

Proposed smart automated shopping cart for automation in retail using CV & IIoT

Full size image

The system comprises of many functional blocks: (1) Pi Camera and OpenCV based object tracking for making the shopping cart track the movement of the registered shopper and follow automatically. (2) Raspberry Pi4 as the embedded edge computation platform. (3) Mechanical and drive sub-system comprises of a L298N motor drive, which controls two motors (left and right) using a novel Finite State Machine (FSM) based software sub-routine. (4) Battery and power supply sub-system. (5) MFRC522 module for RFID-based product sensing for automatic billing. (6) Touch screen display unit for providing user interface/experience (UI/UX). The objective of this proposed system is to get a complete automation in retail sector using fully automatic shopping cart. The cart would automatically follow the registered shopper and the shopper can buy products by simply dropping them on to the cart and the RFID module of the cart can sense the passive RFID tags (attached with all the products) hence having a list and database of all the products inside the cart. Finally at the time of check out the total products and the price can be computed and a bill can be generated automatically and processed for online payment using integrated payment gateway solutions. This proposed system would provide a complete end to end solution for automation in retail sector with the vision of industry 4.0 using computer vision, embedded system control of mechanical system using Industrial IoT technology. Figure 2 shows the block diagram of our proposed smart automated shopping cart system. The Pi Camera is interfaced with Raspberry Pi4 for acquisition of real-time video feed of the registered shopper. The computer vision algorithm implemented on the raspberry pi using OpenCV python-library to track the movement of the registered shopper. Subsequently the OpenCV would give a tracking output as movement of the target as five distinct states left movement, right movement, forward movement, backward movement or stop- no movement.

Fig. 2
figure 2

Block diagram of proposed smart automated shopping cart

Full size image

These five tracking states are feed to the FSM to create software sub-routine to control the motor drive L298N which subsequently controls the direction of Battery Operated (BO) DC motors. Two motors are equipped in the cart for direction and movement control in the mechanical sub-system. Figure 3 shows the automatic billing sub-system. The registered shopper would put products inside the cart. The cart is equipped with a RFID reader which would send Electronic Product code (EPC), Product info, unit price, quantity and sum price to the billing server using IEEE 802.11 (Wi-Fi) of Raspberry Pi. The data is going through a computation process and database handler to send all product purchase information to the billing server. The billing server is integrated with internet to provide the billing information and payment option to the shopper. Subsequently the shopper may pay the bills using secured online payment gateways to finish the shopping transaction.

Fig. 3
figure 3

RFID based automatic billing system for smart shopping cart

Full size image

3 System Implementation

The proposed system is implemented using Raspberry Pi 4 and other required axillary hardware and software (1) Pi Camera, (2) Power supply unit (DC-DC buck converter) and battery and voltage regulator and charger circuit. (3) Motor drive and Motor as mechanical control and drive sub-system. (4) MFRC522 RFID reader module, (5) Touch screen display unit for UI/UX. Figure 4 shows the flow chart of the software running on the raspberry pi to perform all the required actions. For functional modularity we have used two distinct python processes one for the object tracking and mechanical drive sub-system control and other one for RFID based auto-billing. The two processes are running through two independent terminal programs using raspberry pi OS program scheduler. First process starts with switching on the Pi camera and acquiring real-time video feed. First the target shopper image is registered and locked. We are using Channel Spatial Reliability Tracking (CSRT) correlation-based filer for tracking the target object. The output of OpenCV-based correlation filter is obtained as left, right, forward, backward or stop as tracked movement and the same is fed to our proposed novel FSM. FSM would send control signals to the L298N motor driver module. The motor driver module then controls the direction of two motors to maneuver the automated cart system. Second process is for a RFID-based purchased product tracking and auto billing. The MFRC522 RFID reader is interfaced with raspberry pi using I2C (Inter-integrated Circuit)/Serial Peripheral Interface (SPI) protocol for sending passive RFID tag information for computation and communication. The program would read the passive RFID tag and first stored the EPC of a particular tag (attached with the product). MFRC522 would next fetch the data sector and block wise from the tag and this process would provide all the basic information of the purchased product like product ID, name, unit price. This process would be repeated for the entire set of products put into the cart for purchasing. All these data would be computed and communicated to the billing server using IEEE 802.11 (Wi-Fi) data network using IoT. At the checkout all purchased product and their billing information would be displayed on the UI/UX using the display unit and also on the android app on shopper/s mobile device. Then the shopper would complete the online payment using relevant payment gateways and finish the shopping process.

Fig. 4
figure 4

Smart automated shopping cart system flowchart

Full size image

Figure 5 shows the complete system implementation comprising of all the sub-systems discussed above. We have using Firtzing software tool to model the prototype of electronic and embedded sub-system. The mechanical and motor drive sub-system is controlled using FSM driven by computer vision (OpenCV) algorithms implemented through software sub-routines and stand-alone processes (python programming).

Fig. 5
figure 5

System implementation

Full size image

We have studied different techniques of real-time tracking of object and by the study we have found CSRT technique is promising for our application and hence we have used the same in our proposed system.

4 Results and Discussion

In this section, we are presenting our results after applying the CSRT algorithm for real-time object tracking. Figure 6 shows the output of CSRT algorithm for real-time of the target object in this case our registered shopper.

Fig. 6
figure 6

Computer vision based CSRT algorithm for real-time target tracking implemented in our proposed shopper follower automatic mechanized shopping cart a Register Region of Interest (ROI), b Tracking front movement of target object c Tracking stop d Tracking left movement e Tracking back movement f Tracking right movement

Full size image

Based on the real-time tracking of the target object CSRT algorithm would provide five distinct states (1) forward (F), (2) stop (S), (3) backward (B), (4) left (L) and (5) right. Figure 7 shows the proposed novel FSM to send control signals to the L298N motor drive module. The state-machine is designed taking into consideration all the excitations possible in each of the states.

Fig. 7
figure 7

Finite State Machine (FSM) for smart automated shopping cart movement

Full size image

Figure 8a shows the FSM-based signaling timing diagram for L298N Motor drive control. S1(forward), S2(left), S3(right), S4(backward) & S5(stop). The signaling timing-diagram shows (IN-input) IN1 & IN2 pins of L298N motor driver for Motor 1 (Left Motor) control and IN3 & IN4 pins for Motor 2 (Right Motor) control. This system would give a complete navigation to the automation in mechanized drivability of the cart system by controlling the direction of motors hence ensuring a practical maneuverability based on real-time tacking of shopper (object) based on computer vision algorithm. Figure 8b shows the photograph as results of automatic billing screen of android app for the smart cart system. The screen shows the EPC of all purchased items and total price computer at the checkout using IoT and RFID technologies.

Fig. 8
figure 8

a FSM based signaling timing diagram for L298N Motor drive control using real-time tracking of target shopper for automatic mechanized shopper follower cart b Smart shopping cart automatic billing Android app: RFID based product tracking and bill payment feature

Full size image

5 Conclusion

In this paper, we have proposed a complete automation process of shopping cart by real-time object tracking using computer vision and RFID-based auto-billing feature. This proposed work would be instrumental in automation in retail section in the vision of industry 4.0 and industrial IoT and allied technologies. CSRT algorithm locks the target and track it real-time based on the Region of Interest (ROI), hence other object coming on the frame is not going to create hindrance in tracking. This feature would enable the cart to track and follow the registered shopper even in an overwhelming scenario of Indian shopping complex. Further enhancement in CV algorithm for better tracking mechanism in Indian scenario may be a research motivation for future work. This is a CV-based tracking and shopper following cart hence does not required any specific global location awareness rather the localization is achieved by the frame of reference of the registered shopper.