A General Internet of Thing System with Person Emotion Detection Function

Abstract

This paper proposes an internet of things system which can generates the map of people emotion around the campus. The system can identify someone’s emotion who stays in a school. Based on image retrieving, face detection and emotion recognition method, the emotion information can be gotten, store in the database and user can query the instance of emotion around the campus by the system. The query function is combined into an exist system which the authors proposed previously. The application of the system includes provide the professor to observe the students’ statement and adjust the teaching mode, activity result evaluation. Some system demonstration is also shown in the paper.

1 Introduction

Nowadays, it is convenient for us to get the image from the cameras. There are already many applications, such as smartphone, monitor and so on. It is common for not only teenagers but also elders to take a selfie with their phone and upload to social media to share their life. That is, it is easy to get images and analyze them. After analyzing images, it can use the technology of Internet of Thing (IoT) to upload the data to the Internet platform. In this paper, an IoT system which can generate the map of people emotion around the campus has been proposed.

Emotion Information is an important information in many field, such as shopping mall management, performance evaluation, student reaction information …and so on feedback. For instance, a department manager choices right method to increase the performance of promotion by their customer’s emotion distribution data. Based on these feedbacks, these occupations can clearly know their customers’ reaction. According to the information showing on the map, these occupations can adjust their plan. In order to analyze emotion from images and upload the information to the IoT platform, there are three steps: First of all, capturing the photo from the camera and detecting the human face. Second, build a database of human facial emotion and build the emotion chart. Last, upload the result to the platform and show on the correspondence map. With this map, it is possible to get the information of the emotions on this map at a glance. In this paper, it experiments in the campus. Count people’s emotion in the campus and uploading to the cloud. Finally, the emotion statistic can be shown on the map.

The rest of this paper is organized as follows: the related work and introduce the technology is shown in Sect. 2; the method is proposed in Sect. 3; Experimental results are shown in Sect. 4. Finally, the discussion is represented in Sect. 5.

2 Related Work

2.1 Emotion Detect

Facial Expression is directly way to express sentiment, it is common to identify emotion by detecting facial expression. There are some common ways to detect facial expression. One is capturing facial features, through those positions to determine expression [1]. The other one is Facial Action Coding System (FACS) which is one of the most widely system for the analysis of facial expression [2]. There are also some exist products that can detect emotion, such as Google, Microsoft and so on. These products can make it easy to plug sensors into the network, but it is not often combine with complex devices to network. In this paper, it proposes to combine the high-level information.

2.2 Internet of Things (IoT)

Internet of Things (IoT) is the network that realize the things interconnection through the communication or Internet. The basic idea of IoT is able to interact with other things through unique addressing schemes, such as RFID or wireless sensor [3]. To connect devices and network, the company of IBM and Eurotech introduce a message protocol-Message Queue Telemetry Transport (MQTT). MQTT is a connectivity protocol of machine-to-machine (M2M) and Internet of Things (IoT). It is able to provide routing for small, cheap, low power and low memory devices in vulnerable and low bandwidth networks [4].

3 System Architecture

This paper proposes to generate a map of people emotion in campus. To realize the emotion map, it is divided to two parts. One is detecting emotion, the other is the Internet platform of internet of thing (See Fig. 1). To approach this system, there are two methods, one is Cloud Computing (See Fig. 2(a)), the other is Edge Computing (See Fig. 2(b)). In Fig. 3, it proposes the way to determine the method, referring to the image size, Ram usage or network transmission quality. According to the different requirement, it can decide the different method to make it efficient.

Fig. 1.

The system architecture

Fig. 2.

System architecture (a) cloud computing solution, (b) edge Computing solution

Fig. 3.

The program flow

In this paper, it detects human face by some camera. Therefore, it gets the images from the video streaming. After getting the image, separating from the background to recognize the human face, and capturing the features on the face. According to the position of the features and expression database, mating the emotion result. In this paper, it designs an emotion chart which contains the scores and the corresponding emotions. With this chart, adding up the nearby score. Then, uploading the score to the IoT system. In this paper, it also proposes a platform to record these data and show on the map.

4 Experiment

There are some experiment results about this system. In Fig. 4, it processed the image to detect the human and capture the features on the face. After capturing the features, it can measure the feature position to estimate the emotion. According to the emotion database, mark the result on the image shown in Fig. 5, adding up the score of the emotion chart which the authors proposed previously, and average it to percent. The result of this image is 88%. Then, the IoT platform update the information and in Fig. 6, the information is shown on the map. In Fig. 7, it lists the Statistics chart of this experiment. In Fig. 8, this IoT platform combine with the traditional sensors and emotion detection, the value of the emotion detection is stand for the proportion of the joy in the picture. This is system was also experiment in class, the professor did the roll call by taking a photo, using this system to analyze the students’ statement (Fig. 9). The experiment picture of students' statement using online emotion recognition API in the proposed system.

Fig. 4.

An example of detecting human face and capturing the features

Fig. 5.

An example of emotion detecting result

Fig. 6.

An example which show the map with emotion information

Fig. 7.

The statistics of the experiment on the IoT platform

Fig. 8.

The meter page in the proposed IoT platform

Fig. 9.

The experiment picture of multi-face recognition

5 Discuss and Future Work

This paper proposes an internet of thing system which can generate the map of people emotion around the campus. The traditional IoT platform usually record about rainfall, humidity, temperature and so on. These data are recorded not frequently. In proposed system, it not only retrieves environment sensor, but also get emotion recognition result from video streaming with image based method. Because the character of video streaming, not every frame has he information of emotion or the recognition rate is not 100%. For the sake to reduce the cost of system, the bandwidth and the computational power are limited, there are two method to realize the system: Cloud Computing and Edge Computing to solve this problem. Cloud Computing is computing the data after uploading to the cloud. Edge Computing is transferring the information after computing the data. Using the method of Edge Computing can filter the image to decline the probability of unable to recognize. To make the system more efficiently, it can according to the camera level to decide the computing method. Though the microprocessor can transfer the data to the server, there are some limit, such as Arduino can transfer the data through the Internet, it cannot process too complex information. In Fig. 10 is the comparison of the exit internet of things platform. It is obviously that the platform that proposes in this paper plug the emotion sensor. In this paper, this system connects with Arduino and camera to realize the high-information Internet of Things platform, and it also can register sensor with mobile, as shown in Fig. 11. At present, no one has seen the combination of the emotion and the Internet of Things. In the future, it will combine with more high-level device to make a hybrid platform.

Fig. 10.

The Comparison (a) one of the IoT platform in Taiwan (b) one of the IoT platform abroad (c) this system

Fig. 11.

Smartphone also can be a sensor