1 Introduction

In Japan, vocational schools, technical high schools and junior high schools offer classes for circuit implementation. For circuit implementation, the students use soldering iron. However, many students are using the soldering iron for the first time, which can cause burns, expose other students to the soldering iron, or cause circuits to fail. Teachers monitor students to prevent accidents and mistakes, but it is difficult to completely prevent the aforementioned mistakes. One of the reasons why students make soldering mistakes is because they are not holding the soldering iron correctly. If the soldering iron is not held correctly, the soldering iron will not be stable, which will increase the likelihood of mistakes. In this paper, we propose a ambient intelligence  [1,2,3] based support system for training to hold the soldering iron.

Fig. 1.
figure 1

Structure of proposed system.

Full size image
Fig. 2.
figure 2

Flowchart of the proposed system.

Full size image

2 Proposed System

In this section, we present the proposed system. In Fig. 1 and Fig. 2 are shown the structure of proposed system and flow chart.

2.1 Object Recognition

From the streaming video acquired from the USB Video Class (UVC) camera, the coordinates of the key points of the hand and the coordinates of the tip in soldering iron, which is the soldering iron hazard area are obtained.

For detection of the tip in soldering iron, we train the cascade classifier using the image of tip in soldering iron. The OpenCV cascade classifier was used to detect the tip in soldering iron. The xml classifier file for the cascade classifier is created with opencv_traincascade. The Local Binary Pattern (LBP)  [5] based cascade classifier is used in proposed system. The positive image set including the tip in soldering iron and the negative image set including everything we did not want to detect except for the tip in soldering iron were actually prepared by taking pictures with a UVC camera. We developed the highest weight value extracting algorithm for more than one object is detected in cascade classifier, and the proposed system use this algorithm.

To determine the dangerous grip of the soldering iron, we consider the hybrid method which the tip in soldering iron is detected by the cascade classifier and the finger is recognized by the Open Source Computer Vision Library (OpenCV)  [4] Deep Neural Network (DNN) module. For recognition of finger, the DNN recognize the key points of the finger joints using obtained. The finger joints are touching each other finger joints, the key points of the finger joints of the touching each other are paired, otherwise calculate the difference between coordination of key points in fingertip and finger root joints pair the key points.

2.2 Automation Surprise

For automation supplies with voice output near the soldering iron user, the computer for computational intelligence and the computer for voice output are connected wirelessly. The Secure Shell (SSH) connection was based on the Paramiko and the sound in automation surprise was generated using Open JTalk  [6]. The computer for computational intelligence obtains an image from the UVC camera and recognizes the object to determine if the finger is close to the tip in soldering iron, if the soldering iron is not dangerous to hold, or if it is not dangerous to hold the iron, or if it is neither. For computer for voice output, we use Google AIY Voice Kit V2. When it is decision that the fingers are close to the tip in soldering iron or the way to hold the soldering iron is dangerous, the voice output for warnings using Open JTalk is executed according to the decision.

If a student has a dangerous grip on the soldering iron, an audio output based on automation surprise will be provided. When the coordinates of the key points of the index finger and the coordinates of the tip in soldering iron are at a certain distance, the proposed system emits the voice as automation surprise. We describes the examples of the warning messages in automation surprise. When the distance between the finger and the tip in soldering iron and the coordinate of the is closer than a certain value, the kit emits a voice message “The finger is close to the tip in soldering iron and dangerous”, and when the distance between the finger and the tip in soldering iron is farther than a certain value, the kit emits a voice message “The way you hold the tip in soldering iron is dangerous”.

Fig. 3.
figure 3

Experimental environment.

Full size image

3 Experimental Results

In this section, we describe the experimental results. A picture of the mounting environment is shown in Fig. 3. Table 1 shows the configuration of the experimental environment.

Table 1. Composition of the experimental environment.
Full size table
Fig. 4.
figure 4

Examples of detection types of tip in soldering iron.

Full size image

3.1 Experimental Settings

As shown in the Fig. 4, a rectangle is generated to identify the object, and if the tip in soldering iron is within the rectangle, it is decision to be the correct answer (Fig. 4(a)). If a rectangle is generated as shown in the Fig. 4(b), but the tip in soldering iron is not included in the rectangle or multiple parts are recognized as shown in the Fig. 4(c), it is decision to be a false detection. If the rectangle identifying the object is not generated as shown in the Fig. 4(d), it is decision to be undetected.

The opencv_traincascade parameters are shown in the Table 2. We describes the opencv_traincascade parameters in following: The featureType is the type of features. The numStages is number of cascade stages to be trained. The minHitRate is minimal desired hit rate for each stage of the classifier. The maxFalseAlarmRate is maximal desired false alarm rate for each stage of the classifier. The sampleWidth and sampleHeight is height and width of the sample created by the training samples. All parameters are the same except for the number of images.

3.2 Experimental Scenario 1: Detection of Tip in Soldering Iron Using Cascade Classifier

As shown in the Fig. 5, the image with the tip in soldering iron is the positive image (Fig. 5(a)) and the image that contains everything except the tip in soldering iron is a negative image (Fig. 5(b)). The positive images and negative images in a ratio of 1:2, and positive and negative images were combined into a dataset of 1500 [images], 3000 [images], 4500 [images] and 6000 [images]. An evaluation experiment was conducted on 100 [images] containing the tip in soldering iron. In Fig. 6, we show the experimental results not using the highest weight value extracting algorithm and using the highest weight value extracting algorithm. Figure 6(a) shows the experimental result of cascading the tip in soldering iron. Figure 6(b) shows the experimental result of using the highest weight extracting algorithm which determines one object when multiple objects are detected. The experimental results showed that the algorithm to determine one object improved the detection rate by 18 [\(\%\)] with up to 1500 samples. In both experimental results, the undetected rate increased as the number of samples increased, suggesting that the present experiment is overtrained. Using the example with 1500 [images], which had the highest detection rate, we tested whether the finger coordinates and the coordinates of the center of gravity of the tip in soldering iron are considered dangerous if they fall below a certain value.

Fig. 5.
figure 5

Examples of positive and negative images.

Full size image
Table 2. Parameters of the opencv_traincascade.
Full size table
Fig. 6.
figure 6

Experimental results of the cascade classifier.

Full size image
Fig. 7.
figure 7

Example of estimation of the state between hand and tip in soldering iro.

Full size image

3.3 Experimental Scenario 2: Estimation of the State Between Finger and Tip in Soldering Iron Using Cascade Classifier and DNN Hybrid Method

We present the experimental results of estimation of the state between hand and tip in soldering iron using cascade classifier and DNN hybrid method. The cascade classifier and DNN hybrid method get the a finger near the tip in soldering based on the coordinates of the key point of the finger and the center of gravity of the tip in soldering iron as dangerous. For experimental results, 100 [images] in which the index finger is close to the tip in soldering iron is prepared as shown in the Fig. 7. The experimental result show that 76 [\(\%\)] of the images is found to be dangerous.

4 Conclusion

In this paper, we proposed a support system for soldering iron training. In cascade classifier, we find that there is only one object to be detected in the image and developed the highest weight value extracting algorithm for more than one object is detected in cascade classifier. Also, we find that object recognition is possible even when training with few images by using an extracting algorithm that determines the highest weight value of the detected objects when multiple objects are detected. The experimental results show that the proposed system is capable of communicating the inherent dangers of the soldering iron to the user by voice. In the future, we will conduct in other scenarios, such as detection of other incorrectly held objects and support systems.