Automated thermal fuse inspection using machine vision and artificial neural networks

Abstract

Machine vision is an excellent tool for inspecting a variety of items such as textiles, fruit, printed circuit boards, electrical components, labels, integrated circuits, machine tools, etc. This paper presents an intelligent system that incorporates machine vision with artificial intelligent networks to automatically inspect thermal fuses. An effective inspection flow is proposed to detect four commonly seen defects, including black-dot, small-head, bur, and flake during the production of thermal fuses. Backpropagation neural networks and learning vector quantization performance is compared in detecting the bur defect because of its illegibility. Different numbers of defective samples were screened out from a production line in a case study company and used to demonstrate the efficacy of the proposed system. Currently, the proposed inspection system is operating at the case study company, replacing four to six human inspectors. The system not only ensures the quality of the thermal fuses produced, but also reduced the cost of manual visual inspection.

Introduction

A thermal fuse (TF), also known as a thermal cutoff (TC), is an important component in both electrical and electronic devices. Thermal fuses prevent circuits from overheating or becoming overloaded. While fuses generally allow the passage of current, they can short circuit to cut power to appliances as a safety feature. Figure 1 shows two types of fuses. Electronic fuses (Fig. 1a) perform a cutoff when current in the circuit exceeds a pre-specified value, while thermal cutoffs prevent overheating. They are both used in many electronic devices, such as hair dryers, rice cookers, coffee makers, etc. Thermal fuses are “one-shot”, non-resettable, temperature-sensitive devices that provide a failsafe temperature protection in safety-critical circuits. They are mainly defined by their temperature setting but also by the current rating, which is the maximum continuous current that can be applied before it breaks. Currently, experienced human inspectors are used for quality control of thermal fuses during manufacturing. These inspectors are subject to fatigue, distraction and subjective decision-making regarding quality variability. Defects are not only tiny, but can be difficult to detect manually due to the low intensity contrast, thus automatic detection of metallic surface defects could significantly improve quality control.

Fig. 1

Two types of fuses. a Electronic fuses. b Thermal fuses

Over the past three decades, machine vision has been widely used in manufacturing to inspect the quality of printed circuit boards (Lin 2007; Torres et al. 2002; Benedek et al. 2013) electric components (Lahajnar et al. 2002; Liang et al. 2012), chip alignment, wire bonding for Integrated Circuits (IC) (Wang et al. 2002; Su et al. 2013), machine tools (Tien et al. 2004; Zhang et al. 2013a, b), metal parts (Zheng et al. 2002; Wu and Hou 2003; Steiner and Katz 2007; Sun et al. 2010; Scholz-Reiter et al. 2012; Shen et al. 2012; Ghorai et al. 2013), LCD panels (Gan and Zhao 2013) and semiconductor wafers (Chang et al. 2011; Sun et al. 2011; Li and Tsai 2012). Various techniques used for these inspection applications have been reviewed by many studies (Chin and Harlow 1982; Chin 1988; Newman and Jain 1995; Thomas et al. 1995; Malamas et al. 2003). However, inspecting thermal fuses for surface defects is both one of the most common and most difficult problems in the area of machine vision. Yamashina and Okumura (1996) developed an automated measurement system for detection of drilling tool malfunctions such as wear and chipping using two CCD cameras. Tien et al. (2004) developed an automated visual inspection system that effectively inspected microdrill blades. Li and Tso (2006) developed an X-ray-based inspection systems for identification and evaluation of internal defects in casting by using 2D wavelet transform methods. Lin (2007) proposed a wavelet characteristic-based approach for the automated visual inspection of ripple defects in the surface barrier layer (SBL) chips of ceramic capacitors. Steiner and Katz (2007) used machine vision to inspect porous flaws on machined surfaces. Torres et al. (2002) presented several automatic object searching techniques to locate multiple components on a PCB in a grey-scale captured image through accelerated species based particle swarm optimization (ASPSO) and genetic algorithm (GA) approaches. Sun et al. (2010) applied machine vision to inspect surface defects on electrical contacts. Sun et al. (2011) developed machine a vision-based post-slice wafer inspection system, which successfully detects hole, chip, and ellipse defects, and determines the feasibility of regrinding defective wafers. Chang et al. (2011) developed an automatic inspection system using artificial neural networks to recognize defective LED dies. Chen et al. (2012) presented an automatic damage detection system for ceramic machined surfaces using image processing techniques, pattern recognition, and machine vision. Liang et al. (2012) proposed a two-step segmentation method to detect defects with the Gabor filter, using the unsupervised and fast segmentation features of the Fuzzy C-Means algorithm to detect most OLED defects. Su et al. (2013) proposed an ultrasonic inspection for flip chip solder joints. The image of the flip chip was acquired from a scanning acoustic microscope and segmented; a backpropagation network was then used to classify and recognized the geometric features extracted from the image. Scholz-Reiter et al. 2012 proposed an inline surface inspection technique based on texture analysis and statistical image processing methods for cold-form micro-parts. Shen et al. 2012 developed a machine vision system to detect various types of defects on bearing covers, such as deformations, rust, and scratches. Li and Tsai (2012) proposed wavelet-based method that uses the wavelet coefficients in individual decomposition levels as features and the difference of the coefficient values between two consecutive resolution levels as the weights to detect local defects in a crystal grain background, and generates a discriminant measure to identify different defects in multicrystalline solar wafers. Gan and Zhao (2013) proposed a machine vision inspection method using a modified local binary fitting model to detect brightness levels defects in LCD panels, along with size and shape inconsistencies. Ghorai et al. (2013) developed an automated visual inspection system for an integrated steel plant to capture surface images using different wavelet feature sets at different decomposition levels derived from 32 $\times$ 32 contiguous pixel blocks of steel surface images. Benedek et al. (2013) introduced an automated Bayesian visual inspection framework for printed circuit board assemblies which can simultaneously deal with various shaped circuit elements (CEs) on multiple scales. Tolba (2011) introduced a novel hybrid approach for defect detection and localization in homogeneous flat surface products based on statistical decision theory, multi-scale and multi-directional analysis and a neural network implementation of the optimal Bayesian classifier. Xie et al. (2014) proposed a localized defect image model (LDIM) for defect inspection combining features of manual inspection and a local defect merit function to quantify the cost of defective pixels. However, the issue of thermal fuse surface defect detection has not been previously discussed in the literature.

This paper presents an intelligent automated visual inspection system that effectively detects defective thermal fuses using artificial neural networks and machine vision. The remainder of this paper is organized as follows. Second section describes thermal fuse defect types. Third section presents the proposed machine vision-based inspection system. Fourth section demonstrates the experimental validation process, and conclusions are drawn in the last section.

Thermal fuse defects

In general, two types of thermal fuses, axial and radial, are typically used in various forms and sizes, depending on working requirements (Prijic et al. 2008). One commonly-used type of thermal cutoff is based on inserting a conductive bridge into the winding or heater circuit. This bridge consists of a metal strip joined to the leads by soldering sheets made from a thermally sensitive alloy. This study focuses on the work-in-process of thermal fuses produced by the case study company. Such fuses are composed of a case and a lead wire (Fig. 2), and defects mainly occur while inserting the lead wire into the case.

Fig. 2

WIP of thermal fuses

Figure 3 shows the top-view image of a defect-free thermal fuse, which can be separated into three parts: the outer ring (case wall), the white ring (case base), and the inner circle (lead head). Figure 4a–d depicts common defects, including small-head, burs, black dots, and flakes, respectively. The causes and description of these concerned defects are explained below:

• (1) Bur (Fig. 4a): this defect is a deckle edge, which occurs on the wall of the fuse case. It is mainly caused by the use of abnormal materials for the lead or worn out punch heads. As in the image, the bur defect usually appears in the outer-ring area as a small grey dot. Even an experienced inspector may have difficulty identifying this type of defect.

  Fig. 3

  

  Top view of fuse. a Fuse image. b Fuse image separation

  Fig. 4

  

  Defects of thermal fuse. a Bur, b black-dot, c small-head, d flake

• (2) Black dot (Fig. 4b): this defect is mainly caused by the intrusion of dirt or dust during the assembly of the case and lead, appearing here as a black dot around the inner circle area.

• (3) Small-head (Fig. 4c): this defect occurs at the punching stage of the manufacturing process when insufficient force is used to punch the lead into the wall, resulting in a smaller head area than normal. The standard radius of the fuse head is between 2.11 and 2.76 mm, depending on the fuse specification.

• (4) Flake (Fig. 4d): this type of defect is caused by the chip-off of the case wall during the electroplating process. A flake usually is shown as a block in the white-ring area, and is much larger than a bur.

These defects are manually screened by experienced human inspectors, but human error can result in many defective thermal fuses passing inspection, negatively impacting corporate revenues and reputation. Therefore, this study aims to develop a machine vision-based inspection system to automatically identify defects (including small-head, bur, flake, and black-dot) in thermal fuses, thus increasing screening accuracy and reducing manufacturing costs.

Proposed machine vision-based inspection system

A pilot study showed that small-head and black-dot defects can be detected by a similar sequence of inspection flow, while the bur and flake defects require different inspection algorithms due to their distinctive features. The proposed inspection flow is shown in Fig. 5 and the details of the inspection flow are described in the following subsections.

Fig. 5

Inspection flow for thermal fuses

Image acquisition and segmentation

A thermal fuse is first inserted into the system and a digital image is captured. A threshold technique is used to segment the fuse from its background based on an image histogram representing the distribution of fuse images. Figure 6a, b shows an example that applies the Otsu threshold method (1979) to separate the fuse object and background. After thresholding, the fuse object can be identified with some tolerable noise.

Fig. 6

Image pre-process of thermal fuse. a Original Image. b Histogram analysis

Fuse locating and separation

The fuse is differentiated from noise using blob analysis which assigns pixels in an isolated blob to a single index, thus distinguishing different objects. The features of each blob (e.g., area, gravity, location, object width and object height) can be calculated to screen out noise. This study used fast connected component labeling (FCCL) to label the fuse object in the image (see He et al. 2009 for a description of FCCL). After locating the fuse, the double-thresholding technique is used to separate the fuse object into three parts: the outer ring, white ring, and inner circle.

Small-head and black-dot inspection

The characteristics of small-head and black-dot defects are similar in that they can be detected using the same algorithm. After locating and separating the inspected fuse, the white ring and inner circle can be segmented and the radius of the inner circle can be easily measured through a circle fitting algorithm as shown in Fig. 7. Meanwhile, the black dot, which always occurs at the margin between inner circle and white ring, can be detected by thresholding the white ring area with a pre-defined intensity value. An example of black dot detection is illustrated in Fig. 8.

Fig. 7

Small-head defect detection. a Original small-head image. b Inner circle locating. c Blob analysis. d Circle fitting

Fig. 8

Black-dot defect detection. a Black-dot image. b Locating and separation. c Black-dot detection

Flake detection

The flake is a severe defect which can cause a thermal fuse to malfunction. The flake defects appear in the white ring area as shown in Fig. 9a. The inspection process first separates the white ring area from the base area through image thresholding with a proper threshold value. A boundary following procedure can be then used to trace the contour of the white ring area. If the length of the contour is greater than a pre-specified value $L$, the fuse is determined to have a flake defect. An example is shown in Fig. 9 a–c to demonstrate the effectiveness of the process.

Fig. 9

Flake detection. a Flake defect image. b Thresholding. c Boundary following

Bur detection

A bur is a tiny defect that appears as a small indistinct dot in the outer ring area. This defect can barely be seen by a human inspector using a magnifier. Several segmentation methods, such as global/local thresholding, and morphological operations have failed to consistently identify this defect. Therefore, two artificial neural networks (ANNs), backpropagation neural networks (BPN) (Zhang et al. 2013a, b) and learning Vector Quantization Networks (LVQ), are adopted to solve this problem and compared (Basheer and Hajmeer 2000).

• (a) Backpropagation neural networks (BPN): In general, a BPN includes: (1) an input layer with nodes representing input variables to the problem, (2) an output layer with nodes representing the dependent variables, and (3) one or more hidden layers containing nodes to help capture the nonlinearity in the data. Its architecture is shown in Fig. 10. The neurons between layers can be fully or partially interconnected between layers with weight ($w_{ij}$). The data are first fed forward from the input layer, through hidden layer, to output layer without feedback. Then, based on the feedforward error-backpropagation learning algorithm, the BPN searches the error surface using the theory of gradient descent for point(s) with minimum error.

  Fig. 10

  

  Architecture of BPN

• (b) Learning Vector Quantization (LVQ) is also a supervised learning neural network. This network consists of three layers: the input layer, which receives the input signals; the hidden layer, which is a competitive layer; and the output layer, which is a linear mapping layer. As a supervised method, LVQ uses known target output classifications for each input feature. LVQ adopts a winner-takes-all Hebbian learning-based approach, memorizing the input features into one node called the winner. The winner node represents the most similar characteristics of the input vectors. To classify the data correctly, the weights of the connections to this winner node are updated in each epoch. Figure 11 shows the architecture of the LVQ neural network (see Kumar 2004 for a detailed description of LVQ networks.

  Fig. 11

  

  LVQ neural network

Network learning for both BPN and LVQ requires the extraction of representative input features of defects. After consulting with the case study firm’s product engineer, we found the appearance of burs is highly correlated to its background (outer ring). Therefore, we first segmented possible burs in the outer ring area by a thresholding operation. If the intensity of any blob exceeds a pre-defined threshold, we considered it a bur candidate and calculate its features in its 15-degree wedge area. Figure 12 expresses the bur wedge area in an outer ring. Four parameters for the given wedge (the mean, variance, range, and entropy) are calculated as follows:

$$\begin{aligned}&\hbox {Mean}:\;\mu =\sum _{i=1}^n {z_i p(z_i )}\end{aligned}$$

(1)

$$\begin{aligned}&\hbox {Variance}:\;\sigma ^{2}=\frac{1}{n-1}\sum _{i=1}^n {(z_i-\bar{{X}})^{2}}\end{aligned}$$

(2)

$$\begin{aligned}&\hbox {Range}:\;R=Max(z_i )-Min(z_i)\end{aligned}$$

(3)

$$\begin{aligned}&\hbox {Entropy}:\;e=-\sum _{i=0}^n {p(z_i )} \log _2 p(z_i) \end{aligned}$$

(4)

where $z_{i}$ is the pixel intensity in the wedge area, $p(z_{i})$ is the probability of $z_{i}$ occurring in the wedge area, and $n$ is the number of pixels in the area. Among these four parameters, the mean ($\upmu$) represents the average intensity of a given wedge; the variance ($\upsigma ^{2}$) represents intensity dispersion; the range ($R$) depicts the extreme (maximum and minimum) conditions of intensity; and the entropy ($e$) expresses the disorder or randomness in the area. The intensity of a bur defect is slightly brighter than its neighboring pixels, so a wedge with a bur defect will have a higher average intensity. The variance ($\upsigma ^{2}$) and the range ($R$) of the sampled wedge may also be greater than in a normal wedge area, while entropy (e) may vary for different defective cases. These four features are then fed into the both BPN and LVQ networks for learning.

Fig. 12

Bur wedges at the outer ring

Implementation

The proposed inspection process was implemented in C++ on a PC with a Pentium IV processor (2.4 GHz) and 4 Gigabytes Memory running Microsoft Windows XP. Different numbers of samples were screened out by experienced human inspectors in advance for system validation.

System configuration

Figure 13 illustrates the configuration of the proposed automated visual inspection system. It consists of a charged-couple device (CCD) camera, a frame grabber, a PC, a sorter, and transporting (feeder), screening and sensing devices. When a thermal fuse was first sorted and fed into the system, the proximity sensor is triggered and a transistor–transistor logic (TTL) signal is sent to activate the CCD camera to acquire a digital image. The inspection algorithm was then initiated to detect defects in the fuses. If any defect was found, a signal is sent to the controller to screen out the defective fuse by a designed mechanism.

Fig. 13

System configuration

Inspection of small-head defects

The system was first calibrated by comparing 30 measurements with the result of the Nikon MM-800 optical measuring machine. The average radii measurements from the proposed system and Nikon MM-800 were 2.0861 and 2.066 mm, respectively, with standard deviations 0.397 and 0.380 mm. The radii measured by the proposed system were slightly larger than those measured by the Nikon MM-800. Therefore, the deviance was corrected by adding an off-set to the measured value. Another 30 small-head defective samples were then screened out by experienced QC inspectors, and then fed into the designed inspection system. Figure 14 shows five example small-head fuses with their inspection results. The specification was $2.435 \pm 0.125$ mm, as provided by the manufacturer. These 30 samples were measured both by the proposed inspection and the Nikon MM-800 for validation. The measurement results are presented in Table 1. Several pieces were screened out by incorrect human judgment, including samples #11, #12, #13, #15, #24, #25, #26, #27, #30.

Fig. 14

Five small-head examples and their inspection results

Table 1 Small-head measurement results

Flake detection

Thirty defective flake samples and 30 good samples were first collected for validation. Through trial-and-error, the threshold value to segment the white ring area was set at 200. After analyzing the normal and defective contours of white ring areas, we found the contours of a normal fuse were around 1,000 to 1,300 pixels, so the cut-off point was set at 1,600. Under this setup, another 30 samples were collected and tested. The proposed inspection process successfully detected all flake samples without false-positives for the good samples. Figure 15 shows five defective samples to demonstrate the inspection result. Figure 15a shows five defective flake samples, and Fig. 15b shows the result of boundary following, with contour lengths of 2004, 1989, 2008, 1904, and 2014 respectively. Figure 15c shows their inspection results, with the flakes marked in red.

Fig. 15

Five flake examples and their inspection results. a Original five flake images. b Flake image after boundary following. c Flake image with detection

Bur detection

For bur detection, two sets of experiments were conducted for validation and comparison. Off-line data collection and network training were done for both BPN and LVQ. A total of 640 samples of bur defective wedge images (50 % good and 50 % defective) were collected. Among these, 384 samples (60 %) were used for training and 128 samples (20 %) were used for the testing, while the remaining 128 samples were used for final validation.

BPN network experiment

When using a BPN for pattern recognition, parameters including learning rate, momentum, types of transfer function, and structure of networks (number of hidden layers and number of nodes in the hidden layers) must be determined in advance. In this study, the nonlinear sigmoid or tansig transfer function was used in the hidden and output layers. The number of input nodes was four, determined by the dimension of features and the number of output nodes was two, either (1, 0) or (0, 1), where 1 represents the occurrence of a bur defect. The number of hidden layers was set to either one or two layers for simplicity. The number of nodes in the hidden layers was first set as

$$\begin{aligned} \hbox {No. of hidden nodes} = \frac{\hbox {Input nodes} + \hbox {output nodes}}{2}. \end{aligned}$$

(5)

Neurons were then added or deleted based on learning performance. However, the learning rate, momentum and final architecture of the networks were determined by a simple experiment of design. During the experiment, the network was stopped at 1,000 epochs or when SSE reached 0.05. Table 2 demonstrates part of experimental training results for best detection results with accuracy as high as 98.43 %, showing the BPN with structure (4-20-1), learning rate equal to 0.01, and stopping epochs set as 800.

Table 2 BP setting and results

LVQ network experiment

The same training and testing patterns were also used to train the LVQ networks. The key to the learning process was to determine the learning rate and the number of hidden neurons, which is usually much larger than the number used in BPN. Table 3 shows different setups (i.e., different numbers of hidden neurons and learning rates) of LVQ networks, and their corresponding results. During the learning process, the training epoch was set at two levels: 200 or 300. The LVQ network with the structure 4-160-2 and a learning rate of 0.1 obtained the best detecting performance, with recognition rates for the training and tested samples of 93.33 and 92.22 %, respectively. The learning process of best LVQ is shown in Fig. 16.

Fig. 16

Learning process of LVQ with 4-160-2 structure

Table 3 LVQ setting and results

Finally, the last 128 samples were used to validate these two networks. A comparative result between BPN and LVQ is illustrated in Table 4, which shows the BPN has the better classification rate (98.43 %) though it requires a little more computation time.

Table 4 Comparative results between BPN and LVQ

Discussion and conclusions

The proposed system was physically implemented and is currently running in the case study company as shown in Fig. 17a. The graphic user interface is shown in Fig. 17b. All the settings were aimed at eliminating the type II error in electrical appliances for safety considerations. For the small-head inspection, the proposed method replaced unscientific human guessing with a precise measurement. Following simple calibration, the small-head measurement reached an accuracy equal to that of an industrial optical measuring machine. Whereas a human inspector would subjectively guess the size of the lead head, the proposed algorithm provided an accurate objective measurement. The system detected flake defect with 100 % accuracy. Two supervised ANNs (BPN and LVQ) were trained to classify the bur defect. Though slightly slower, BPN provided better detection performance than LVQ, and was thus selected for bur defect inspection in the deployed system.

Fig. 17

The implemented system. a Inspection system in case study company. b System GUI

This study presents a machine vision-based inspection system using artificial neural networks to detect four major defects (small-head, bur, flake, and black-dot) in the manufacture of thermal fuses. An implemented inspection flow was shown to reliably identify these defects. The study makes the following major contributions: (1) four features were successfully extracted and applied to two artificial neural networks for the machine vision-based inspection system; (2) the proposed system not only reduces costs associated with human inspectors but also increases inspection reliability and speed.