1 Introduction

With the increasing demand for flawless quality in the PCB assembly process, electronics manufacturers constantly request higher levels of precision in PCB inspection. In general, a good PCB inspection system has to satisfy the requirements of high inspection speed, high detection accuracy, and low false-alarm rate. Commonly used PCB defect detection procedures can be divided into two categories: (a) electrical/contact methods and (b) non-electrical/non-contact methods. The electrical methods can detect some defects, but the inspection capabilities are limited. In addition, electrical methods may cause mechanical damages during the probe contact. For the non-electrical/non-contact PCB inspection methods, numerous techniques have been proposed for Automated Optical Inspection (AOI). They can be categorized into three main approaches: reference comparison, non-reference verification, and hybrid approach [1]. The referential approach compares the tested PCB image with a reference PCB image. The main limitations of this approach are the need for very precise alignment and the vulnerability to differences in illumination. The second type of approach (non-reference verification) tests the design rules of the PCB features to determine whether each feature falls within the required range. However, it is not easy for users to design the rules as constraints on image features in the rule-based verification, and it lacks flexibility to distorted features. The hybrid approach concurrently combines the advantages of the aforementioned approaches. As hardware gets cheaper and faster, it is becoming more popular to use hybrid methods that benefit from both approaches.

Numerous PCB inspection algorithms have been proposed and can be found in the literature [2–4]. The earliest developed PCB inspection algorithm is known as template-matching technique as proposed by Akiyarnai et al. [5]. In the following decade, various PCB inspection methods were developed. For example, Tominaga [6] used a dichromatic reflection model to describe object surfaces followed by a hyper-sphere method for defective object identification. Spence [7] converted the currents moving between the devices on PCBs to a magnetic field image for locating the possible faults, and interpreted the magnetic field patterns with artificial neural networks for fault identification. Later, Hodges et al. [8] diagnosed solder pad arrays on PCBs based on template matching, where histogram features were used. In [9], Scaman et al. utilized wide-angled blue lights to differentiate the metal patterns from polyimide background for PCB inspection. Wu et al. [10] proposed the use of an elimination–subtraction procedure to locate and analyze PCB defects by subtracting the template image from inspection image. Loh et al. [11] inspected the surface mount technology (SMT) components on PCBs by using structured-lighting. There, a slant map was extracted as the shape information of the solder joint surface, followed by a rule-based defect classification procedure. In addition, Kim et al. [12] built a PCB inspection system based on a technique that matches blocks between reference and inspected images. Tominaga et al. [13] also proposed a PCB inspection system that first sampled the surface reflectance with a liquid-crystal tunable filter imaging system, and a pixel-wise classification was next performed using k-means algorithm. In the system developed by Ibrahim et al. [14], the image difference operation was applied in the HAAR-wavelet domain. An XOR logic operation was next performed to generate the final result for inspection. Garcia et al. [15] explored the multivariate stepwise discriminant methods (MSDM) to select the best features and applied rules for inspection. Acciani et al. [16] exploited neural-networks for diagnosing the solder joint defects on PCBs. Chang et al. [17] adopted the hybrid approach to construct a two-phase PCB inspection system. In the first phase, a set of defect images of several existing basic patterns were stored to form a concept space, where several empirical rules were used for defect classification. In the second phase, a new pattern was evolutionally grabbed by calculating the relative position of several similar cases in the concept space.

Although inspecting bare PCBs is already a successful industrial application, its full potential has not yet been exploited. One reason for the absence of AOI systems with continuous improvements is the lack of flexibility to adapt its algorithms for inspecting new defects. The problem of reconfiguration and reprogramming, combined with the rapid introduction and retirement of electronic products, has deterred equipment manufacturers from investing in the development of AOI systems. Besides, as can be seen, most existing inspection systems are rule-based and designed from human observations. From a statistical point of view, they are not robust enough and unable to generalize well for the unseen defects. Another problem is the requirement to identify both functional and cosmetic defects from their visual appearance [1]. Therefore, for PCB inspection, it is essential to develop methodologies that allow rapid adaptation and generalize well for different variations of defects like human beings do. This research introduces the development of a novel AOI system for PCBs, which has several advantages and contributions. First, instead of using the rules to classify the detected defects heuristically, a learning-based solution is developed through solving an optimization problem where the generalization ability is theoretically maximized. Second, the proposed system is flexible, allowing for potential function extension. By adding defective samples into the database, the inspection system has the capability to classify new defects. It significantly cuts down cost for the eventual upgrading of the inspection system. Third, for cosmetic defects with irregular appearance, such as weave exposure, blowhole, scratches, solder on trace, and problems from solder mask, such as air inclusion, a probabilistic solution is developed to identify the defective regions and measure the defectiveness quantitatively. The key requirements of the reliability and flexibility are therefore achieved.

This paper is organized as follows: In Section 2, some theories related to this research are briefly reviewed. Section 3 describes the components of the proposed inspection system with application to PCBs. Section 4 presents some experimental settings and results. Finally, Section 5 concludes this paper.

2 Related Theories

2.1 Hausdorff Distance

Hausdorff distance [18] is useful in determining the degree of similarity between the edge maps of two templates. It provides a means of computing the distance between two point sets. Let A and B be two compact subsets of points in a metric space M, the definition of Hausdorff distance between A and B is given by

$$ {d_H}(A,B): = \max \left\{ {h\left( {A,B} \right),h\left( {B,A} \right)} \right\}, $$
(1)
$$ h\left( {A,B} \right): = \mathop{{\max }}\limits_{{a \in A}} \mathop{{\min }}\limits_{{b \in B}} d\left( {a,b} \right) $$
(2)

where d(a,b) denotes the distance between two points a and b in M, usually the Euclidean distance. In practice, to match a template image and a target image, it is necessary to first apply an edge detector to derive the corresponding point sets of the images. In Section 3.1, Hausdorff distance computation is exploited as the technique to align an inspection image I with a reference image R by finding the minimal distance value with respect to different geometric transformations.

2.2 Support Vector Machine

Support Vector Machine (SVM) [19] is a learning system for efficiently training linear learning machines from optimization theory, while the formulation leads to better generalization with the maximum margin criterion [20]. Let \( S = \left\{ {{{\mathbf{x}}_l},{y_l}} \right\}_{{l = 1}}^m{ } \) be a set of training examples, where \( {{\mathbf{x}}_l} \in {\mathbb{R}^n} \) is the l th example associated with the class label \( {y_l} \in \{ 1, - 1\} \). The aim is to define a separating hyperplane for which the margin between the positive and negative examples is maximized. This amounts to finding the normal w and the bias b so the distance between the closest points to the hyperplane, i.e. 1/||w||, is maximized:

$$ \begin{array}{*{20}{c}} {\mathop{{\min }}\limits_{{{\mathbf{w}},\xi }} \frac{1}{2}{{\left\| {\mathbf{w}} \right\|}^2} + C\sum\nolimits_{{l = 1}}^m {{\xi_l}} } \\{s.t.\,{y_l}\left( {{{\mathbf{w}}^T}{{\mathbf{x}}_l} + b} \right) \geqslant 1 - {\xi_l}} \\{{\xi_l} \geqslant 0,\,l = 1,2,...,m} \\\end{array} . $$
(3)

Here, \( {\xi_l} \) is the positive slack variable introduced to handle the non-separable cases [19], and C is chosen to control the tradeoff between errors and functional complexity. Since ||w||2 is convex, minimization can be achieved by differentiating the Lagrangian of (3), and it leads to an equivalent quadratic programming (QP) problem:

$$ \begin{array}{*{20}{c}} {\mathop{{\min }}\limits_{\alpha } {\alpha^T}Q\alpha - {{\mathbf{e}}^T}\alpha } \hfill \\{s.t.\,0 \leqslant {\alpha_l} \leqslant C,\,l = 1,2,...,m} \hfill \\\end{array} $$
(4)

where Q i,j = y i y j K(x i ,x j ) is an m x m matrix, and K is a symmetric kernel function [19, 21]. Some frequently used kernels include polynomial kernel K(x i ,x j ):=(x i T x j +1)d and RBF kernel K(x i ,x j ):=exp(-||x i ,x j ||2/2σ2). Assume the optimal solution α*=(α 1,α 2,…, α m ) is found by solving Eq. 4 with a standard QP solver [22]. For a new test sample x test , it is classified by \( f({{\mathbf{x}}_{{test}}};{\alpha^{ * }},{b^{ * }}) = {\rm sgn} \left( {\sum\nolimits_{{l = 1}}^m {{\alpha_l}{y_l}K({{\mathbf{x}}_l},{{\mathbf{x}}_{{test}}}) + {b^{ * }}} } \right) \). In Section 3.2, the located PCB defects are expressed by the extracted features and fed into the SVM learning system for training and testing procedures.

2.3 Multi-Class SVM Classification and Probability Estimates

Since SVM is designed as a binary classifier, different approaches have been proposed to use the two-class decisions for multi-class classification [23, 24]. In many situations, however, probability estimates are desired to quantify the results rather than only predicting the class labels. The scenario of SVM multi-class probability estimation can be summarized as follows: Let \( S = \left\{ {{{\mathbf{x}}_l},{y_l}} \right\}_{{l = 1}}^m \)be the set of training examples and \( {y_l} \in \left\{ {{ }1,2,...,k} \right\} \) be the class label associated to x l . One first needs to construct k(k-1)/2 pair-wised SVM classifiers where each one is trained on data from two classes, say, the i th and j th classes. For any new test sample x test , the pair-wise probabilistic outputs are computed from the SVM for the i th and j th classes, called the pair-wise class probabilities \( {r_{{ij}}} = P({y_{{test}}} = i|{{\mathbf{x}}_{{test}}},{y_{{test}}} \in \{ i,j\} ) \), and r ji =1-r ij . Next, the multi-class probability distribution p=(p 1,…,p k )T is computed from all pair-wise probabilities r ij ’s. In this research, Platt’s method [23] is adopted to calculate r ij between all class-pairs for x test . As a result, x test ’s multi-class probability distribution p is estimated by solving the following optimization problem [25]:

$$ \begin{array}{*{20}{c}} {\mathop{{\min }}\limits_{{\mathbf{p}}} \sum\nolimits_{{i = 1}}^k {\sum\nolimits_{{j:j \ne i}} {{{\left( {{r_{{ji}}}{p_i} - {r_{{ij}}}{p_j}} \right)}^2}} } } \hfill \\{s.t.\,\sum\nolimits_{{i = 1}}^k {{p_i} = 1,{p_i} \geqslant 0,\forall i} } \hfill \\\end{array} . $$
(5)

where p i :=P(y test =i|x test ), i=1,...,k is the posterior class probability. Equation 5 can be rewritten as:

$$ \begin{array}{*{20}{c}} {\mathop{{\min }}\limits_{{\mathbf{p}}} 2{{\mathbf{p}}^T}Q\prime {\mathbf{p}}} \hfill \\{s.t.\,{p_1} + {p_2} + ...{p_k} = 1} \hfill \\\end{array}, $$
(6)

where \( Q{\prime_{{ij}}} = \sum\nolimits_{{s:s \ne i}} {{r_{{si}}}^2{ }} \) if i=j, and \( Q{\prime_{{ij}}} = - {r_{{ji}}}{r_{{ij}}} \) otherwise. Since Q’ is a positive semi-definite matrix, p is the global optimum if it satisfies the first order necessary optimality condition:

$$ \left[ {\begin{array}{*{20}{c}} {Q\prime } & {\mathbf{e}} \\{{{\mathbf{e}}^T}} & 0 \\\end{array} } \right]\left[ {\begin{array}{*{20}{c}} {\mathbf{p}} \\\lambda \\\end{array} } \right] = \left[ {\begin{array}{*{20}{c}} {\mathbf{0}} \\1 \\\end{array} } \right], $$
(7)

where λ is the Lagrangian multiplier, e and 0 are k by 1 vectors for all ones and all zeros respectively. In Section 3.3, the cosmetic defectiveness is evaluated through block-wise comparing the multi-class probability distributions of image blocks in the inspection-reference image pairs.

3 The Learning-Based Inspection System

As the flow diagram in Fig. 1 shows, the PCB inspection system consists of an image acquisition system, and a processing system to perform the inspection on the acquired image. The image acquisition system has a mechanical positioning tool to scan and acquire the PCB image. The inspection process itself has two steps: image alignment and image diagnosis. First, the image acquired by the camera is aligned to the stored reference image as described in Section 3.1. Next, the PCB diagnosis is performed by detecting two kinds of defects in two different ways, namely: 1) the defects that appear as “patterns,” and 2) the cosmetic defects without regular appearance, which are usually caused by a failed electroplating process or a failed rising procedure between fabrication stages. The diagnosis processes of feature extraction and defect identification are respectively described in Section 3.2 and Section 3.3 for two kinds of defects. The diagnosis results of these two sequences provide the defect inspection of the PCBs. Each component of these steps is detailed in the following subsections.

Figure 1
figure 1

Flowchart of the proposed PCB inspection system.

Full size image

3.1 Image Alignment

The inspection process starts with the Hausdorff-distance based image alignment procedure that aligns the inspection image I to the reference image R. The notation R stands for an idealized image on which the alignment and the subsequent inspection procedures rely, and it can be obtained by a compilation or a rolling average of several defect-free boards in practice. In the experiments, R is constructed from the compilation of several good PCBs. The image alignment procedure is as follows: First, a Sobel edge detector is applied to I and R to extract the corresponding point sets I edge and R edge respectively. Next, by finding the minimal Hausdorff-distance value between I edge and R edge with respect to different geometric transformations, i.e. \( \arg {\min_{\theta }}{d_H}({I_{{edge}}},{a_{\theta }}({R_{{edge}}})){ } \), where a θ (•) is the rigid transformation with parameter θ, the image alignment process is accomplished.

To enhance the robustness of alignment, the one-way Hausdorff distance from R edge to I edge was actually used for reducing the influence of outlier points in I edge . In addition, the original max–min function in Eq. 2 was modified from calculating the maximum of minima to the mean of minima in the implementation because the result of applying the max function is sensitive to a few points located in the previously identified problem areas. The modified h function is defined as:

$$ {h_{{mean}}}(A,B): = \frac{{\sum\limits_{{a \in A}} {\mathop{{\min }}\limits_{{b \in B}} ||a - b||} }}{{\left| A \right|}}, $$
(8)

where |A| denotes the cardinality of point set A. Furthermore, the techniques of image pyramid and distance map images were used to speedup the alignment process. On the former technique, by constructing a layered image pyramid, full-search is applied only at the coarsest layer, and image alignment is successively refined through local searches on the candidates from the coarser layer. The searching procedure is stopped at θ * of the lowest value in the finest layer. And on the latter technique, a distance map R dist is used. The numerator of Eq. 8 can be rapidly retrieved in the execution stage by accessing R dist from storage. Figure 2 illustrates the use of the coarse-to-fine approach and the distance map for speeding up the image alignment process.

Figure 2
figure 2

Speeding up of alignment process using an image pyramid. a An example of three-layer pyramid for image alignment, b the local search area for refinement in the next layer, c an example of three-layer edge map pyramid computed from an inspection image I, and d the three-layer pyramid of distance map R dist constructed from the corresponding reference image R.

Full size image

3.2 Defect Detection and Classification

Given a well-aligned image pair of inspection and reference images, the following step is to handle defective patterns with regular appearance. The defects are first located using the partial Hausdorff distance based strategy, and subsequently categorized by SVM learning approach.

3.2.1 Defect Detection via Partial Hausdorff Distance

Suppose \( {e_p} \in {I_{{edge}}} \) is an edge point at position p on the inspection image I. It is easy to see, when referring e p to R dist at position p, a large value will be derived if the reference image does not contain such an edge point at p. This observation points at the possibility of locating defects by accessing the stored Hausdorff-distance map R dist . However, if outliers exist in the compared objects, the mismatch will become too large. For this reason, use of the partial Hausdorff distance is more reasonable than the original Hausdorff distance computation. The partial Hausdorff distance takes the K th ranked highest value instead of the overall maximum:

$$ { }d_H^p\left( {A,B} \right) = \min \left( {{h_p}\left( {A,B} \right),{h_p}\left( {B,A} \right)} \right), $$

where \( { }{h_p}(A,B): = K_{{a \in A}}^{{th}}{\min_{{b \in B}}}\left\| {a - b} \right\| = K_{{a \in A}}^{{th}}{d_B}(a). \)

Here \( K_{{a \in A}}^{{th}} \) denotes the K th ranked value in the set of d B (a) distances, which are the minimum distance values from each location a to the point set B. It is empirically set as the third quartile in the experiments.

In summary, given an inspected image I, the defect location procedure is as follows: Initially, by referring every edge point e p to R dist at position p, the distance map I dist is generated. Next, the edge pixels are marked as defective if the associated partial Hausdorff distance values on I dist are larger than a threshold. Finally, any defective pixels within a certain distance are labeled as one defect. Figure 3 demonstrates some example images for illustration. In the next section, the image patches of defects are fed into SVM for further evaluation.

Figure 3
figure 3

Two examples of defect detection: a an example of open defect, b edge points are marked as defective, c the image patch of defect is cropped for the use of subsequent classification, d–f another example of short defect.

Full size image

3.2.2 Defect Classification via Support Vector Machine

For an inspection system, the ability to analyze the statistics of defects in order to reduce the defective rate during manufacturing is extremely important. However, one of the most crucial factors for the performance of an inspection system is the quality of the inspection features. Here, two kinds of descriptors are designed to extract the discriminant features to better characterize the defective patterns for classification. They are:

  1. a.

    Horizontal / Vertical cross-count: The cross-counts with 16 equally-spaced scan-lines in horizontal and vertical directions are measured.

  2. b.

    Projection variations: The variation in projection with respect to 16 equally-spaced scan-lines is measured for both horizontal and vertical directions.

Having a set of collected samples of defective image patches, the descriptors were applied to convert image patches from pixels into the feature vectors for inspection. Specifically, let \( S = \left\{ {{{\mathbf{x}}_l},{y_l}} \right\}_{{l = 1}}^m \) be the set of collected defective patterns, where x l is the feature vector converted from the l th defective pattern, and \( {y_l} \in \left\{ {{ }1,2,...,k} \right\} \) is the defect type associated with x l . An SVM optimization problem is then formulated as introduced in Section 2.2 in order to train the classifier for the types of defective regions. In the testing phase, the types of defective regions are recognized using the trained SVM classifier for categorization.

3.3 Cosmetic Defect Detection

On the one hand, it is viable to identify some defects because of the regular patterns in defect appearance. On the other hand, because of the diverse appearance and the difficulty to characterize them, identifying cosmetic defects, which are caused by failures in the electroplating or rising processes between fabrication stages, is not easy. In the following section, through block-wise comparison of the multi-class probability distributions of image blocks, such non-regular defects are thus identified. Furthermore, a quantitative measure for the regional defectiveness is provided for the defective regions. The procedure for detecting the cosmetic defects consists of: 1) block feature extraction, and 2) block-wise and regional defect probability computation; which are elaborated below.

3.3.1 Block Feature Extraction

Identification of cosmetic defects is performed by block-wise comparing the SVM multi-class probability of PCB material blocks between the inspection and reference images. Suppose a set of image blocks of various PCB materials is collected in advance. To train the SVM model for estimating the multi-class probability for every image block, an effective feature extraction procedure is required. To better discriminate color textures, for each image block the feature is extracted by: First, the mean values with respect to each color channel \( \left( {{\mu^R},{\mu^G},{\mu^B}} \right) \) and the color of the central pixel \( \left( {{c^R},{c^G},{c^B}} \right) \) are calculated and stored. Second, the variance to each color channel (i.e. \( {v^R},{v^G},{v^B} \)), is considered as well, to keep the block’s color variation information:

$$ {v^{{color}}} = \sum\nolimits_{{i = 1}}^h {\sum\nolimits_{{j = 1}}^w {\left| {P_{{ij}}^{{color}} - {\mu^{{color}}}} \right|} }, $$
(9)

where \( P_{{ij}}^{{color}} \) denotes the intensity with respect to each color channel of {R,G,B}, and (i, j) stands for the pixel position in the h x w block. Third, three rotationally-invariant features (l R,l G,l B) are computed by applying the LoG (Laplacian of Gaussian) filter to the image block. In summary, for each image block of PCB material, a 12-dimentional feature vector (μ R, μ G, μ B , c R, c G, c B , v R, v G, v B , l R, l G, l B) is extracted. Let \( S = \left\{ {{{\mathbf{x}}_l},{y_l}} \right\}_{{l = 1}}^m \) be the set of examples, where \( {{\mathbf{x}}_l} \in {\mathbb{R}^{{12}}} \) is the corresponding feature vector associated with \( {y_l} \in \left\{ {1,2,...,k} \right\} \), the type of PCB material. As introduced in Section 2.3, the SVM model θ SVM for multi-class probability estimation is trained with S, and it will be used to assign the probability of the material type to image blocks in the next section.

3.3.2 Defect Region Determination

Let \( {\mathbf{b}}_I^{{(i,j)}} \) be the (i,j)th image block of the inspection image I. After learning from S, the SVM model θ SVM is able to associate \( {\mathbf{b}}_I^{{(i,j)}} \) with a probability\( P({\mathbf{b}}_I^{{(i,j)}}|{\theta_{{SVM}}}) = {(p_1^{{(i,j)}},p_2^{{(i,j)}},...,p_k^{{(i,j)}})^T} \), which represents the probability of the material type for \( {\mathbf{b}}_I^{{(i,j)}} \). The defectiveness of \( {\mathbf{b}}_I^{{(i,j)}} \)is next measured by computing the distance between \( P({\mathbf{b}}_I^{{(i,j)}}) \) and \( P({\mathbf{b}}_R^{{(i,j)}}) \), where \( P({\mathbf{b}}_R^{{(i,j)}}) \) denotes the probability distribution computed from the corresponding block \( {\mathbf{b}}_R^{{(i,j)}} \) in the reference image. Clearly, the farther the distributions \( P({\mathbf{b}}_I^{{(i,j)}}) \) and \( P({\mathbf{b}}_R^{{(i,j)}}) \) are, the more likely that block \( {\mathbf{b}}_I^{{(i,j)}} \) is defective. The relative entropy [26], also known as Kullback-Leibler (KL) distance, is used here to measure the distance between the two probability distributions \( P({\mathbf{b}}_I^{{(i,j)}}) \) and\( P({\mathbf{b}}_R^{{(i,j)}}) \):

$$ \begin{array}{*{20}{c}} {KL\left( {P\left( {{\mathbf{b}}_I^{{\left( {i,j} \right)}}} \right),P\left( {{\mathbf{b}}_R^{{\left( {i,j} \right)}}} \right)} \right)} \hfill \\{ = \sum\nolimits_{{l = 1}}^k {P{{\left( {{\mathbf{b}}_I^{{\left( {i,j} \right)}}} \right)}_{{{ }l}}}\log \left( {\frac{{P{{\left( {{\mathbf{b}}_I^{{\left( {i,j} \right)}}} \right)}_{{{ }l}}}}}{{P{{\left( {{\mathbf{b}}_R^{{\left( {i,j} \right)}}} \right)}_{{{ }l}}}}}} \right)} } \hfill \\\end{array}, $$
(10)

where \( P{\left( {{\mathbf{b}}_I^{{(i,j)}}} \right)_l} \) and \( P{\left( {{\mathbf{b}}_R^{{(i,j)}}} \right)_l} \) denote the l th entry of the corresponding probability distributions. Finally, a logistic sigmoid function is fit to transform \( KL(P({\mathbf{b}}_I^{{(i,j)}}),P({\mathbf{b}}_R^{{(i,j)}})){ } \) to the probability of the block defectiveness:

$$ p({\mathbf{b}}_I^{{(i,j)}}) = 1 - \exp ( - \beta \times KL(P({\mathbf{b}}_I^{{(i,j)}}),P({\mathbf{b}}_R^{{(i,j)}}))){, } $$
(11)

where β is an empirically-chosen parameter. For better representation, the neighbouring defective blocks are grouped as a single defective region using the standard morphological operators and the connected component approach [27]. The regional defectiveness of this cosmetic defect is defined as:

$$ {R_{{defect}}}(D): = \exp \left( {\frac{1}{{\left| D \right|}}\sum\limits_{{{\mathbf{b}}_I^{{(i,j)}} \in D}} {\ln p\left( {{\mathbf{b}}_I^{{(i,j)}}} \right)} } \right), $$
(12)

where |D| denotes the total number of the defective blocks inside D. To recap, as explained above, through block-wise comparing the SVM multi-class distributions between the inspection and reference image blocks, the defective blocks are identified. These defective blocks are then grouped into a defective region with the defectiveness R defect (D). For better illustration, the intermediate results of each stage are visualized in Fig. 4. The performance of the proposed inspection system is evaluated in the next section.

Figure 4
figure 4

The process of defect detection and defectiveness computation. a The inspected image. b The normal reference image. c The block defect probability displayed by intensity. d The defect region after applying closing operator. e The labeled defect region with defectiveness R defect .

Full size image

4 Experimental Results

4.1 Imaging System

The image acquisition unit of the inspection system was composed of a monochrome area CCD, lens, and a 3-color LED light source. The lighting mode was switched rapidly among R, G, B by the LED light source, and inspection images of the corresponding color channels were obtained by the simultaneously triggered CCD. Before taking the PCB images, the CCD was calibrated with the flat field correction technique to ensure the uniformity of the obtained images. The imaging acquisition component is shown in Fig. 5a, together with a snapshot of the system output in Fig. 5b. Some examples of the observed metal as well as the substrate are shown in Fig. 5c. The images acquired from this imaging system were subsequently processed by the proposed inspection algorithms to test their performance.

Figure 5
figure 5

Image acquisition components and the examples for SVM training and testing. a Image acquisition set-up for PCB defect inspection, b a snapshot of image acquisition system output, and c examples of the observed metal and substrate images.

Full size image

4.2 Pattern Defects

As previously reported in the literature, most common defects on PCBs are caused by errors of thermal expansion of the artwork during printing, dirt on board, air bubbles from electrolysis, incorrect electrolysis timing, or mechanical mis-registrations, which appear on the finished PCB products and cause defects as: open, short, pinhole, under-etch, or mousebite [1, 28]. The experiments of this work focused on these five common defects. In the experiments, the defective examples in the database were from the failed PCBs during the manufacturing process and acquired by the imaging platform. Eighty images for each error type were obtained from real defect images or by simulating defect images from non-defective images, thus yielding a total of four hundred PCB images in the database. Some examples of defect detection results are given in Fig. 6, where the detected defects are marked by rectangles. In the experiments, 100% detection rate was achieved using the proposed partial Hausdorff-distance based method. The detected patches were subsequently fed into SVM to further classify their defect types.

Figure 6
figure 6

Inspection results with different defects marked on PCBs.

Full size image

The system performance for defect classification was evaluated using five-fold cross validation. That is, the detected image patches were randomly divided into five portions. Each portion was then applied once for assessing the performance of the classifier by using the rest four portions as the training data. In Table 1, the average classification accuracy and total numbers of support vectors for SVMs in conjunction with various kernels (i.e. polynomial and RBF kernels) are compared. The classification accuracy of a single layer perceptron (SLP), which consists of ten hidden nodes associated with weights trained by the LMS method [29], is also provided for benchmarking. From this table, we see the SVMs incorporated with the polynomial kernels (with different degrees d) performed properly in terms of the classification accuracy. On the other hand, the RBF kernel SVM with an appropriate kernel parameter setting achieved the highest classification accuracy, but it requires the highest computational complexity and the largest number of support vectors (SVs) compared to other kernels in the comparison. Applying more complex kernels (i.e. higher order polynomial kernels) here to map the data to higher dimensions may lead to smaller training errors; however, it may cause the system to overfit the training data and thus degrades the accuracy of the trained system during testing. The overfitting phenomenon occurs probably because the available PCB defect samples are limited. As a result, the linear kernel was adopted for its computational efficiency and satisfactory performance with this limited experimental dataset. However, it may need to use a higher-order polynomial or a more sophisticated kernel for more complicated PCB classification tasks.

Table 1 Comparison of SLP and SVMs with different kernels by 5-fold cross validation.
Full size table

The classification result was further evaluated to study the confusions among various defect types. In Table 2, a confusion matrix of the accumulated defect classification results over five rounds is given, where each row represents the instances in an actual class and each column represents the instances in a predicted class. From the table, there are more misclassification cases between “open” and “mousebite” defect types, which are quite similar in appearance. The average system execution time is reported in Table 3 for each phase of the proposed inspection system. The computer platform is equipped with 1.86 GHz CPU and 1G RAM onboard. Compared to the state-of-the-art approach of Chang et al. [17], the proposed system inspects defects faster, and it is more reliable in defect detection and more accurate for defect classification. Besides, the proposed learning-based inspection system is easy to adapt for new defects, while the rule-based system [17] needs to design additional rules from the new defect observations and requires checking the dependency between rules. Details of the comparison were summarized in Table 4. Note that, in real situations, the defect patterns may not belong to any of the categories covered by the database. This will lead to mis-classification. A verification procedure (e.g. non-reference verification) can be employed to give warning of possible mis-classifications. By collecting new samples and training the corresponding sub-classifiers, the proposed learning-based inspection system can be easily extended to recognize new types of defects.

Table 2 The confusion matrix of the defect classification results.
Full size table
Table 3 Average system execution time.
Full size table
Table 4 Comparison of the proposed method and CBR evolutionary model [17].
Full size table

4.3 Cosmetic Defects

In the following experiments, the defective PCBs with cosmetic defects of skip plating, copper exposure, and those with abnormal oxide on board were inspected by block-wise comparing their probability distributions. The multi-class probability distributions were estimated by the SVM with linear kernel. Figures 7a and 7g are two problematic PCB images being inspected, while 7b and 7h are the corresponding defect-free reference images. In Fig. 7a, an irregular region containing foreign oxidized material is seen on a Ni-Cu image, and in Fig. 7g a defective region of copper exposure is shown. Figures 7c, 7d and 7i, 7j are the SVM material classification results for the corresponding inspection and reference images. As for Figs. 7e and 7k, they are the intensity representation of the block defect probabilities for the inspection images in Figs. 7a and 7g, respectively. The final inspection results were shown in Figs. 7f and 7l depicting the detected regions with the values for defectiveness. Higher defectiveness value is shown if a region consists of blocks more dissimilar from the normal board, so the region is more certainly identified as a defect.

Figure 7
figure 7

Experimental results for material defect inspection: a–f Ni-Cu image with foreign oxidized material, and g–l PCB image with copper exposure.

Full size image

Figures 8a and 8g are another two problematic PCB images being inspected, while 8b and 8h are the corresponding defect-free reference images. The layout of the figures is similar to Fig. 7. As for the first problematic case shown in Fig. 8a, there is an irregular region with foreign oxidized material on an Au-Ni image. The proposed system successfully detected the defective region with a high confidence value because the defective region has a very different appearance from that of the corresponding reference image. On the other case shown in Fig. 8g, the PCB image concurrently has defects of skip plating and copper exposure. In Fig. 8k, the pad region of skip plating displayed with bright intensity was successfully detected with high probability. It is evident that the system also detects the regions with slight copper exposure. Yet, because copper is quite similar to gold in natural appearance, those blocks were not as firmly recognized to be defective and were associated with the lower regional defectiveness value in Fig. 8l. Thus, the reliability and the effectiveness of the proposed inspection system for identifying the cosmetic defects were validated through this experiment.

Figure 8
figure 8

Experimental results for material defect inspection: a–f Au-Ni images with foreign oxidized material, and g–l pads of skip plating and copper exposure on the PCB.

Full size image

5 Conclusions

This paper presents an innovative and automated learning-based PCB inspection system. It compares the inspection image to a reference image, corrects the misalignment errors, and has the ability to accurately assess defects before they propagate further down the assembly line, while providing very rapid data analysis. In the experiments, the effectiveness of the system was demonstrated for defective patterns (opens, shorts, pinholes, mousebite, under-etch), and for cosmetic defects of irregular appearance (copper exposure, skip plating and oxides on board). The experiments indicated that the learning-based AOI system provides a reliable and flexible solution for PCB inspection, and it can be easily extended to deal with unknown defects for future PCB inspection tasks.

Several future directions for extending the proposed AOI system are discussed in the following. A possible improvement is to include the feature selection step into the system for enhancing both the generalization capability and the computation speed of the learned model. Since the proposed algorithms are flexible, handling different diagnostic tasks with the proposed algorithms according to the positions on the PCB assembly line is also feasible, such as SMT component placement inspection. Finally, incorporating different imaging technologies by using visible or invisible light, e.g. X-ray, into the system may also help the inspection for multi-layer PCBs.