Machine vision-based intelligent manufacturing using a novel dual-template matching: a case study for lithium battery positioning

Abstract

The fast and precise positioning of lithium battery is crucial for effective manufacturing of mass production. In order to acquire position information of lithium batteries rapidly and accurately, a novel dual-template matching algorithm is proposed to properly locate and segment each battery for fast and precise mass production. Initially, an image down-sampling method is applied to build up a multi-layer image pyramid for speeding up target search, and a novel mixed matching template is designed to increase the matching precision. A row of lithium batteries is likely tilt during rolling, and the images of batteries captured by the CCD camera are distorted, which may generate a negative effect on next procedure. Hence, a two-level correction algorithm for battery angle and location is applied to obtain rough areas of the batteries and improve the accuracy of template matching. Lastly, the comparison with other state-of-the-art algorithms is done to locate each battery in a row with high speed and precision. The precision rates of the proposed algorithm, improved SAD algorithm, and YOLOv3 algorithm are 99.44%, 95.98%, and 93.64 for normal battery images and 97.86%, 89.19%, and 85.10 for tilted battery images, respectively. Compared with improved SAD matching algorithm and YOLOv3 algorithm, the positioning accuracy of the proposed method is significantly increased, and the matching robustness is improved in spite of large battery inclination angle.

1 Introduction

The problems of single energy structure, fossil fuel reserves, and greenhouse gas overshoot are increasingly serious that leads to look for alternative energy sources in academia. In certain cases, the best solution for above challenges is choosing other power supplies. Lithium battery, which has advantages of high energy density, long service life, and less harm to environment, is one of the best substitutes and has been generally used in day to day life [1]. In lithium battery industries, the mass production of qualified products is related to the success of the enterprise, and the need to detect lithium batteries and defects on the battery surface automatically is urgent. Unfortunately, the inspection of lithium battery is monotonous and time-consuming for humans. In order to acquire available battery picture for subsequent processing, locating lithium batteries accurately and obtaining the segmented image of each battery from original image are the essential and key steps for follow-up works. Thus, there is a need of fast and precise algorithm to locating and recognizing lithium batteries automatically. A great interest in the development of machine vision-based automatic positioning and detecting method has been aroused.

With the rapid development of machine vision [2, 3], object detection and bar code recognition of lithium battery based on image captured by CCD or CMOS camera have been rapidly developed. The accuracy and speed of positioning directly affect the efficiency of battery detection and barcode positioning [4,5,6] that can be realized by carefully designed detection algorithm. The main approaches of object positioning are classified as template matching-based methods, image content-based object positioning, and deep learning-based object detection methods. For image content-based object positioning and detection methods, the image content must be analyzed first. Based on the subjective analysis results, the specific algorithm can be designed that leads to inefficient or complexity. In last decades, deep learning provided many best solutions in image processing, natural language recognition, and autopilot. However, the disadvantages of large training samples and computational resources limited its development in some particular field. The common steps for template matching-based-methods are as follows: generating template(s), computing similarity, post-processing, and getting results. Although it is one of the simplest approaches, the precision rate and speed are depended on the quality of the selected template.

To meet the requirements of fast detection, the lithium batteries are designed to roll down from the slope. To the best of our knowledge, there is no certain algorithm designed for lithium battery positioning while batteries rolling down. However, the rolling battery is tending to tilt, and the grayscale of the rolling battery image with barcode changes dramatically during the process of rolling down. Considering above factors, the traditional template matching method cannot have good performance, and the positioning accuracy of existing template matching methods is significantly deteriorated. In order to bridge this research gap, a novel template matching-based lithium battery positioning method is proposed in this paper. Based on this method, the tilt lithium batteries are corrected and positioned during the rolling down process by dual-template matching algorithm. The unique matching template applied in following steps was carefully designed that showed outstanding improvement compared to other contrast algorithms.

The remainder of this paper is structured as follows. In Section 2, some related works are discussed based on literatures. Section 3 describes the summary of our algorithm and the design theory of the matching template used in our algorithm. The improved template matching algorithm is proposed, and the detailed implementation is presented in Section 4. The experimental process and results are discussed in Section 5. Section 6 concludes the paper and describes the future works.

2 Literature review

Recent literatures relevant to this paper are mainly concerned with two research streams: object detection methods and template matching approaches. Thus, the relevant literature has been tried to summarize in this paper.

2.1 Object detection methods

Object detection is the foundation of object categorization, inspection, and measurement that plays an essential role for industrial automation. Many researches have been performed to improve object detection precision and efficiency. Some typical issues were discussed in [7], and an image segmentation method based on systematic algorithm workflow and global threshold was proposed, which resulted in high segmentation performance for all sectioning images. Do et al. [8] designed an embedding method which mapped image describing features to a higher dimension and tried to solve image retrieval problem. A novel contextual multivariate information bottleneck (CMIB) method was proposed in [9], and two Bayesian networks are built to make a compromise between data compression and information preservation, which was enlightening in some ways. A new model pipeline based on joint inference and context fusion was proposed to alleviate two challenges of dense visual concept annotations and the large amount of visual concepts in [10].

Also there are many outstanding deep learning-based object detection algorithms. A deep learning-based defect detection approach of lithium battery electrode in [11] was proposed, and the pretrained networks for microstructure classification were applied. The method showed high classification accuracies in datasets. Lin et al. [12] proposed a semantic image segmentation approach using patch-patch context and patch-background context in deep CNNs. The results showed high performance on amount of semantic segmentation datasets. There are mainly two categories in the field of object detection, e.g., RCNN series method and the YOLO series method, which is based on the region proposal and the regression, respectively. Girshick et al. [13] acquire a bunch of potential regions by the method of selective search, and the features are extracted based CNNs in these selected regions for judgment. The YOLO [14] series method predicted multiple bounding box and categories simultaneously, which can achieve fast speed. And the improvement version, e.g., YOLOv3, can set focus on speed or accuracy by modifying the size of the model structure. A novel residual learning framework was proposed by He et al. [15] to reduce training difficulty, and evidence showed the residual networks being better performance on many datasets. By predicting the rotated bounding boxes by leveraging a segmentation task, Wang et al. [16] developed a new training and inference mechanism to improve the detection accuracy for high resolution images. Seongdeok et al. [17] developed a novel image augmentation method with generative adversarial network (GAN) to improve construction resource detection, and the results were optimal for images captured by unmanned aerial vehicle (UVA).

2.2 Template matching approaches

Talmi et al. [18] proposed a template matching algorithm, named Deformable Diversity Similarity, to improve the matching robustness to some certain situations, and the evaluation showed that the proposed algorithm performed better than other methods in complexity image processing. A combined method in [19] based on deep learning- and template-based algorithm is applied in battery production, which shows a significant improvement in both speed and accuracy. Liang et al. [20] proposed a novel image matching method based on gradient space in which image pyramid and Hessian matrix is used to detect the interest points with invariable scale. The experiments showed robustness to variation of image parameters. In order to realize object tracking, object grasping, and other tasks based on vision for humanoid robot, Lopezfranco et al. [21] proposed an image template matching method based on differential evolution algorithm. Korman et al [22] applied a branch-and-bound-like scheme to accelerate the matching process, and the result is a good compromise between performance and quality. Bao et al. [23] proposed a multi-template matching method with limited template matching library cucumber image, which still has potential improvement for recognition accuracy. Bekkers et al. [24] proposed a two-dimensional image target template matching method based on directional graph features, which used a generalized linear regression framework to construct appropriate templates with smooth splines. Dekel et al. [25] also improved the BBS and acquired a novel template matching method which showed consistent success on real-world datasets. For the high resolution images, Chen et al. [26] introduced a line-based matching method to overcome the computational consuming which was robust and efficient in some certain conditions. A multi-source image matching method [27] with the implementation of best-buddies direction pairs was proposed to decrease the influence caused by intensity changes or image noise, which demonstrated great computational efficiency and matching performance. Xia et al. [28] proposed a rapid template matching method adopting the shape-shifting Best-Buddies Similarity (BBS) measure which can match patches of different sizes, so it had higher accuracy than the original method based on BBS. To address the problems of long computation time of target matching algorithm and incorrect positioning of rotating scale target, Yu and Fei [29] proposed a spiral target matching algorithm on the basis of pyramid image structure and Hu moments. Pedrosa et al. [30] improved pre-process for template matching method by morphological image processing (MIP) and fast Fourier transform (FFT) and successfully detected impact craters in a set of images from Thermal Emission Imaging System onboard the 2001 Mars Odyssey Space probe. Lee et al. [31] proposed an improved template matching method called deformable template matching (DTM) based on deformable part model (DPM) to acquire deformation by predefined rules, and the experimental results showed an increased number of matching features. A hash-based template matching algorithm was introduced in [32]. With less complexity and speeding-up processing, the experimental results showed improved coding efficiency. Adrian et al. [33] proposed an effective method to predict the matching of templates before actual execution, which can significantly reduce computational cost and avoid unnecessary resource wastes. Due to the color similarity of cucumber and its branches and leaves, it is difficult to identify the cucumber with high precision by simply adopting color features and shape features.

The proposed algorithm in this paper provides the advantages of accuracy, stability, and efficiency. The above algorithms can be divided into two strategies. One is the template matching method based on the contour features of images, which improves the accuracy of target positioning and can also locate fuzzy, even distorted target images. The other method improves the searching speed and searching strategy by constructing image pyramid structure or improving the deformation template algorithm, but does not improve the self-adaptability and the robustness, especially for multi-target image matching, is poor.

2.3 Discussion

Although many approaches to detect and segment particular object applied in industrial domain have been proposed in above literatures, they have some common disadvantages summarized as follows. Firstly, approaches in above literatures attempt to detect objects that were suitable for specific situation and cannot be used in lithium battery detection with simple modification. Secondly, because of the specificity of the application scenario, the compromise between performance and quality was not ideal. Finally, since the target images with barcode on battery surface have large gray-level variations and the batteries may be tilted while rolling down from the slope, template match approach with single template was not suitable and cannot achieve optimal accuracy and efficiency. The image processing algorithm based on deep learning is one of the potential approaches. Due to practical conditions, there are only hundreds of lithium batteries including some defective ones, which is far from enough for training CNN models. Besides the industrial PC applied in lithium battery positioning does not have graphic process unit (GPU), the validation on trained models would be inefficient.

In this paper, a dual-template matching method for multiple lithium batteries is introduced, in which the pyramid-level search strategy with reducing sampling was applied to establish the image structure and improve the search speed. A simulation example and comparison with other algorithms are carried out. Based on the comparison results, the proposed method is proved accurate and efficient.

3 The improved dual-template matching method

3.1 Framework of the proposed method

The improved dual-template matching method consists of image pre-process, matching template design, algorithm implementation, and experimental verification, which is shown in Figure 1. The preprocessing step consists of down-sampling, image graying, and filtering processes, binarization, and morphological operation. The pyramid down-sampling is necessary since the original pixel of frame captured from industrial camera is 1920×1200, e.g., 2,304,000, and the template pixel without preprocessed is around 160×573, e.g., 91680. Hence sliding through each down-sampled frame by the improved matching template decreased time-consuming significantly. After the preprocessing, the output standard image is obtained which has little noise and can be applied in template matching process. Since the propose template matching method with diverse dual templates is unique from others, the matching template needs to be designed elaborately. Considering the situation of rolling lithium battery and the discrete appearance of barcode on battery surface, the template design was divided into two steps, e.g., top template design and bottom template design. The core step of the proposed algorithm is using carefully designed dual template to recognize each battery in down-sampled image and mapping the identical battery coordinates to original image, which consists of approximate matching and elaborate matching.

Fig. 1.

Framework of the proposed dual-template matching method.

3.2 Preparation

Template matching is to slide the template along the target image pixel by pixel to find the best matching value. Template design has a great influence on image matching. Tiny difference value of pixel and template tilt angle will cause great matching precision decrease for target image, especially the template matching of multi-target image, for which if the morphological position and geometric structure of the target image are significantly different from the features of the template, all the targets in the figure cannot be located. So initial template design has highest priority.

When a lithium battery rolls, the barcode along its side may be captured entirely, partially, or none. Therefore, under the same acquisition environment, the surface information value of the same lithium battery may be significantly different. Lithium battery rolling on the slope and tilt is nearly inevitable, which has a huge effect on positioning accuracy of lithium battery.

This paper uses a template designed by single lithium battery. The bar code on the rolling lithium battery may appear in different states including no bar code, half bar code, and entire bar code on exposed surface, which is shown in Figure 2 with each lithium battery numbered.

Fig. 2.

Nos. 1, 2, 5, and 8 have no barcode. Nos. 4 and 6 have partial barcode. Nos. 3 and 7 have a complete barcode.

Rolling lithium battery will present above several states, so we need to determine whether all these states are suitable for making the matching template. By analyzing the influence between the presence and absence of barcode on the matching accuracy, the lithium battery with barcode on the exposed surface was taken to design the template, and the processing result is shown in Figure 3. The steps are as follows:

• Step 1: Image pre-processing after image pyramid down-sampling, including image graying, image filtering, and morphological expansion operation after adaptive binarization.

• Step 2: The tilt angle calculation and correction by rotating battery.

• Step 3: Battery boundary discovery for the first time. Adjust parameters for image binarization and morphological processing until lithium battery located.

Fig. 3.

a Original image, b processed rotated image, and c finally located image.

Template matching is used to analyze the similarity between the image and the template slid from top to bottom, left to right of the target image. Higher matching degree means higher similarity with the sub-image specified in the target. Since the pyramid down-sampling is applied in preprocessing stage, the scales of image and template have changed. To improve the matching accuracy and eliminate the potential influence from changing scales, the correlation coefficient template matching algorithm is applied, which is to match the mean relative value of the template with the sub-image. The matching result 1, -1, and 0 represents a perfect, worst, and undesired match, respectively. The expression is as follows: (1). Since the barcode area contained in the battery template is sometimes larger than the whole template, the pixel value of the barcode area differs greatly from the pixel value of the non-barcode area. The matching degree is calculated after average value calculation; thus, the matching degree is barely satisfactory:

$$ {R}_{\left(x,y\right)}=\frac{\sum_{a^{\prime },{b}^{\prime }}\left({T}^{\prime}\left({a}^{\prime },{b}^{\prime}\right)\bullet {I}^{\prime}\left(x+{a}^{\prime },y+{b}^{\prime}\right)\right)}{\sqrt{\sum_{a^{\prime },{b}^{\prime }}{T}^{\prime }{\left({a}^{\prime },{b}^{\prime}\right)}^2\bullet {\sum}_{a^{\prime },{b}^{\prime }}{I}^{\prime }{\left(x+{a}^{\prime },y+{b}^{\prime}\right)}^2}} $$

(1)

where R_(x, y) represents the similarity between the template and sub-image, T^′(a^′, b^′) represents the relative pixel value of the template in (a^′, b^′), and I^′(x + a^′, y + b^′) represents the relative pixel value of the target image overlapped by template, and its expression is as follows:

$$ \left\{\begin{array}{c}{T}^{\prime}\left({a}^{\prime },{b}^{\prime}\right)=T\left({a}^{\prime },{b}^{\prime}\right)-\overline{T}\\ {}\overline{T}=\frac{\sum_{a,b}T\left(a,b\right)}{w\bullet h}\end{array}\right. $$

(2)

$$ \left\{\begin{array}{c}{I}^{\prime}\left(x+{a}^{\prime },y+{b}^{\prime}\right)=I\left(x+{a}^{\prime },y+{b}^{\prime}\right)-\overline{I}\\ {}\overline{I}=\frac{\sum_{a^{\prime },{b}^{\prime }}{I}^{\prime}\left(x+{a}^{\prime },y+{b}^{\prime}\right)}{w\bullet h}\end{array}\right. $$

(3)

where $\overline{T}$ represents the average value of the template which is only computed once and $\overline{I}$ represents the average value of template overlapped region in the target image; w and h represent the width and height of the template respectively.

From the analysis of the above equations, it can be seen that the extra barcode pattern will increase the difference of pixel value leading to increase output value from Equation (2). The similarity between template and target image would decrease because denominator value increases faster than numerator in Equation (1).

3.3 Lithium battery template design

The separated lithium batteries with or without barcode are analyzed in Figure 4, with a green vertical line crossing through the lithium battery center. The grayscale curves corresponding to above two lines with identical color are shown in the histogram. It is obvious that the curve in Figure 4c has dramatic changes compared to the one in Figure 4d, which is corresponding to the barcode absence. The lithium batteries are rolling from the slope, and the barcode on the lithium battery occurs repeatedly. Hence, the whole lithium battery chosen as the matching template is not ideal since the significant variation of grayscale. However, compared with each other, the grayscale curves in the red circle and blue square regions, which are corresponding to the top and bottom side of the lithium battery, are stable and similar. Furthermore, the rolling batteries are placed randomly, e.g., the “+” side of lithium battery may appear in the top or bottom of image randomly. So, two sets of dual template are chosen to locate lithium batteries in a row, which are shown in Figure 4 e and f.

Fig. 4.

Comparisons of lithium batteries with or without barcode. a The lithium battery with barcode. b The lithium battery without barcode. c The grayscale curve corresponding to the lithium battery with barcode. d The grayscale curve corresponding to the lithium battery without barcode. e The dual template with “+” side at the top. f The dual template with “+” side at the bottom.

3.3.1 Top template design

In order to reach maximum speed of lithium battery template matching, the size of the target and template image should be down-sampled. The smaller image size could significantly increase matching speed with the cost of slightly reducing the matching accuracy of the original image.

A lithium battery image can be divided into three parts, which are barcode region and above and below barcode regions. For the matching of top-level pyramid image, the template is only used for rough positioning. The top part pixel differences between images with or without code on the exposed surface are slight, so the battery without code on the exposed surface can be chosen as the template, as shown in Figure 5.

Fig. 5.

a Battery surface feature distribution. b Final battery template.

3.3.2 Bottom template design

The bottom template is used to match the lithium battery in the original image, and the plate has the same design idea as the top layer module. It not only contains the typical characteristics of lithium battery, but also makes the template size not too large, which means optimum balance between performance and accuracy.

Initially, the image containing lithium battery has been preprocessed, then perform morphological operation. Referring to Section 2.1 for the detailed process, the battery tilt angle calculated from previous procedure is being used to correct the battery position. The wheel width detection algorithm is adopted to divide the battery. Combined with the size of the top layer template, the bottom layer template size is preliminarily set and determined by histogram, which is shown in Figure 6.

Fig. 6.

a Separated lithium batteries. b Correspond to the pixel values of the two lines in a. c The underlying template.

The separated lithium battery is analyzed in Figure 6a, with a red vertical line crossing through the bar code and a blue vertical line along the lithium battery center. The pixel value curves corresponding to above two lines with identical color are shown in the histogram. The green circle above in Figure 6a corresponding to point 1 in the histogram (Fig. 6b) is the dividing point between the battery and background, and the value is 51; the green circle below in Figure 6a corresponding to point 2 in the histogram (Fig. 6b) is the starting point of the bar code, and the value is 218. It can be seen from the histogram that the pixel value in middle area of the battery is higher than that in the side area. So we were able to obtain the required template from the segmentation lithium battery image by specific area, which was determined to be 140 pixels high (with a black background of 1/7) and 255 pixels wide.

4 Positioning method

4.1 Positioning statement

In order to locate the barcode of lithium battery accurately, it is necessary to segment lithium battery precisely. The positioning system of machine vision realizes the positioning of lithium battery through image recognition technology. Figure 7 shows the schematic of image acquisition device for lithium battery. Ten lithium batteries from the upper of inclined pathway (with Θ degree inclined angle) rolled into the image acquisition area. A specific template matching algorithm was used to locate the collected image and calibrate the lithium battery sequence; therefore, it is necessary to locate each lithium battery in the picture. The battery tilts and blurs at the edges during rolling. By adjusting the angle of the matching template to vertical direction, the positioning image with high precision can be obtained.

Fig. 7.

Schematic diagram of image acquisition platform for lithium battery.

4.2 Improved algorithm

In order to realize fast and high precision positioning of lithium battery, an improved template matching algorithm was proposed based on the original template matching algorithm. The Section 2gives the matching template corresponding to the target image of the top layer and the bottom layer of the pyramid, which has a positive effect on the template matching speed of the improved algorithm. The algorithm flow chart is shown in Figure 8, as follows:

• Step 1: The original image is processed with mean filter of small template and down-sampled twice to acquire image pyramid.

• Step 2: Preprocess the down-sampled images by grayscale, adaptive binarization, and image morphology.

• Step 3: Calculate the coordinate and approximate inclination angle for each lithium battery and rotate the image according to the average inclination angle ${\overline{\uptheta}}^{{}^{\circ}}$ of all batteries. Then divide the image by each battery’s coordinate and match each segmented image with the specified template to get the coordinate of the maximum matching G_max.

• Step 4: Obtain the source image by rotating the original image with ${\overline{\uptheta}}^{{}^{\circ}}$ and then map the best coordinate of matching degree in step 3 to the source image. After the initial template matching, determine whether the matching degree is bigger than G_max.

• Step 5: The images containing several matching features of lithium battery have been extracted from the source by referring to the best mapping matching coordinate, and roughly match them with the image template in inclination range of [-α, +α]. Calculate the best matching value and its coordinate and then select the angle range within [-α/10, +α/10] around the best matching degree obtained from previous step for accurate template matching.

• Step 6: The specific matching coordinate is mapped into the source image; then each lithium battery is located and segmented according to the size of the battery template.

Fig. 8.

Flow chart of improved template matching algorithm.

4.3 Battery tilt angle calculation

Lithium batteries tilt when they roll on a tilted track without guard bar on either side, so each battery in a row captured by industrial camera is tilted at a small, slightly different angle. Thus the tilt angle of the battery will lead to decrease of positioning accuracy, or even being unable to locate, which results to the confusion of the whole lithium battery sequence. In order to effectively improve the positioning accuracy of lithium battery, the tilt angle should be calculated firstly. The calculation process of tilt angle is shown in Figure 9a.

Fig. 9.

Battery tilt angle calculation process. a Flow chart for calculating inclination angle of lithium battery. b The battery search path of coordinate values

First of all, preprocess the image and select rectangular structural elements to corrode the image for several times. Then use the same structural elements to expand the previous image. The result is the binary image of 0 or 255 with all the noise points removed, entire basic features of the battery, and gaps between adjacent batteries, as shown in Figure 9b.

Secondly, according to the distribution characteristics of lithium batteries in the picture, a horizontal line passing through the center of the picture must pass through all lithium batteries. At the center line M, the expression is as follows (4): the approximate coordinate of each battery boundary was obtained by using the difference of pixel values on the battery boundary, and the number of batteries in the figure was determined. Then, the approximate coordinate and the number of batteries were calculated at the distance T up and down from the horizontal centerline, namely M_up and M_down; the expression is as follows (5) and (6). The coordinates, cell number, and M contrast with each other. If somewhere coordinates differ with the set value or the battery number range, search new M_up and M_down by reducing T with ∆T and compare coordinate differences and cell number again, until meet the requirements or reach the limit.

Finally, calculate the approximate inclination angle θ_i° of each lithium battery and the average inclination angle $\overline{\uptheta}$ of all batteries by using the corresponding coordinate differences.

$$ M=\mathit{\operatorname{int}}\left(H/2\right) $$

(4)

$$ {M}_{up}=M-T+\delta \ast \Delta T $$

(5)

$$ {M}_{down}=M+T-\delta \ast \Delta T $$

(6)

Η and M in Equation (4) indicate the height of the graphics and centerline, respectively. In Equations (5) and (6), M_up is a horizontal line at the above of M, the delta is scale factor of M_up, and set a stop cycle the scale factor of Ln. In the same way M_down is a horizontal line at the below of M. Their direction are shown in Figure 9b by blue arrow, moving distance is one ΔΤ at a time, and when the search battery in the horizontal coordinates and M coordinates ΔX is not greater than the difference between the P, the horizontal line stop searching. P is determined by the limit of the height of the battery and its tilt angle θ_max.

4.4 Improved template matching

4.4.1 Approximate template matching for original image

Template matching algorithm selects a specified place similar to the template when sliding chosen template across the target image, but the battery barcode position in the picture is uncertain, which will result in the target picture with a big difference from the template tending to the failure of matching and unable to meet the precise positioning of the battery. Therefore, instead of multi-target template matching, the concept of template matching for multiple single targets is proposed. In this method, the image is counterclockwise rotated ${\overline{\uptheta}}^{{}^{\circ}}$ with (W/2,H/2) as the center, and the batteries in a row are divided into n parts according to the coordinate difference between adjacent batteries on the horizontal line H_r/2 of the rotated image so as to avoid missing lithium battery during matching. Next, we use the designed template to match each segmented image and calculate the best matching coordinates and matching degree of each battery image. With all matching degree values compared, the maximum value G_max is selected.

The relationship between the coordinates of the image before and after rotation is determined by Equations (7~10), in which (7) and (8) are image rotation to the left and similarly to the right are calculated by (9) and (10). Through coordinate transformation, the segmentation problem of batteries in a row is solved, and the accuracy of the battery image is significantly improved.

$$ {y}_l=y\cos \overline{\theta}+\left(w-x\right)\sin \overline{\theta} $$

(7)

$$ {y}_l=y\cos \overline{\theta}+\left(w-x\right)\sin \overline{\theta} $$

(8)

$$ {x}_r=x\cos \overline{\theta}+\left(h-y\right)\sin \overline{\theta} $$

(9)

$$ {y}_r=y\cos \overline{\theta}+x\sin \overline{\theta} $$

(10)

In these equations, x_l and x_r and y_l and y_r are the horizontal and vertical coordinates of the rotated image, respectively; x and y are the original horizontal and vertical coordinates of the image; h is the height of the image before rotation, and w is the width of the image after rotation.

4.4.2 Elaborate template matching for the underlying image

Traditional template matching can search similar target objects in larger images. Although it has good stability and simple operation, it is time-consuming and less accurate which cannot meet the needs of precision and speed of lithium battery positioning. We choose to find its rough location first and then narrow down the template search area to determine the exact location of the battery.

The rough positioning process requires that the search area is narrowed down as quickly as possible, which can be achieved by the following processes for coarse positioning: Firstly, the battery tilt angle is determined by the coordinate difference method. Secondly, the rotation angle of the picture is determined by the average of the inclination angle of all the batteries in the picture, which is usually different from the inclination angle of some certain batteries in the picture. Finally, after the initial matching of the top layer image of the pyramid image, the upper left coordinates of the position of each battery are obtained and map to the real positioning coordinate of the bottom image of pyramid image. We get closer to the optimal locating point through the above three steps. In order to determine the lithium battery position precisely, we need to find the coordinates which can accurately locate on the bottom image.

The improved template matching algorithm is a correlation coefficient algorithm based on the normalized product correlation algorithm. It is a classic template matching algorithm in OpenCV visual library, which can meet the normal image positioning requirements. The precise positioning of lithium battery is template matching on the bottom level of image pyramid, and the result of template matching is approached to the best by narrowing down the search area of the target image and adjusting the template tilt angle constantly. The specific matching process is shown in Figure 10.

• Step 1: Rotate $\overline{\uptheta}$ counterclockwise after the bottom image is grayed.

• Step 2: Get the coordinates of the best matching position from the template matching at the top level of image pyramid and map to the coordinates of (x₁, y₁), (x₂, y₂) ... (x_n, y_n) on the bottom level of image pyramid. According to the size of the template, the original image is divided into n search areas.

• Step 3: Complete template matching was performed for each lithium battery, and the optimal coordinates (x_{i _ max}, y_{i _ max}) and G_{i _ d} were recorded, and the optimal value was compared with G_max. If G_{i _ d} ≥ G_max, the optimal coordinate point was found; otherwise, the next step was carried out.

• Step 4: First, the inclination angle of the template has been adjusted in a large range and selected 20 inclination values from [-α, +α]. Template matching was carried out with the segmented image respectively to find the maximum matching degree in this interval. If the maximum position appears at both ends, continue to expand the search range along the direction where the maximum appears until the maximum is found and the corresponding inclination angle θ_{i _ max 1} is calculated.

• Step 5: Reduce the angle range of the image to [-α/10, +α/10] and search for the optimal matching point near the angle θ_{i _ max 1}, with the matching value G_(θ) as the ordinate and the inclination θ as the horizontal coordinates, the parabolic model was established by G_(θ) = aθ² + bθ + c, and the maximum point of the parabola, namely the maximum point of the matching degree, was calculated. The corresponding inclination angle θ_{i _ max 2} was taken as the tilt angles of the battery.

Fig. 10.

Precise matching flow chart for lithium battery.

The rough matching process which selected the template inclination variation range [−a, +a] for template matching obtained the matching value G_{i _ max 1} corresponding to each inclination angle. Then the best match has been found and might appear as shown in Figure 11. If the maximum appears in the interval, the inclination corresponding to the maximum value would be selected as the exact matching initial template inclination angle. If it appears on the edges of the interval, the search interval should be enlarged. For example, if the maximum value appears at −a, it should be necessary to search for the maximum value in the left scope [−3a, −a]. In the meanwhile, if the maximum value appears at +a, the range [+a, +3a] would be searched next. Comparing the maximum of the new search range with the matching value in [−a, +a], if the maximum is still not found, search range [(2k − 1)a, (2k + 1)a] for next step, where k =  ± 2, ± 3, ± 4…, until the optimal matching degree is found and the corresponding inclination angle is determined.

Fig. 11.

The maximum value appears in the position diagram. a Maximum is within the interval. b Maximum is at the right endpoint. c Maximum is at the left endpoint

5 Experiments and analysis

5.1 Experimental setup

The effectiveness of the improved algorithm was verified by the image matching experiment of lithium battery on the designed experimental platform (Figure 12). The framework is made of aluminum alloy plate, and the bottom plate is made of 5 aluminum profiles with the size of 20mm×80mm×800mm, and the camera bracket is made of SBR16 cylindrical linear slide track. The battery is the LR 18650SZ power packed lithium battery produced by LISHEN, which is placed on the working table and rolls down by gravity. The configuration of the laptop is Intel i5-8250U, 8G RAM, and 256G SSD. The camera is A5201M/CU150 USB3.0 with CMOS sensor whose frame rate is 150fps and resolution is 1920×1200. The camera is black and white whose lens is MH1220S with 12mm focal length manufactured by HUARUI technology. The light source is PGB-350-12 with two bar LED light sources combined.

Fig. 12.

Experiment platform and introduction to each parts.

The lithium battery is cylindrical, 65mm long, and 18mm in diameter. Ten batteries rolled down the ramp at a time, and the average speed was 60mm/s, with a camera capturing images at a rate of 20 frames per second and an exposure gain of 2000. Considering the distance from lens to ramp, the maximum battery quantity that could be captured in one single view was 10, which can make sure the barcode on the lithium battery’s surface capturable. In each experiment, about 30 images can be collected, and an improved template matching algorithm was adopted to process each image and locate the position of each battery.

The upper machine program is coded by Python3 with PyQt5, OpenCV3, and several other third-party packages and shown in Figure 13. The main window contains two sections, real-time image acquisition in the left part and post-processing in the right part. For the image acquisition part, the preview window is used to preview the acquisition process which displays each frame captured by industrial camera in real time. When the program is running, the upper machine program is initialed and invokes industrial camera to capture frame which would be displayed in preview window. In less than half second delay, the information would be printed in bottom part which includes battery positioning status, time consumed, number of batteries, and recognition rate. Since this paper is focused on positioning, the barcode recognition will not be discussed in details.

Fig. 13.

The main windows of upper machine program, which displays almost vertical batteries by coincidence in preview window and leads to 100% precision.

5.2 Visual positioning by improved template matching

5.2.1 Positioning stability

During image collection, the lithium battery gradually entered and left the collection area, which made the number of lithium batteries inconstant. According to the number and distribution of batteries in the picture, the picture was divided into three categories: battery distribution on the left side, right side, and in the middle. Before the introduction of localization experiment, the comparison results of with and without image pyramid down-sampling is shown in Table 1. Each set contained 100 frames and repeated 10 times. The set 1 was the original frame captured from industrial PC and template of the whole separated battery. The sets 2 to 4 were down-sampled images once, twice, and three times, respectively. The time consuming was almost decreasing at 4 times in sequence since the pixel processed was reducing dramatically. In the meanwhile, the accuracy rate kept 100%. The lithium battery localization experiment was carried out on the images of three battery distribution types to verify the stability of the matching algorithm. Ten pictures for each have been selected, and the number of batteries in the pictures may be different.

Table 1. Matching rate under different battery distribution.

Figure 14 shows the battery matching results for three typical distributions. There are lithium batteries near the right, near the left, and on both sides. Batteries close to the right mean that there is no battery in the left area of the battery. Similarly, the right area is the same. Both left and right areas were non-empty when the lithium battery filled the entire image, left and right borders, and the middle of the image. The improved template matching algorithm can accurately locate the lithium battery and complete the positioning for the battery in the boundary and incomplete shape.

Fig. 14.

Stability of the multi-objective matching algorithm battery distribution: close to the right side (1~3), both the left and right sides (4~6), and the left side (7~9).

Table 2 shows the matching rate of boundary and non-boundary batteries. The matching rate of non-boundary lithium battery is 100%, which is much higher than the matching rate of left and right boundary. First, the main reason for the low matching rate of lithium battery near the boundary is that the improved template cannot match lithium battery due to the incomplete battery characteristics near the boundary. Second, compared with the intermediate lithium battery, the light intensity for lithium battery near the boundary is relatively weak. So there was deviation in the image correction, which will cause the displacement of the matching optimal point and lead to the positioning deviation. However, it does not affect the sequence of the entire batch of batteries. The batteries on the edge can be abandoned, because they would definitely appear in the middle of certain image.

Table 2. Matching rate under different battery distribution.

5.2.2 Practicability

The battery tilt angle is determined by the following steps. First rotate the original image and the single lithium battery image in sequence. Then rotate the template to match the lithium battery. An image with multiple lithium batteries has been selected randomly to verify the practicability of the algorithm.

Image binarization and morphological operation are performed as preprocessing. Then the approximate inclination angle of each battery is calculated by using multi-line detection, and the battery in the figure is segmented and rotated. After the image is sampled, the specified template is matched with the initial lithium battery contour, and the image is segmented again. Finally, template matching is performed twice on the full-size lithium battery. For the first time of matching, large interval rotation angle is selected, and for the second time, small interval rotation angle matching is selected near the optimal rotation angle.

Figure 15a and b show the curve between rotation angle and matching value of the matching template of no. 5 battery. In Figure 15a, the rotation range of the template is within [−2.0^°, +2.0^°], with an interval of 0.40°. The matching value of the template reaches maximum with rotation angle of 0°, and the best matching value can be determined within the interval [−0.40^°, +0.40^°]. In Figure 15b, target is rotated spaced at 0.04° between the determined sub-interval [−0.40^°, +0.40^°]. The quadratic polynomial is used to fit the discrete points shown by the red dotted line, and the rotation angle corresponding to the maximum is calculated to be 0.246°.

Fig. 15.

Rotary battery template matching analysis: (a) large angle rotation; (b) small angle rotation; (c) relationship between the battery position and the rotation angle; (d) improved matching accuracy

Pinpoint each lithium battery in this image. Figure 15c shows the relationship between the battery pack’s rough or precise position and the rotation angle. Based on the rough rotation angle, the rotation template is precisely determined. Figure 15d shows the improved matching accuracy of each lithium battery after adjusting the template angle, where the blue line represents the template matching result after sampling. The matching values of nos. 3, 4, and 5 lithium batteries are quite different from the matching rate of the last two times. The reasons can be concluded as follows: This specified matching template selected the whole lithium battery; or the template was a lithium battery without bar code, while the target image had a bar code.

The best matching rates of no. 1 and no. 7 lithium batteries are 0.916 and 0.891, respectively, which are relatively low compared with others, because they are at the edge of the image and influenced by camera distortion. The best matching rates of other lithium batteries are all above 0.95.

For massive production in lithium battery industry, each battery is needed for quality inspection. If it is detected by human worker, overall efficiency cannot be guaranteed. So the semi-finished batteries are inspected preliminarily during transfer to next station. Since the advantage of the cylindrical shape, taking picture while rolling is the most efficient way to conduct positioning and detecting. But a problem remains. Because batteries are rolling down from the slope with no interference, the battery inclination angle can be any possible value in range of −10° and 10° roughly. If the absolute value of inclination angle is larger than 10°, the batteries would roll out of the edge from the slope. To validate the stability of the proposed algorithm in large inclination angle, 10 sets of randomized repeat experiments are carried out. In consider of recognition rate varying with the battery inclination angle, each set of the experiments is with the same parameters. Because of the randomness, the inclination angle in each experiment which being named as set 1, set 2, …, and set 10, respectively, is set to different range, such as [−10, −8], [−8, −6], …, [8, 10].

As is shown in Table 3 and Figure 16, the average recognition rate of sets 3 to 8 is 97.83%, and the average time consumption is 1.044 s in which 100 batteries are recognized, so the recognition speed is 95.79 per second. The best recognition rate is 98.67% in set 4, and the corresponding inclination angle is between −4° and −2°, which means that there are 3.99 batteries missed in 300 batteries. The inclination angle of rolling batteries, −6° to 6°, is relatively large and barely reached in industrial production, combined with the fact that the time consumption curve is flat and smooth, all the evidences proves that the proposed algorithm is efficient and stable.

Table 3. Matching rate and time consumption under different battery inclination angle.

Fig. 16.

The curve of recognition rate and time consumption; the green dot curve has obvious peak; the blue square curve is relatively smooth.

5.2.3 Positioning results

The barcode of lithium battery in batch has been tested and located each battery while sorting each lithium battery in the group. Numbering each battery is based on the position of the lithium battery in the image. The localization rate of non-edge lithium battery can be reached to 100% by the improved template matching algorithm, in the meanwhile the localization rate of edge lithium battery image to 93.5%.

The improved template matching algorithm was used to locate lithium batteries and obtain the coordinates and inclination angles of the upper left corner for every battery. A group of images from multiple experimental data were selected for algorithm verification. Figure 17a shows the positioning results of lithium batteries, in which all the lithium batteries in the middle of the image were located and some batteries in the edge of the image are failed to be located. Since the battery rolled in the horizontal direction (x direction), the pixel changes in the y direction are relatively small and the battery inclination angle is almost similar. Therefore, the position coordinates in the x direction are selected to analyze the movement of the battery.

Fig. 17.

Distribution of each lithium battery. a The positioning result of a set of images. b The battery distribution is located.

Figure 17 (b) shows the coordinate changes of lithium batteries in x direction for several images. The dotted yellow line in Figure 17b indicated that all ten lithium batteries can be seen and corresponding coordinates can be found in Figure 17. For the point near the line battery no. 1, it can be seen that battery no. 1 is about to leave the field of camera view. The distribution curve of each lithium battery in Figure 17b is relatively smooth, and the slope of the battery curve increases successively from no. 1 to no. 10 batteries. It can be seen that the number of batteries in image field decreases after dotted yellow line, which is consistent with the process of the batteries accelerating rolling out. Each x-coordinate when the corresponding battery left the view field is less than the width of the matching template, which indicates that the batteries were sorted correctly.

5.2.4 Positioning precision and speed

To validate the performance of the proposed algorithm in this paper, two typical template matching algorithms, the improved SAD and NCC template matching algorithm, are studied. The improved sum of absolute differences (SAD) matching algorithm is designed to measure the similarity between images in pixel-level. The principle is computing absolute difference between each pixel in the original and template images and the differences are summed as a simple metric. The matching is ideal if the summed value is zero. The NCC matching template is implemented in OpenCV which is similar with the proposed algorithm, but the different matching template, a whole battery without barcode, is applied. The inclination angle of lithium batteries is controlled from −10° to 10°, and the experiment with 250 batteries is conducted every two degree, repeated for four times, which means 1000 lithium batteries are detected in every inclination angle interval.

As shown in Figure 18a, the proposed algorithm and improved SAD algorithm can detect lithium batteries at arbitrary angle in [−10°, 10°], but the traditional algorithm with relatively simple matching template shows very poor performance. In combination with Table 4, the positioning speed is decreasing slightly as the precision rate is increasing. The reason for this is that the high computational load for large inclination angle correcting and more images processing need more time. The traditional algorithm can detect lithium battery at small inclination angle and recognition rate is poor and unstable, which cannot meet the requirements of practical application.

Fig. 18.

a The recognition rate curves of the proposed and two other algorithms; red dot line represents the recognition rate of the proposed algorithm; purple triangle line, green square line, and blue triangle line represent the YOLOv3, the improved SAD, and traditional template matching algorithm, respectively. b and c The error comparison between the proposed and the improved SAD algorithm

Table 4. The comparison of different algorithms in positioning precision and speed

Specifically, the comparison of the four different algorithms shows that the YOLOv3 algorithm achieved the highest speed but not enough precision. In Figure 19a, the performance of YOLOv3 was great, but was not ideal in Figure 19b and c. The YOLOv3 method did not have the ability to correct the inclined batteries; hence, it cannot meet the requirements of subsequent defect detection.

Fig. 19.

The recognition results of YOLOv3 algorithm. a The inclination angle of batteries is zero. b The inclination angle is negative. c The inclination angle is positive.

In consideration of the similar performance between the proposed algorithm and the improved SAD algorithm, the gray histogram analysis method was used to analyze lithium battery images and verify a certain lithium battery for many times until the error of the battery size was within 1 pixel. With 60 lithium batteries measured for three times and mean value of all data acquired was calculated, the battery size was 224 pixels in width and 816 pixels in height, which were taken as the standard battery size. The experimental results in Figure 19b and c and Table 5 were averaged by 20 sets and each set with 10 batteries for five times repeated.

Table 5. Positioning comparison in different directions (in pixel level).

Figure 19b and c show the comparison between the proposed algorithm and the improved SAD algorithm in the center position error of lithium batteries, namely radial error and axial error. The points acquired by the proposed template matching algorithm and the SAD algorithm were all distributed near the center line 0. The error in most positions of the former is within 1 pixel and distributed evenly, in the mean while the latter was scattered. The points with large errors were mainly distributed in the image edge, because of the limitation of camera view and the information loss of lithium battery image near the edge.

Table 5 compares the matching results of the proposed and the improved SAD algorithm. The variance of the proposed algorithm in the width and length direction of lithium battery is relatively smaller than that of the SAD algorithm. The maximum error of the proposed algorithm for image edge is also better than that of the SAD algorithm with the maximum error of 2.643 pixels in the width direction and 2.297 pixels in the length direction, which accounted for 1.532% and 0.281% of the battery size. The result of proposed method meets the requirements of subsequent defect detection.

6 Conclusions and future works

A multi-template matching algorithm for lithium battery visual localization based on morphological operation and linear inclination detection is proposed. The experimental results show that the proposed algorithm can match almost all targets in the image, and the recognition rate of matching non-edge targets is 100%. Compared with the other state-of-the-art object detection algorithms, the modified battery matching algorithm was high-performance and robust. The tilt angle of the battery can be calculated to one thousandth accuracy, and the precise positioning of lithium battery was realized. A group of pictures were selected for the experiment, and the lithium battery in the figure could be detected in initial order of lithium batteries.

Since each lithium battery is positioned precisely, we believe that detecting and recognizing the barcode printed on the surface and lithium battery defect detection should be challenging but successful in the future work. The proposed algorithm is suitable for local image invariant objects. With the several local images of target object, the proposed dual-template or multi-template matching algorithm and tilt correction method can be applied. Hence, for the similar products, such as rectangular or square products, the proposed algorithm with a few specific changes would still work. So, the future works should focus on the improvement of robust and precise proposed algorithm to expand the application fields.