Automatic Processing of Microhardness Images Using Computer Vision Methods

Abstract—

Many algorithms that are capable of determining the indent area in automatic processing of optical images of imprints have found application in hardness measurements using the Vickers method. A robust interactive algorithm based on the maximum separation of color image components in the indentation area and an unstrained surface is described. The efficiency and stability of the presented method are evaluated using indents on several materials with different morphology and deformation character.

INTRODUCTION

The classical hardness test of materials using the Vickers and Knoop methods involves determination of the indent area by the diagonals measured on an optical image. Although measurements can be carried out by an operator, nevertheless, for several decades, manufacturers of equipment offer microhardness testers with an automatic function for recognizing the boundaries and diagonals of an imprint in accordance with the GOST 6507-1-2007 Russian state standard and the ISO 6507-1:2005 international standard [1, 2]. These methods show the highest efficiency on metals with indents that have a pronounced square rhombic shape. However, a variety of effects that violate the ideal indent shape (piles, elastic recovery, and features of the structure and relief) result in deterioration of the quality of automatic processing and algorithmic failures; therefore, the principles of image processing are still a subject of research [3].

The most common approaches are based on thresholding, the Hough transform, and the Canny edge detector [3, 4]. These approaches exhibit high accuracy on high-contrast indentations with a weakly structured background and luminance surface structure. Algorithms based on these approaches specifically respond to various defects and produce erroneous data. Thus, thresholding is ineffective on low-contrast images, while the Hough transform and the Canny edge detector are less dependent on the contrast of samples and the absolute values ​​of brightness, but are sensitive to artifacts of the indent shape and the brightness relief of the background.

THE MORPHOLOGY OF RESIDUAL INDENTS

A square is the simplest and most expected form of a residual indent when measuring Vickers hardness, but this ideal situation is quite rare. As follows from [5], pile-up or sink-in can be observed around the indent depending on the ratio of the elastic modulus to the yield point. The image may contain extended defects or relief elements, which also affect the quality of determining the dimensions of the indent. In addition, in some cases, the brightness of the inner surface of the indent can be comparable to a flat surface, which reduces the contrast and also makes the indent edges hard to distinguish.

A black square against a bright surface is the simplest and most convenient image of a Vickers residual indent, which is quite typical for impressions on metals. The conventional thresholding can be used for such images, after which the indent vertices are localized on a marked image using simple and efficient algorithms based on finding the points that are the farthest from the center of the segmented area [6]. Thresholding allows marking indents with curved edges; at the same time, it is very sensitive to the selection of the binarization threshold (Fig. 1), which creates difficulties for automatic marking of images with illumination gradient and defects.

Fig. 1.

(a) An initial image and (b–f) results of the thresholding at different threshold values.

The problem of selecting the optimal threshold for binarization is an object of relevant studies. Effective algorithms based both on the analysis of brightness histograms and on the minimization of energy functions have been proposed [7–9]. In addition, methods have been developed that allow construction of surfaces of threshold values [10]. These methods provide high-accuracy segmentation of images with uniform and uneven brightness, but are incapable of separating the imprint and relief defects.

Another approach relies on edge detection and subsequent Hough transform [11, 12]. Edge detection can be complicated by a low contrast and by noise that is present in images. In addition, the sheer number of false candidates in the form of straight sections of the image (e.g., machining marks) makes it difficult to find the four small areas that are actually the edges of the print. Algorithms based on the Hough transform can be conditionally divided into two groups.

The first group of algorithms works by searching the image for sections of straight lines that are the faces of an imprint [11]. The vertex coordinates are calculated as the intersection points of these lines. These algorithms localize edges well and provide high accuracy on imprints with straight edges. At the same time, depending on the ratio of the elastic modulus to the yield point, pile-up or sink-in can be observed around the indent, as follows from [5]. The accuracy of the algorithms is significantly reduced on indents with convex edges. An example of this situation is illustrated in Fig. 2.

Fig. 2.

(a) Results of applying the Hough transform to mark the indentation area and (b) an enlarged area in the vicinity of one of the corners (marked with a rectangle in Fig. 2a).

The second group of algorithms uses the Hough transform of the neighborhood of the indent vertices [12]. This approach can significantly increase the accuracy on imprints with convex or concave edges by analyzing segments of straight lines lying in close proximity to the vertex. This family of algorithms is suitable only for specifying the coordinates of the indent vertices and is sensitive to image artifacts that are located directly at an imprint vertex. This situation is illustrated in Fig. 3.

Fig. 3.

The marking of the indentation area using the Hough transform in the region of ​​the corner with a defect.

DESCRIPTION OF THE ALGORITHM

The algorithm proposed in this paper is based on comparison of two-dimensional diagrams of the optical properties of the outer and inner parts of an imprint on the obtained image. It is convenient to show the operation principle of the algorithm by working with only one color characteristic in the case of a one-dimensional image with the subsequent generalization for the case of a color image.

A priori information about the indent is used, primarily the position of its center, to determine the position, size, and orientation of an imprint. After direct indenting by pressing the tip into the surface, the device moves the sample from under the indenter under the optical microscope. The distance between these two modules was preliminarily calibrated in such a way that the indent center lies at the center of the frame with an accuracy of 2–5 µm.

A number of circles with radii in the range from 5 μm to 90% of the frame height are constructed from the frame center. In the one-dimensional case under consideration, the Hue from the HSV (Hue, Saturation, Value) color scheme is calculated for each of the image pixels. Two histograms ${\mathbf{A}}_{{{\text{Hue}}\;~}}^{{{\text{in}}}}$ and ${\mathbf{A}}_{{{\text{Hue}}}}^{{{\text{out}}}}$ of the Hue distributions are constructed for each of the selected radii, i.e., the histograms for pixels that lie both inside and outside the circle. Traditionally, the Hue parameter takes 256 values, which determines the size of the vectors Aⁱⁿ and A^out; however, the use of 256 values ​​is redundant in practical cases. It significantly increases computational costs and memory consumption for multidimensional histograms. It also makes the histogram sparse and prevents pixels with slightly different colors in the outer and inner regions from affecting the value of the split function. A histogram with 25 intervals is used in our algorithm.

For the best separation of the indent regions and the rest of the sample surface, the peaks of the histograms are spaced at the maximum distance. The normalized scalar product of vectors is used as a characteristic of the quality of peak separation:

$$f({{{\mathbf{A}}}^{{{\text{in}}}}},{{{\mathbf{A}}}^{{{\text{out}}}}}) = ~\sqrt {\frac{{{{{({{{\mathbf{A}}}^{{{\text{in}}}}},{{{\mathbf{A}}}^{{{\text{out}}}}})}}^{2}}}}{{({{{\mathbf{A}}}^{{{\text{in}}}}},{{{\mathbf{A}}}^{{{\text{in}}}}})({{{\mathbf{A}}}^{{{\text{out}}}}},{{{\mathbf{A}}}^{{{\text{out}}}}})}}} .$$

The minimum value of this product provides the best separation by the Aⁱⁿ and A^out vectors.

A circle was selected as the initial form that approximates the indentation area. An example of the approximation of a model rhombic imprint at different radii of the circle is shown in Fig. 4. This form is used because the exact a priori position of the indent is unknown; however, the two diagonals of the imprint are located in horizontal and vertical planes with an accuracy of 5°. For a certain value of the radius R, the normalized scalar product is minimal (see Fig. 4).

Fig. 4.

The approximation of the model rhombus by circles with different diameters (top) and the form of the vectors Aⁱⁿ and A^out for each case (bottom). The value of the normalized scalar product  f of the vector components is presented above each graph of their distribution.

The next step in the implementation of the algorithm is the approximation of an imprint on an optical image with a rhombus whose diagonals are located in horizontal and vertical planes; the initial length of the diagonals d is obtained from the relation d = 2.8R. The coefficient of 2.8 was selected by numerical approximation with the circles of the model imprint. A number of rhombuses (model imprints) were generated, whose diagonal lengths d_i were given as a percentage of the frame height. A circle R_i was selected for each rhombus so that it minimized the scalar product f(Aⁱⁿ, A^out). The d_i/R_i ratio for indents with various sizes is shown in Fig. 5.

Fig. 5.

The dependence of the ratio of the diagonal length d_i  to the radius R of the approximating circle on the rhombus diagonal length is expressed as a percentage of the frame size H.

As can be seen from the figure, the ratio of the diagonal length to the optimal diameter complies with the value of $2\sqrt 2$with a high degree of accuracy at a small indentation size. As the indentation size increases, this ratio changes only slightly, by 5% or less. Thus, the selected coefficient can be considered a good approximation for indentations with arbitrary sizes.

At the next stage, the shape of the approximating rhombus is changed. This procedure is also performed by minimizing the function f(Aⁱⁿ, A^out), which is defined by the Hue of the pixels that lie both inside and outside the rhombus, and proceeds as follows. One of the rhombus vertices is selected and is then moved in the vertical and horizontal directions. The function  f(Aⁱⁿ, A^out) is recalculated after each step and subsequent movement is in the direction of a decrease in this function. The amplitude of each subsequent shift is random and varies in the range from 3 pixels to 3% of the frame size; this approach allows one to find the global minimum of the described function. The random step allows the algorithm to avoid getting stuck at local minima. The rhombus vertex is set at the corresponding point and the remaining three vertices are traversed. The rhombus vertices are iterated over cyclically until their position (determined with an accuracy of one pixel) stops changing during five complete traversals of all vertices.

Let us demonstrate the above approach in the simplest one-dimensional case, within which the operation of the algorithm will be visualized for one of the indent angles. At this stage, the histograms formed by the Aⁱⁿ and A^out vectors are plotted based on the values ​​of the HSV color model. For the considered indent angle, the rhombus approximating it degenerates into a triangle, whose two vertices are fixed at the lower corners of the image and the position of the third vertex (X₃, Y₃) is to be determined. By changing the position of the third vertex, one can obtain an array of f(Aⁱⁿ, A^out) values according to the above formula ​​for each pair of coordinates (X₃, Y₃) (Fig. 6). The position of the minimum of this function corresponds to the position of the third triangle vertex (Fig. 6a).

Fig. 6.

(a) An optical image of the surface with a mask corresponding to a certain indent area; (b) the histograms of the Value (HSV) distribution ​​for the outer and inner indent regions; and (c) the distribution of the optimized function  f(Aⁱⁿ, A^out) vs. the position of the third vertex.

After the imprint is approximated by a rhombus, the imprint shape is approximated by a rhombus with curved edges, which allows one to take the deviation of the actual indent shape from the rhombic one into account. To do this, each edge is approximated by a four-link broken line. The edges are divided into four parts in a ratio of 10 : 40 : 40 : 10. The coordinates of the broken-line vertices are determined by vector relations:

$${\mathbf{OA}} = {\mathbf{a}}(1 + K);~$$

$${\mathbf{OB}} = {\mathbf{b}}(1 + K);~$$

$${\mathbf{OA}}{{'}} = 0.9{\mathbf{a}} + ~~0.1{\mathbf{b}};$$

$${\mathbf{OB}}{{'}} = ~~0.1{\mathbf{a}} + ~~0.9{\mathbf{b}};~$$

$${\mathbf{OO}}{{'}} = ~(~0.5{\mathbf{a}} + 0.5{\mathbf{b}})(1~-~K),$$

where a and b are vectors directed from the center of the original rhombus towards its vertices (they coincide with the semi-diagonals); the coefficient K reflects the degree of the convexity of the indent.

The change in the shape of the rhombus sides for different values ​​of the coefficient K is shown in Fig. 7.

Fig. 7.

The change in the rhombus side at different values ​​of the convexity coefficient K.

The algorithm is initialized at a zero coefficient K. The K is then changed by +0.0025 and –0.0025 and a new search for the vertices of the rhombus with modified sides is performed after each change. Determination of new coordinates of the vertices, as well as the selection of the coefficient K is also carried out by minimizing the function f(Aⁱⁿ, A^out). The algorithm terminates after two complete iterations do not change the coordinates of the vertices and curvature. This stage of the algorithm is final.

As mentioned above, the algorithm implemented in the software uses two-dimensional histograms: distributions are plotted not only using the Hue channel (tone), but also the Value channel (brightness). The arrays $A_{{i,j}}^{{{\text{in}}}}$ and $A_{{i,j}}^{{{\text{out}}}}$ are used instead of vectors Aⁱⁿ and A^out, and the normalized scalar product is replaced with its analogue, i.e., the function

$$f({{A}^{{{\text{in}}}}},{{A}^{{{\text{out}}}}}) = \sqrt {\frac{{{{{(\Sigma A_{{i,j}}^{{{\text{in}}}}~A_{{i,j}}^{{{\text{out}}}})}}^{2}}}}{{\Sigma A_{{i,j}}^{{{\text{in}}}}A_{{i,j}}^{{{\text{in}}}}~\sum A_{{i,j}}^{{{\text{out}}}}A_{{i,j}}^{{{\text{out}}}}}}} .$$

Similarly to the one-dimensional case (when a single color channel is used), the function f of the two-dimensional case is applied initially in the selection of the circle radius and then in the setting of the rhombus boundaries.

THE RESULTS OF THE INDENT MARKING

The quality of the algorithm was estimated by using a reference steel block of MTB hardness (HV5 446 hardness) and a measure of MV 010 microhardness, which had the following hardness values ​​prescribed by verification: HV0.2 858. They were measured using an experimental NanoScan-HV microhardness tester and the above algorithm and by manually setting of indent boundaries. The hardness measurements were also performed using a DuraScan 20 microhardness meter with manually set indent edges. The measured and passport values are shown in Table 1. The half-width of the 95% confidence interval based on the Student’s t-distribution is indicated as the inaccuracy of the measured values. Ten measurements were taken for each block.

Table 1. The test results for the hardness blocks

The presented results demonstrate a good agreement between the lengths of the indent diagonals obtained using the manual and automatic methods. In three out of the four cases, the average hardness value measured according to the algorithm fell within the confidence interval of the hardness measurements with the manually marked sample. In the case of measurements of the MTB block with an HV2 load, the 95% confidence intervals intersected only at their extreme values. However, it should be noted that the difference between the average values ​​of the measured hardness was only 1%.

An example of the automatically marked boundaries of the indent diagonals in the cases of 0.2- and 1-kg loads is shown in Fig. 8. These images demonstrate the efficiency of the algorithm for an indent with concave edges, as well as in the presence of a surface relief defect.

Fig. 8.

The boundaries of the indent diagonals, which were determined using the automatic algorithm: (a) MV010 block and a load of 0.2 kg; and (b) MTV block and a load of 1 kg.

CONCLUSIONS

The described algorithm for determining the indent area has demonstrated its efficiency in images with low contrast and false objects in the field of view of the microscope. This approach provides the wide coverage of indents with various shapes and a high accuracy in determining the area of an imprint with a quasi-diamond shape. The typical value of random scatter and systematic deviation of the data obtained using the algorithm was approximately 1% for all materials presented in the study with an indentation size of more than 5 μm.