Quantitative Stereovision in a Scanning Electron Microscope

Abstract

Accurate, 3D full-field measurements at the micron-level are of interest in a wide range of applications, including both facilitation of mechanical experiments at reduced length scales and accurate profiling of specimen surfaces. Scanning electron microscope systems (SEMs) are a natural platform for acquiring high magnification images for stereo-reconstruction. In this work, an integrated methodology for accurate three-dimensional metric reconstruction and deformation measurements using single column SEM imaging systems is described. In these studies, the specimen stage is rotated in order to obtain stereo views of the specimen as it undergoes mechanical or thermal loading. Simulations and preliminary experimental studies at 300× demonstrate that (a) spatially-varying image distortions can be removed from images using a non-parametric distortion model, (b) the system can be reliably calibrated using distortion-corrected images of a planar object and grid at various orientations and (c) specimen rotation variability during the measurement phase can be controlled so that baseline strain errors are within the range of ±150 µε. Benchmark rigid body motion experiments using calibrated SEM views demonstrate that all components of strain in the reconstructed object have a mean value around O(10⁻⁴) and a random spatial distribution with standard deviation ≈ 300 micro-strain.

Introduction

Deformation and profile measurements continue to be important issues for mechanical testing at micro/nano scales [1–3]. At these scales, Scanning Electron Microscope (SEM) images have been combined with modern non-contacting measurement methods such as Digital Image Correlation (DIC) to make accurate 2D measurements on nominally planar specimens [4–6]. As the experiments at smaller scales become increasingly sophisticated and material behavior in the vicinity of 3D micro-scale features is required, the need for three-dimensional shape and full field strain and deformation measurements has expanded. One discipline where such measurements are of interest is biomedical engineering [7–11].

The ability to obtain accurate 3D measurements using single column SEM images presents several challenges, including (a) three-dimensional metric reconstruction using single column SEM images, (b) correction of image distortions, (c) development of an effective procedure for acquiring images from multiple views, (d) calibration of the stereo-image views and (e) appropriate theoretical description for stereo-reconstruction over a wide range of applications. Three-dimensional profiling using one imaging device and multiple images acquired from different orientations is a classic problem in photogrammetry for which there are multiple applications using SEM images [12–27]. In general, such applications have three shortcomings when considering extension for deformation measurements. First, since a typical photogrammetric reconstruction is scale-free [28], the approach is inappropriate for deformation or profile measurements. Secondly, the accuracy of the reconstruction is significantly degraded by disregarding the distortion associated with SEM images. For example, when using DIC to extract profile measurements, Lockwood et al. [27] obtained results assuming a reduced form for the imaging equations and negligible distortions. In fact, it was only with the publication of a series of recent articles [4, 12, 29, 30] that the effects of spatially-varying and temporarily-varying distortions in SEM images were identified as being important, and then quantified at magnifications varying from 200× to 20,000× [4, 29, 30]. Finally, the operations necessary to obtain and analyze several SEM images to profile a specimen are tedious, especially if multiple profiles are needed over time.

In this paper, a method for obtaining three-dimensional deformations using a sequence of SEM images acquired during out-of-plane specimen rotation is described. Vision system parameters for each view are separated for effective calibration through independent procedures. The non-parametric spatial distortion model [31] is employed to remove spatial distortions from each image view using in-plane motions of a planar target. All SEM images used for calibration and deformation measurements are converted into distortion-free SEM images by removing the spatial distortions. To demonstrate the methodology, a benchmark experiment is performed to reconstruct the three-dimensional motions and deformations for a planar specimen undergoing three separate 3D rigid body motions. Residual strain fields computed from the reconstructed motions are presented which demonstrate the accuracy of the SEM stereovision measurement system.

Imaging in an SEM

The imaging process in an SEM system is fundamentally different from an optical system. Whereas an optical imaging system employs lenses to collect and focus light waves, a modern SEM system scans an e-beam across the specimen surface and collects back scattered (or secondary) electrons using a fixed detector to construct an image.

A simplified schematic of the imaging process for an SEM is shown in Fig. 1. As the electron beam scans across the surface, backscattered electrons (BSEs) ejected from the interaction volume are collected by a stationary BSE detector (generally centrally located near the e-beam) for each “point” in the scan area. Since the electron scan and the BSE location do not change as the specimen is translated out-of-plane, the resulting image of the translated specimen will be for-shortened, as shown in Fig. 1 for a specific pair of feature points.

Fig. 1

Perspective effect in SEM imaging process

Since optical images will also be for-shortened when the specimen is translated out-of-plane (or tilted), an established optical imaging model (pinhole model) has been shown to be appropriate to describe the perspective effect for our relatively low magnification SEM studies. A mathematical description for the pinhole model given in equation (1) [31–33] that relates two-dimensional coordinates in an SEM image to the coordinates of the corresponding 3D world coordinates of a feature point on the specimen can be written;

$$ \rho \cdot {\mathbf{m}} = {\left[ {\mathbf{A}} \right]_{3 \times 3}} \cdot {\left[ {R\quad t} \right]_{3 \times 4}} \cdot {\mathbf{M}} $$

(1)

where

$$ \begin{gathered} {\mathbf{m}} = \left\{ {\begin{array}{*{20}{c}} {{x_s}} \\ {{y_s}} \\ 1 \\ \end{array} } \right\};{\mathbf{M}} = \left\{ {\begin{array}{*{20}{c}} {{X_w}} \\ {{Y_w}} \\ {{Z_w}} \\ 1 \\ \end{array} } \right\};{\left[ {\mathbf{A}} \right]_{3 \times 3}} = \left[ {\begin{array}{*{20}{c}} {{f_x}} & {{f_s}} & {{C_x}} \\ 0 & {{f_y}} & {{C_y}} \\ 0 & 0 & 1 \\ \end{array} } \right]; \hfill \\ {\mathbf{t}} = \left\{ {\begin{array}{*{20}{c}} {{t_x}} \\ {{t_y}} \\ {{t_z}} \\ \end{array} } \right\};{\mathbf{R}} = \left[ {\begin{array}{*{20}{c}} {\cos \gamma } & { - \sin \gamma } & 0 \\ {\sin \gamma } & {\cos \gamma } & 0 \\ 0 & 0 & 1 \\ \end{array} } \right] \cdot \left[ {\begin{array}{*{20}{c}} {\cos \beta } & 0 & { - \sin \beta } \\ 0 & 1 & 0 \\ {\sin \beta } & 0 & {\cos \beta } \\ \end{array} } \right] \cdot \left[ {\begin{array}{*{20}{c}} 1 & 0 & 0 \\ 0 & {\cos \alpha } & { - \sin \alpha } \\ 0 & {\sin \alpha } & {\cos \alpha } \\ \end{array} } \right] \hfill \\ \end{gathered} $$

where ${\left\{ {{x_s}\;\;{y_s}} \right\}^T}$ represents two-dimensional sensor coordinates of a point in a digital SEM image in pixel units; ρ is a factor in metric units;¹ ${\left\{ {\begin{array}{*{20}{c}} {{X_w}} & {{Y_w}} & {{Z_w}} \\\end{array} } \right\}^T}$ represents the three-dimensional world coordinates of point. The matrix [A] contains the intrinsic camera parameters including f _x , f _y , f _s and the image center location, (C _x , C _y ) in pixels that describe the camera-to-image projection transformation. The matrix R is the rotation matrix relating the world and camera coordinate system orientation. The form shown in equation (1) is associated with Euler angles α, β and γ [34]. The vector t defines the world-to-camera system translation vector with components t_x, t_y and t_z, in metric units.

Single Camera Stereovision: Equivalency with Dual Camera

Quantitative 3D reconstruction can be performed using a single “camera,” even when there is no a priori knowledge of camera parameters. Using established photogrammetric principles [28], a general procedure for such situations is to acquire a sequence of images of an object using different “camera” orientations and positions. In such cases, it is well known that three-dimensional reconstruction using images from several unknown views [28] can be performed up to an unknown scale factor [28]. In such cases, it is common practice to place an object with known metric distance in each view, providing sufficient information to define the scale factor.

In a similar manner, photogrammetric approaches can be employed to develop an equivalent stereo-vision methodology for use with standard, single column SEM imaging systems. Figure 2 demonstrates the relationship between multi-camera stereovision systems and multi-view, single camera stereovision systems. In fact, the three stereovision systems shown in 2(a), 2(b) and 2(c) are equivalent. The mathematical model in equation (1) for the standard system in 2(a) can be applied to the other two systems directly, with differences in interpretation of the extrinsic parameters defined in the following section. By imposing constraints that require the two viewing angles (specimen rotations² in 2(c)) to be repeatable, the SEM can be considered as a Tilting Specimen Stereovision (TSSV) system. As will be demonstrated in the following sections, the operational constraint imposed by this requirement can be controlled experimentally so that the resulting strain errors are relatively small (more detail is provided in “Reconstruction Error Due to Rotation Angle Variations”). For the TSSV system, there is only one imaging device involved, so that only one set of intrinsic parameters is required for both views.

Fig. 2

Equivalent stereovision systems (a) Standard Stereovision (SSV, [31]) with two identical cameras imaging at positions symmetric about the specimen normal axis; (b) Moving Camera Stereovision (MCSV), position 1 and position 2 are the positions of Camera 1 and 2 in 1(a) respectively; (b.1) and (b.2) are individual representations for a camera in position 1 and 2, respectively; (c) Tilting Specimen Stereovision (TSSV), the images of the specimen in position 1 and 2 can be decoupled to give the same perspective as shown in (b.1) and (b.2)

Prior to presenting the formulation for the TSSV system, it must be emphasized that common terms in optical imaging (“view,” “camera,” “orientation of views” and “translation between views”) are still used throughout this article, though the terms have different meaning in SEM imaging. For example, “views” correspond to different specimen tilts in SEM imaging; “camera” corresponds to the combination of EBSD detector and digital image converter in the SEM; “orientation of views” corresponds to different specimen tilts, not different sensor array orientations; “translation between views” corresponds to an artificial motion (the true translation is actually near zero) required by the pinhole model to construct convergent specimen views.

Tilting Specimen Stereovision System

Consider two views of an object point, M _i , as shown schematically in Fig. 3. Letting the coordinate system for the first view be the world system, one can write equation (1) for each view as follows:

Fig. 3

Stereo model parameters. The angle between the two views is the sum of the individual rotations, +/− θ, relative to the normal and is designated α

$$ \left\{ \begin{gathered} {\rho_{1i}} \cdot {m_{1i}} = \left[ A \right] \cdot \left[ {I\;\;0} \right] \cdot {M_i} \hfill \\{\rho_{2i}} \cdot {m_{2i}} = \left[ A \right] \cdot \left[ {R\;\;t} \right] \cdot {M_i} \hfill \\\end{gathered} \right. $$

(2)

where ${{\mathbf{m}}_{1i}} = {\left\{ {\begin{array}{*{20}{c}} {{x_{{s_{1i}}}}} & {{y_{{s_{1i}}}}} & 1 \\\end{array} } \right\}^T}$; ${x_{{s_{1i}}}}$, ${y_{{s_{1i}}}}$ represent the coordinates of the i ^th point in the first SEM view in pixels; ${{\mathbf{m}}_{2i}} = {\left\{ {\begin{array}{*{20}{c}} {{x_{{s_{2i}}}}} & {{y_{{s_{2i}}}}} & 1 \\\end{array} } \right\}^T}$; ${x_{{s_{2i}}}}$, ${y_{{s_{2i}}}}$ represents the coordinates of the i ^th point in the second SEM view in pixels; ρ _1i and ρ _2i are two factors corresponding to each view; M _i represents the world coordinates of the i ^th point in metric units [see equation (1)]. As noted previously, in this description the world coordinate system is set to coincide with the coordinate system of the first view as shown in Fig. 3. Thus, [A] is the intrinsic parameter matrix for both the first and second views, where I is a 3 × 3 identity matrix and 0 is the 3 × 1 zero translation vector. The matrix [R t] contains the extrinsic parameters including [R ] the rotation matrix relating the first and second views and [t] the translation vector from the first view to the second view.

System Calibration and Distortion Removal

The calibration procedure for standard stereovision systems yields optimal parameters by fitting the stereovision model in equation (2) to the corresponding measurement locations in both views simultaneously; details are given in a series of recent publications [31, 32, 35]. Though the approach is appealing due to previous successes, direct adoption for TSSV system calibration will introduce significant error because the specimen rotation angle (α) normally considered constant during calibration will vary slightly during the experiment, making the calibration algorithm unstable. For the above reason, the investigators adopted a sequential approach for system calibration, as described in the following paragraphs.

Distortion correction

It has been shown in recent articles [4, 29, 30] that optimal accuracy in SEM-based image measurements using a pinhole model requires that SEM image distortions be considered and methods developed to quantify these distortions. According to Sutton et al. [4], there are two types of distortion embedded in an SEM image—temporally-varying (drift) and spatially-varying (spatial) distortion. When SEM images of a conducting specimen are taken at low magnification (less than 1,000X), temporally-varying distortion oftentimes can be neglected. For the experiment presented in “Experimental Validation in SEM,” drift distortion is neglected, though straight-forward application of previous drift correction methods is appropriate for use in the SEM stereo imaging at higher magnification.

Due to the complexity of spatially-varying distortion in SEM images (see Fig. 4), non-parametric distortion correction approaches [35] are employed. The procedure described in previous work employs images of a planar object subjected to a series of in-plane translations to identify the distortion field and remove it from subsequent images. For a single column SEM, a distortion correction function is determined by in-plane translation of a planar specimen that is nominally orthogonal to the e-beam central axis. Since the distortion field is assumed to be a function of only the pixel location in the image, the distortion corrections are applied to all images obtained from any orientation.

Fig. 4

Image distortion fields for Quanta 200 at 300× with imaging parameters given in “SEM Stereo Calibration”

Intrinsic parameters calibration

The calibration target (grid with dots at known spacing) shown in the top part of Fig. 5 is placed in the SEM and imaged as it undergoes several rotations and translations. The grid images after distortion correction are used in the calibration process. The standard bundle adjustment approach [31, 32, 36] with equation (1) is used to obtain all intrinsic parameters, extrinsic orientations and grid positions for each grid orientation and position. Only the intrinsic parameters characterizing the SEM image process are retained since the extrinsic parameters are not required for future 3D reconstruction.

Fig. 5

Grid and random pattern specimen. (Black is the etched region)

It is noted that the use of a planar calibration target with known dot spacing imaged at different orientations (two orientations at least) is sufficient to constrain the parameters for the vision system [31, 32]. Repeatability of the intrinsic parameter estimation process is verified by independent experiments conducted under the same imaging conditions (see Table 1).³ Though the image center locations (C _x , C _y ) have variations of O(10²), they are still acceptable considering the large focal lengths (f_x and f_y ≈ 25,000 ). Extrinsic orientation and position parameters in equation (1), which are actually part of the reconstruction of the dots in the world coordinate system, are not included in Table 1.

Table 1 Intrinsic parameters from three independent experiments, with spatially-varying distortion correction, Magnification = 300×, Working distance = 10 mm

Extrinsic parameters calibration

After completing the intrinsic parameter calibration process, the glass slide with the grid is moved slightly so that the dense random speckle pattern shown in Fig. 5 fills the field of view. Since the glass slide already is attached to a eucentric rotation stage, it is rotated out-of-plane by +θ and −θ for use in our experimental studies and SEM images are acquired from both orientations. Figure 6 presents a schematic of the two views, with variability in the angle α = 2θ clearly identified. Constraining [A] to be the same for both views, optimization of the following form with known [A] is used to determine the orientation, R and the position, t of the second view relative to the first view,

$$ r = \mathop {{\min }}\limits_{\left( {\left[ {\begin{array}{*{20}{c}} {\mathbf{R}} & {\mathbf{t}} \\\end{array} } \right],{{\mathbf{M}}_i}} \right)} \sum\limits_{i = 1}^n {\left( {{{\left( {{{\mathbf{m}}_{1i}} - \frac{1}{{{\rho_{1i}}}} \cdot \left[ {\mathbf{A}} \right] \cdot \left[ {I\;\;0} \right] \cdot {{\mathbf{M}}_i}} \right)}^2} + {{\left( {{{\mathbf{m}}_{2i}} - \frac{1}{{{\rho_{2i}}}} \cdot \left[ {\mathbf{A}} \right] \cdot \left[ {R\;\;t} \right] \cdot {{\mathbf{M}}_i}} \right)}^2}} \right)} $$

(3)

where n is the number of data points used for optimization. The optimal solution of [R t] is unique under the following 6 theoretical constraints:

Fig. 6

Schematic for stereo-imaging with variability in the included viewing angle. Grid rows and columns are parallel to the world coordinates, X_w and Y_w respectively, while Z_w is perpendicular to the planar grid

• i) 5 constraints from point correspondences

• ii) at least 1 constraint requiring a metric distance between feature points identifiable in both views.

See Appendix A for a brief description of the underlying theory.

Regarding the need for constraint ii), as shown in Fig. 3 the vector t defines the location of the pinhole in one stereo view relative to the other pinhole. The magnitude, |t|, is also known as the baseline distance, i.e. the distance joining the two optical centers. To remove ambiguity (see Appendix A) and uniquely define |t|, the common approach is to identify at least two feature points in both views with known metric distance between them. Using the calibration parameters and the known pixel positions of the end points, an equation can be written that provides an additional constraint. By solving this equation (or several equations using optimization if multiple pairs of feature points with known metric distance are used), the vector t is uniquely defined.

Using both equation (3) and constraint ii), Table 2 shows the extrinsic camera parameters for Exp 1, Exp 2 and Exp 3. As noted in equation (1), α, β, γ are Euler angles and t _x , t _y , t _z are the translations in the x-, y- and z-directions in the rotated camera 1 coordinate system.

Table 2 Extrinsic parameters from three independent experiments, with spatially-varying distortion correction, Magnification = 300×, Working distance = 10 mm

To obtain these values, all images of the object were distortion-corrected prior to performing the optimization process.⁴ Point correspondence input into equation (3) is obtained by feature registration algorithms such as DIC, where the number of correspondences can reach the order of 10⁴. The large number of point correspondences improves the stability of the optimization process, though at least five point correspondences are required.

Reconstruction Error Due to Rotation Angle Variations

Slight variations in the total specimen rotation angle, which corresponds to the pan angle α in equation (2) (see Figs. 2 and 6), will introduce operational errors in the deformation measurements due to inaccuracies in the calibration relationship defined for the two views and used for image reconstruction. These errors will be in addition to reconstruction errors introduced during subset matching employed in 3D DIC [37–39]. In the following section, relative operational rotation error (Δα) is defined by the following equation:

$$ \Delta \alpha = \left[ {\left( {\alpha_c^{+} - \alpha_c^{-} } \right) - \left( {\alpha_e^{+} - \alpha_e^{-} } \right)} \right] $$

(7)

where $\left( {\alpha_c^{+} - \alpha_c^{-} } \right)$ is the calibrated total pan angle, °; and $\left( {\alpha_e^{+} - \alpha_e^{-} } \right)$ is the experimental total pan angle, °.The relationship between Δα and strain measurement errors is the focus of the following discussion. Numerical simulations (Appendix B) are used to investigate this relationship.

To demonstrate the effect of Δα on strain measurement errors, the authors selected initial rotation angles that were consistent with the upper limits expected for SEM applications, ⁺α = 10° and α⁻= −10°. The range for Δα used in the simulations is −0.20° ≤ Δα ≤ 0.20°, which spans the range of errors obtained from a series of baseline specimen rotation experiments using the SEM specimen rotation stage. No distortion is considered in this simulation.

First, two images of a 9 × 9 grid with 0.04 mm grid spacing are generated using the parameters from Exp 3 in Tables 1 and 2, with the pan angle perturbed by Δα. Then, the “realized” 3D positions of the object are reconstructed using the “known” parameters (i.e., the parameters from Exp 3 in Tables 1 and 2). The artificial displacement from the real 3D positions to the “realized” 3D positions can be computed.

The artificial strains introduced by Δα are determined using a local quadratic fit to all 5 × 5 arrays within the 9 × 9 data set, averaging the resulting data to obtain a singe value for each strain component. The results for the dominant strain error, ε_yy, show a well-defined linear relationship between the strain field bias and Δα. Fitting a straight line to the strain error vs Δα data for each strain component, we have the following form;

$$ \left\{ {\begin{array}{*{20}{c}} {{\varepsilon_{xx}} = - 2.38 \times {{10}^{ - 6}} \cdot \Delta \alpha } \\{{\varepsilon_{yy}} = 1530 \times {{10}^{ - 6}} \cdot \Delta \alpha } \\{{\varepsilon_{xy}} = - 2.50 \times {{10}^{ - 6}} \cdot \Delta \alpha } \\\end{array} } \right. $$

(8)

For the case being considered, equation (8) shows that the strain bias is less than 150 µε for deformation measurements performed at a magnification of 300× when Δα < 0.10°.

Experimental Validation in SEM

To demonstrate the potential of the proposed stereo-vision methodology using SEM images, a benchmark experiment was performed to reconstruct a planar object undergoing three-dimensional rigid body motions. The procedure used to complete the experiments is detailed in Appendix C and includes (a) SEM spatial distortion correction, (b) SEM stereo calibration, (c) SEM experimental process and (d) image processing for three-dimensional shape and motion measurements. All experiments were performed using an FEI Quanta 200 ESEM. The eucentric goniometer stage within the FEI chamber was used to perform all rotations; operational errors were demonstrated to be less than 0.10°. Images were acquired at 300×, with the following imaging parameters: High vacuum mode; 30.0 kV working voltage; Spot size = 5.0; Backscattered electron detector for all imaging; Working distance = 10.0 mm; Frame time = 4.73 s.

Specimen Preparation

Both a 9 by 9 calibration grid and the random speckle pattern used for image correlation were manufactured by etching a glass plate coated with a 120 nm thick layer of Chrome/Chrome Oxide using a multi-step optical reduction process [41] (see Fig. 5). The combination of Chrome/Chrome Oxide and glass provides good contrast at a magnification of 300× in SEM. For higher magnifications, advanced etching processes such as FIB milling may be applied to manufacture the specimen pattern.

Distortion Removal

In this work, drift distortion is neglected since baseline imaging at 300× magnification without motion confirmed that drift was negligible over the 60 min required to complete the experiment. As noted in Appendix C, spatial distortion removal followed procedures used previously by one of the authors [35], employing a series of translated images of the specimen to extract the vector spatial distortion field. Figure 4 shows the as-computed spatial distortions for both viewing directions. These distortions are removed from all SEM images. The complicated shape of the distortion field explains why normal parametric distortion model cannot be used to characterize the distortions embedded within SEM images.

SEM Stereo Calibration

The intrinsic parameters for both views and the rigid body coordinate transformations for each grid view are obtained using the distortion-free SEM images of a standard grid (see Fig. 5 and Appendix C). To obtain the relative orientation and position of the two views (i.e., the extrinsic parameters), distortion-corrected speckle images⁵ from both views are used with the known intrinsic parameters. The number of point correspondences used to calibrate the extrinsic parameters is ≈24,000 in this experiment. To uniquely define the baseline vector, t, for the two stereo views, internal SEM software, XDOC,⁶ is used to estimate the metric distance between two feature points. By using image correlation to locate these feature points in both stereo views, an additional equation is obtained and used in the optimization process to remove ambiguity in t. Equation (A.3) demonstrates one form of the relationship between components of t and any 3D position

Table 3 presents both the intrinsic and extrinsic parameters for the SEM stereo-vision pinhole model. The angle α (which corresponds to the sum of the specimen rotation angles) is consistent with the estimated 20° specimen rotation during the experiment.

Table 3 Calibration parameters for Exp 3

SEM Rigid Body Motion Experiments

A series of rigid body motion experiments was performed to demonstrate the proposed methodology. The motions included (a) in-plane rotation, (b) out-of-plane rotation and (c) large, out-of-plane translation. For each experiment, a total of 140 × 140 subsets with subset size 27 × 27 and subset spacing of 5 pixels were compared to obtain ≈19,600 three-dimensional data points. Figure 7 presents the reconstructed profiles for the translated specimen (red or dark gray), with the original specimen as the reference (green or light gray). Results show that (a) the reconstructed shape of the object is not distorted by rigid body motion and (b) the measured motions are qualitatively consistent with the actual motions.

Fig. 7

Comparison of reconstructed surface before (green, or light color) and after rigid body motion (red, or dark color) for (a) 10° in-plane rotation; (b) 2° out-of-plane rotation; (c) 0.1 mm out-of-plane displacement

Using a least squares procedure, the measured motions for each case were computed. The actual motions of the specimen are read from the goniometer and displacement sensors on the sample manipulation stage, with reported accuracy in these results of 0.01mm. Table 4 presents a direct comparison between the measured and actual motions. As shown in Table 4, the measured results are within the error band for the motions applied to the specimen in each case.

Table 4 Measured and actual motions

Further validation of the measurement method was performed by extracting in-plane strains from the 3D measurements. Using commercial software [40], a 15 × 15 smoothing window and 5 pixel spacing between measurement points, the strain distributions were computed for each case. Figure 8 shows the measured strain fields for out-of-plane rotation of 2°; similar results were obtained for 10^o rotation and 0.10mm out-of-plane motion.

Fig. 8

Measured strain distribution after 2° out-of-plane rotation

As shown in Fig. 8, the measured strains are generally random in nature, confirming that the effects of spatial distortion and other imaging anomalies are properly accounted for in the measurement process. Table 5 presents quantitative data for the measured strains for all three cases.

Table 5 Quantitative data for strain measurements with distortion correction

Table 6 presents the coefficients of least squares fit planes to the artificial strain data. As shown in Table 6, the slopes of the artificial strains with distortion correction are much smaller than the ones without distortion correction which further confirms that the distortion correction removes bias not only in the mean value but also in the distributions. Standard deviations in Table 6 are relative to the model plane instead of the mean value in Table 5.

Table 6 Strain error with and without distortion correction

Discussion of Results

As shown in Table 5, strain data for all motions has relatively high variability due to the effects of intensity noise, scanning inaccuracy and associated SEM imaging artifacts. The largest variability occurred for in-plane rotation. Since in-plane rotation introduces considerable displacement within the complex image distortion field (see Fig. 4), both the slight bias and the variability within the in-plane rotation data are in part due to inaccuracies in the distortion correction process.

The small out-of-plane rotational motion results have quite small bias in all components, confirming that the stereo approach is effective in quantifying the three-dimensional motion of the points when the motions fall within the depth of field for the imaging system. The largest strain bias (e.g. deviation of mean value from zero) was observed for the out-of-plane motion experiment where the specimen translated beyond the calibration volume⁷ and slightly defocused after motion occurred. Even so, the results are encouraging and demonstrate that the stereo approach will remove most of the effects associated with out-of-plane motion, even when the image is near the edges of the focus region.

Finally, it should be noted that an important, yet quite subtle, contribution of this work is to demonstrate conclusively that the spatial distortion correction procedure developed and validated for stationary optical imaging systems can be successfully used to identify and remove similar distortions from all SEM images, including those obtained at various specimen tilt angles. In essence, our work has shown that “spatial distortion” in an SEM image is only a function of the pixel position in the resulting image and is not altered in any way by specimen tilt or other motions.

Concluding Remarks

To ensure that the accuracy of the rotation stage is adequate for measurements, simulations can be performed to quantify the potential strain bias as a function of Δα. Using simulation results, the required accuracy of the positioning stage can be estimated so that the appropriate stage can be selected.

For cases with higher magnification in an SEM, it is likely that drift distortions will be important and hence should be considered to optimize measurement accuracy. The method proposed by Sutton et al. [4] can be used to remove both drift distortion and spatially-varying distortion in each stereo view.

To calibrate the intrinsic parameters at higher magnification, it is possible to manufacture and use high resolution grids by modern focused ion beam technology at both the micro-scale and nano-scale. Tolerance on the accuracy of the grid spacing during manufacturing can be relaxed using bundle-adjustment algorithms [31, 36] so that positions of the slightly offset grid points are also optimized.

For those investigators interested in metric reconstructions, the critical role of determining the scale constant ω must be emphasized, as the entire calibration process outlined in this work requires this constraint.

Conclusions

In this paper, the basic experimental approach for development and use of a stereovision system with SEM images is presented. Using existing stereovision models, with an appropriate interpretation for both calibration and reconstruction to remove scale ambiguity, single column SEM images acquired at different tilt angles can be shown to provide a robust platform for three-dimensional measurements at high magnification.

The effect of operational error introduced during specimen rotation to define two stereo views is analyzed by simulation and results are used to guide the rotation operations for the eucentric goniometer stage. Repeatability and accuracy of measurements is demonstrated through independent calibration experiments and rigid body motion experiments, with strain bias O(10⁻⁴) and strain variability O(3 × 10⁻⁴) demonstrated through multiple experiments.

Appendices

Appendix A: Extrinsic Parameters Constraint

The calibration process described in “System Calibration and Distortion Removal” for the extrinsic parameters requires solving for both the extrinsic parameters and the 3D positions with known intrinsic parameters and image point correspondences (e.g., motion analysis in the computer vision literature). Theoretically, as described in previous work [27], when the intrinsic parameters are known it can be shown that there are 5 constraints on the six extrinsic parameters. The constraints can be obtained by using the essential matrix, [E], and the following formula [28]:

$$ \left[ {\mathbf{R}} \right] \cdot {\left[ {\mathbf{t}} \right]_\times } = \left[ {\mathbf{E}} \right] $$

(A.1)

where [R], t representing the rotation matrix and translation vector defined in eq. (2), respectively, with ${\mathbf{t}} = \left\{ \begin{gathered} {{\text{t}}_{\text{x}}} \hfill \\ {{\text{t}}_{\text{y}}} \hfill \\ {{\text{t}}_{\text{z}}} \hfill \\ \end{gathered} \right\}{; }\;\;{\left[ {\mathbf{t}} \right]_\times } = \left( {\begin{array}{*{20}{c}} 0 & { - {t_z}} & {{t_y}} \\ {{t_z}} & 0 & { - {t_x}} \\ { - {t_y}} & {{t_x}} & 0 \\ \end{array} } \right)$.

By construction, the essential matrix is a function of the unknown extrinsic parameters. Determination of [E] provides sufficient information to obtain the extrinsic parameters. The essential matrix can be determined using a series of equations. First, [E] can be used to relate the sensor positions of corresponding image points.

$$ {\left( {{{{\mathbf{{\overset{\lower0.5em\hbox{$\smash{\scriptscriptstyle\frown}$}} {m}}}}}_{2i}}} \right)^T} \cdot \left[ {\mathbf{E}} \right] \cdot \left( {{{{\mathbf{{\overset{\lower0.5em\hbox{$\smash{\scriptscriptstyle\frown}$}} {m}}}}}_{1i}}} \right) = 0 $$

(A.2)

where ${{\mathbf{{\overset{\lower0.5em\hbox{$\smash{\scriptscriptstyle\frown}$}} {m}}}}_{1i}}$ and ${{\mathbf{{\overset{\lower0.5em\hbox{$\smash{\scriptscriptstyle\frown}$}} {m}}}}_{2i}}$ in metric coordinates can be obtained from the image coordinate using the intrinsic parameters. To obtain the terms in the essential matrix, equation (A.2) can be employed with at least 5 point correspondences to obtain five independent equations. In addition to these equations, the three known constraints on the essential matrix can also be employed,

• i)
  $$ \det \left( {\left[ {\mathbf{E}} \right]} \right) = 0 $$

• ii) Employ the two Kruppa equations [27] with known intrinsic parameters.

In this form, [E] can be obtained up to an arbitrary non-dimensional scale, μ. The non-dimensional scale, μ, typically is associated with the vector defining the location of the pinhole in the second camera relative to the first camera so that t is the true translation vector obtained when the scale, μ, is uniquely determined.

Assuming that the scale is embedded in the translation vector, then the following formulae demonstrates that the scale also affects the 3D position of the corresponding point.

$$ {\hbox{M}} = {\left( {{{\mathbf{Q}}^{\mathbf{T}}}{\mathbf{Q}}} \right)^{{\mathbf{- 1}}}}{{\mathbf{Q}}^{\mathbf{T}}}{ }{\mathbf{b}} $$

(A.3)

where

$$ M = \left[ {\begin{array}{*{20}{c}} {{{\hbox{X}}_{\rm{W}}}} \\{{{\hbox{Y}}_W}} \\{{{\hbox{Z}}_W}} \\\end{array} } \right];{ }{\mathbf{b}} = \left[ {\begin{array}{*{20}{c}} {0} \\{0} \\{ - \left( {{{\hbox{t}}_{\rm{x1}}}{{\hbox{f}}_{\rm{x 2}}} + {{\hbox{t}}_{\rm{y1}}}{f_S}_2 + {{\hbox{t}}_{\rm{z1}}}\left( {{{\hbox{C}}_{\rm{x2}}} - {x_{\rm{S2}}}} \right)} \right)} \\{ - \left( {{{\hbox{t}}_{\rm{y1}}}{{\hbox{f}}_{\rm{y 2}}} + {{\hbox{t}}_{\rm{z1}}}\left( {{{\hbox{C}}_{\rm{y2}}} - {y_{\rm{S2}}}} \right)} \right)} \\\end{array} } \right] $$

$$ {\mathbf{Q}} = \left[ {\begin{array}{*{20}{c}} {{{\hbox{f}}_{\rm{x 1}}}} & {{{\hbox{f}}_S}_1} & {{{\hbox{C}}_{\rm{x1}}} - {x_{\rm{S1}}}} \\{0} & {{{\hbox{f}}_{\rm{y 1}}}} & {{{\hbox{C}}_{\rm{y1}}} - {y_{\rm{S1}}}} \\{{{\hbox{R}}_{11}}{{\hbox{f}}_{\rm{x 2}}} + {{\hbox{R}}_{{21}}}{f_{\rm{S2}}} + {{\hbox{R}}_{31}}({{\hbox{C}}_{\rm{x2}}} - {x_{\rm{S2}}})} & {{\hbox{R}}_{{12}}{{\hbox{f}}_{\rm{x 2}}} + {{\hbox{R}}_{{22}}}{f_{\rm{S2}}} + {\hbox{R}}_{{32}}({{\hbox{C}}_{\rm{x2}}} - {x_{\rm{S2}}})} & {{{\hbox{R}}_{13}}{{\hbox{f}}_{\rm{x 2}}} + {{\hbox{R}}_{{23}}}{f_{\rm{S2}}} + {{\hbox{R}}_{33}}\left( {{{\hbox{C}}_{\rm{x2}}} - {x_{\rm{S2}}}} \right)} \\{{{\hbox{R}}_{21}}{{\hbox{f}}_{\rm{y 2}}} + {{\hbox{R}}_{31}}\left( {{{\hbox{C}}_{\rm{y2}}} - {y_{\rm{S2}}}} \right)} & {{\hbox{R}}_{{22}}{{\hbox{f}}_{\rm{y 2}}} + {\hbox{R}}_{{32}}({{\hbox{C}}_{\rm{y2}}} - {y_{\rm{S2}}})} & {{{\hbox{R}}_{23}}{{\hbox{f}}_{\rm{y 2}}} + {{\hbox{R}}_{33}}\left( {{{\hbox{C}}_{\rm{y2}}} - {y_{\rm{S2}}}} \right)} \\\end{array} } \right] $$

Inspection of this form demonstrates that (a) the vector b is scaled by the parameter μ since each non-zero term contains a component of the vector t and (b) the matrix [Q] is not a function of the translation vector, but rather it is a function of the rotation tensor, the intrinsic camera parameters for both view 1 and view 2 and the sensor positions in views 1 and 2 for the common point. Taken together, eq. (A.3) demonstrates that each 3D position is also scaled by the parameter, μ, since each component is a linear function of the components of t.

Appendix B: Equations for Numerical Simulation of Strain Error Due to Pan-angle Variation

Without loss of generality, we consider the simplest case. Assuming a planar object, no distortion and zero skew, rotations are about x-axis and both translation vectors are [0 0 D]^T, we have the stereovision model [equation (2)] in a simplified form:

$$ \left\{ {\begin{array}{*{20}{c}} {{\rho_{1i}} \cdot {{\mathbf{m}}_{1i}} = \left[ {\mathbf{A}} \right] \cdot \left[ {{R_1}\;\;t} \right] \cdot {M_i}} \\{{\rho_{2i}} \cdot {{\mathbf{m}}_{2i}} = \left[ {\mathbf{A}} \right] \cdot \left[ {{R_2}\;\;t} \right] \cdot {M_i}} \\\end{array} } \right. $$

(B.1.a)

where ${\mathbf{m}} = \left[ {\begin{array}{*{20}{c}} {{x_s}} \\{{y_s}} \\1 \\\end{array} } \right],{\mathbf{M}} = \left[ {\begin{array}{*{20}{c}} {{X_w}} \\{{Y_w}} \\0 \\1 \\\end{array} } \right],\left[ {\mathbf{A}} \right] = \left[ {\begin{array}{*{20}{c}} {{f_x}} & 0 & {{C_x}} \\{} & {{f_y}} & {{C_y}} \\{} & {} & 1 \\\end{array} } \right],\left[ {\mathbf{R}} \right] = \left[ {\begin{array}{*{20}{c}} 1 & 0 & 0 \\0 & {\cos \theta } & { - \sin \theta } \\0 & {\sin \theta } & {\cos \theta } \\\end{array} } \right]$.with β = γ = 0 and the angle α for this case represented by θ.

Equation (B.1.a) can be written in the following,

$$ \left\{ {\begin{array}{*{20}{c}} {{{\mathbf{m}}_{1i}} = {\mathbf{f}}\left( {{f_x},{f_y},{C_x},{C_y},{\theta_1},D,{{\mathbf{M}}_i}} \right)} \\{{{\mathbf{m}}_{2i}} = {\mathbf{f}}\left( {{f_x},{f_y},{C_x},{C_y},{\theta_2},D,{{\mathbf{M}}_i}} \right)} \\\end{array} } \right. $$

(B.1.b)

For a series of points created in 3D space, M _i , the corresponding image position, m _1i and ${\tilde{m}_{2i}}$ are generated by

$$ \left\{ {\begin{array}{*{20}{c}} {{{\mathbf{m}}_{1i}} = {\mathbf{f}}\left( {{f_x},{f_y},{C_x},{C_y},{\theta_1},D,{{\mathbf{M}}_i}} \right)} \\{{{{\mathbf{\tilde{m}}}}_{2i}} = {\mathbf{f}}\left( {{f_x},{f_y},{C_x},{C_y},{{\tilde{\theta }}_2},D,{{\mathbf{M}}_i}} \right)} \\\end{array} } \right. $$

(B.2)

where ${\widetilde{\theta }_2}$ is the perturbed angle, ${\widetilde{\theta }_2} = {\theta_2} + \Delta \theta$. The biased 3D positions ${{\mathbf{\tilde{M}}}_i}$ (${{\mathbf{\tilde{M}}}_i} = {\left[ {\begin{array}{*{20}{c}} {{{\tilde{X}}_w}} & {{{\tilde{Y}}_w}} & {{{\tilde{Z}}_w}} & 1 \\\end{array} } \right]^T}$) due to the angular perturbation ∆θ are reconstructed from the images generated by eq. (B.2) using the least squares fitting procedure,

$$ {r_i} = \mathop {{\min }}\limits_{{{{\mathbf{\tilde{M}}}}_i}} \sum\limits_{j = 1}^2 {{{\left( {{{{\mathbf{\tilde{m}}}}_{ji}} - {\mathbf{f}}\left( {{f_x},{f_y},{C_x},{C_y},{\theta_j},D,{{{\mathbf{\tilde{M}}}}_i}} \right)} \right)}^2}} $$

(B.3)

where r _i is the residual error for the i-th point, ${{\mathbf{\tilde{m}}}_{1i}} = {{\mathbf{m}}_{1i}}$.

The artificial displacement field can then be computed by

$$ {\mathbf{dis}}{{\mathbf{p}}_i}\left( {\Delta \theta } \right) = {{\mathbf{\tilde{M}}}_i} - {{\mathbf{M}}_i} $$

(B.4)

and the strain field can be calculated as described previously [38].

Appendix C: Experimental Procedure for Stereovision in SEM

The experimental procedure includes (1) distortion correction, (2) system calibration, (3) specimen loading and (4) image processing to extract motion measurements.

Spatial Distortion Correction

A speckled planar target is placed in the SEM chamber on the eucentric goniometer, and oriented to be perpendicular to the e-beam. The target can be the experimental specimen if the pattern has good contrast in an SEM. Interruption of e-beam scanning is permitted during this stage, but all imaging parameters must be unchanged when initiating a new scan.

Once installed, the specimen is translated in both horizontal and vertical directions in the plane perpendicular to the optical axis (i.e. γ = 0°) following a cross-type path. Images are acquired at each translated position for use in distortion correction.

System Calibration

To calibrate an experimental setup, images are acquired of the micro-scale grid at various orientations. During the calibration process, the specimen is moved as necessary to maintain adequate focus. Table 7 presents the experimental out-of-plane rotations and in-plane rotations used to acquire 25 images of the micro-grid for calibration. Out-of-plane specimen rotations from −8° to 8° are performed using the eucentric goniometer. In-plane specimen rotations are performed using a built-in Quanta 200 electron beam rotation function.

Table 7 Image sequence for calibration

Experimental Phase

Once the distortion correction process and the calibration procedure have been completed, all SEM settings are maintained and the specimen is placed on the eucentric goniometer in the SEM chamber. Prior to applying load, images of the planar speckled specimen rotated out-of-plane by −10° and +10°, respectively, are acquired as the reference image pair.

Loading is applied to the specimen and images are acquired with the specimen oriented at −10° and +10° respectively. This process of out-of-plane specimen rotation and incremental loading is repeated until the experiment is completed.

Image Processing

After completing the three primary phases of the experimental work, images are evaluated from each phase.⁸ First, the spatial distortion correction images are processed using procedures outlined previously [32] for optical image correction. Second, all calibration images are analyzed to obtain the center-point position of each grid point. Each identified grid point is corrected for spatial distortion and bundle adjustment procedures within VIC-3D are used to obtain (a) the intrinsic parameters for the SEM stereo-vision system and (b) the orientation and position of each calibration grid.

Finally, all speckle images for the −10° and +10° specimen rotations are corrected for spatial distortion and then input into VIC-3D as stereo image pairs for cross-correlation and three-dimensional motion measurements.