Introduction and Background

The development of scientific instruments and computational power has enabled the capture of large amounts of experimental data and spurred the development of novel analysis approaches. One approach to study materials deformation at the microstructural length scale is thermo-mechanical testing inside a scanning electron microscope (SEM). Over the past decades, in-situ SEM testing has focused on the microscale response to mechanical loading, for example the development of dislocation slip [1], void formation [2], fracture and fatigue behavior [3,4,5], and grain boundary sliding and cracking during creep [6]. In addition, the development of micromachining and microelectromechanical-systems (MEMS) has enabled testing at even smaller length scales, such as micro/nano-pillars [7] and thin films [8]. Such experiments provide high-resolution observations and stress-strain measurements of size-dependent material properties. However, these approaches do not probe the large volumes and surfaces needed for statistical representations.

One experimental approach to quantify high-resolution, full-field microscale deformations across statistically representative, mm-scale fields of view is a combination of scanning electron microscopy and digital image correlation (SEM-DIC). Digital image correlation (DIC) is a non-contact, length scale independent technique to measure full-field surface displacements with sub-pixel resolution by tracking the movement of a random speckle pattern on the sample surface. By comparing a ‘reference image’ to a ‘deformed image’ of the same sample area, the displacement fields, and thereby the strain fields, can be calculated. The application of DIC to the field of mechanics originated in the 1980s [9,10,11,12,13,14], and is primarily used with optical imaging systems. In the recent decades, following the pioneering work of Sutton et al. [15, 16], an increasing number of studies have explored the implementation of DIC in the SEM (SEM-DIC).

In order to obtain accurate DIC results, the sample surface needs to have an isotropic, high-contrast, and random speckle pattern. The speckle size must also be tuned to the imaging conditions, including the field-of-view (FOV) size and pixel size. Various patterning methods for SEM-DIC have been developed, including the application of self-assembled gold nanoparticles to a functionalized metallic surface [17, 18], e-beam lithography [19], laser-assisted through thin film ablation [20], preferential etching of microstructural features with an optimized field of view [21, 22], nanoscale remodeling of metallic films [23, 24], and UV photolithography [23].

In addition to small-scale patterning, SEM-DIC requires consideration of spatial and temporal (drift) distortions to the images that are imparted by the SEM. An advantage of SEM over optical imaging is its higher, diffraction-limited spatial resolution. While SEM images reveal detailed features of specimens, various types of (spatial and temporal) image distortions are imparted by the electron optics and become detectable with the application of DIC analysis. Spatial distortions in a SEM are due to lens imperfections, similar in concept to distortions in optical lenses, although in SEM electromagnetic lenses bend electrons instead of light. While optical imaging distortions are often corrected by parametric models [25], spatial distortions in the SEM are position-dependent, and based on the location of the electron beam that is controlled by the electromagnetic lenses during the scan process. The spatial distortion imparted by a SEM is non-parametric, and is more significant at lower magnifications where the effects of lens defects are magnified. In addition, SEM imaging exhibits drift distortions that vary over time as a function of pixel location. This is often an effect of a material (sample) charging, and not necessarily due to the SEM. This effect increases at higher magnifications, where the amount of beam drift is larger relative to the field of view. Following the work of Sutton et al. [15, 16], there are on-going efforts to develop processes for calibration and correction of SEM image distortions [26,27,28].

In addition to distortion correction through calibration, an underutilized way to significantly minimize distortion-related issues in a SEM is by improved scan control of the electron beam. As a SEM image can be viewed as the strength of secondary/backscattered electron signals as a function of beam position, a physical mitigation of distortions can be achieved by improving control over the beam position. One approach is to externally generate a tunable scan signal to use as the external source for the SEM to control the electron beam scan path/position. An external scan control procedure created by Lenthe WC, Charles J, Echlin MP, et al. (Advanced detector signal acquisition and electron beam scanning for high resolution SEM imaging, 2018, submitted) can significantly reduce spatial distortions in SEM-DIC investigations to the point where distortion calibration procedures are unnecessary [29].

Once an optimized speckle pattern and minimal imaging distortions were achieved, a primary challenge was to capture high-resolution microscale displacement fields over relatively large (e.g. mm-scale) fields of view. This relates to a fundamental challenge that is currently being addressed in the mechanics of materials community, namely the capture of experimental data across large surfaces and volumes that is needed to statistically represent microstructure/feature variation, and the development of analytical frameworks to mine the resultant large data. This was accomplished in this work by collecting multi-tile datasets of the full-field displacements, strains, and microstructure during a test for subsequent stitching and alignment. Approaches for stitching are presented in this work; there are a limited number of studies to date on the stitching of displacement and strain data, although this is a critical step in obtaining high-resolution, large FOV data. A methodology for stitching high-resolution strain fields was introduced by Carroll et al. in 2010 [30], on images captured ex-situ using optical microscopy during tensile and fatigue crack growth experiments. The strategies of stitching optical images and then performing DIC, versus performing DIC on individual images and then stitching, were compared. It was found that the former method created artificial bands in the strain field that were non-negligible when the globally applied strain was less than 1%. Although such artifacts could be eliminated in the latter method after interpolation of the strain values, this created sharp displacement discontinuities at the seams that complicated displacement field stitching. A strategy to minimize seam discontinuities for displacement field stitching is discussed in this work.

Polycrystalline microstructures are generally characterized in SEM experiments by electron backscatter diffraction (EBSD). However, it can be difficult to properly focus the beam dynamically across a large area due to the high sample tilt during EBSD data collection, and a trapezium distortion can exist if the tilt geometry is not compensated by the system [31]. In addition, spatial distortions are magnified as the FOV size increases. Therefore, rather than perform a single scan over a large area, EBSD scans of multiple small areas were captured in this work using a combination of beam control and stage control, and then stitched in post-processing. Multi-tile EBSD data can also be collected by an automated procedure supported by the built-in functions of the TSL OIM Data Collection software, and stitched by the TSL OIM Analysis software (EDAX, Inc., Mahwah, NJ, 07430, USA). Customization of these approaches has been previously investigated [32,34,34]; for example, an approach to process multi-tile EBSD data was outlined by Pilchak et al. [32], in which EBSD data was stitched based on manual inputs of the tile overlap and a constant relative shift. This approach was later updated [33] to first locate grain boundaries and phase boundaries in the overlapping regions, and then use cross-correlation algorithms to determine the relative position of individual scan tiles.

Statistical and unsupervised learning approaches are being increasingly used to examine interactions between microstructure and microscale deformation mechanisms, and the impact of this activity on the macroscopic material behavior. This paper details the development of experimental frameworks needed to capture high-resolution, microscale deformation mechanisms over mm-scale fields of view, which serves as a basis for statistical approaches to examine interactions between microstructure, microscale strain localization, and macroscopic properties in the future.

Experimental Approach

Displacement and strain fields were captured during in-SEM testing across large mm-scale FOVs, by the automated capture of individual μm-scale FOVs that were processed using DIC and stitched together using custom approaches. The procedure for doing this, including automation of the image capture process, DIC data stitching, EBSD data stitching, and alignment of the DIC and EBSD data, is discussed below.

Experiments that collect multi-tile datasets in-situ generally involve prolonged, continuous operation that limits the number of individual tiles. Automation of the testing procedure can significantly increase experimental quality and repeatability. In this work, an external scan control was combined with a FEI iFast script and custom C++ scripts to run an automated procedure that included SEM stage translation, auto focus, auto brightness and contrast, auto stigmation, and image capture. This significantly improved the quality of the displacement/strain fields, as discussed below.

Mechanical Testing

A WE43 magnesium alloy provided by Magnesium Elektron Ltd. was used in this work. The as-received material (T5 condition) was hot rolled plate aged for 48 h at 204 °C. The material was solution treated at 525 °C for eight hours, water quenched, and peaked-aged (T6 condition) at 250 °C for 16 h. The composition was 3.74 wt% Y, 2.10 wt% Nd, 0.52 wt% Gd, 0.45 wt% Zr, 0.016 wt% Zn, and Mg (balance). Dogbone-shaped test specimens of the geometry shown in Fig. 1 were cut using electrical discharge machining and mechanically polished with SiC paper and diamond paste to a 1 μm finish, followed by a final polishing with a high pH colloidal alumina/silica mixture. The sample was etched with a mixture of 50 ml methanol, 6 ml hydrochloric acid, and 4 ml nitric acid before EBSD. A microstructural analysis revealed the grain distribution shown in Fig. 1, with an average grain size of approximately 116 μm.

Fig. 1
figure 1

(a) Dogbone-shaped test specimen geometry, (b) grain size distribution, (c) EBSD inverse pole figure (IPF) map in the normal direction (ND) of the Mg WE43 subjected to uniaxial compressive loading, and (d) secondary electron image of a representative image tile. A small window marked by the black rectangle in (d) is magnified to show the speckle pattern quality. Please note that patterning of magnesium and its alloys requires a different approach than for other metallic alloys, due to its high susceptibility to corrosion, and typically results in sparser patterns

Full size image

A globally applied uniaxial compressive strain of εxx = −4.7% was applied to the WE43-T6 Mg alloy using a thermo-mechanical testing stage (Kammrath and Weiss) in a FEI Teneo SEM. SEM-DIC was performed to capture high-resolution full-field strains at the sub-grain scale. Before mechanical testing, the sample was patterned (see Fig. 1(d)) with 300 nm diameter gold nanoparticles purchased from Sigma-Aldrich, following the method described in Kammers and Daly [17]. Displacement-controlled loading was applied, and the loading was interrupted at specified globally-applied displacements to capture secondary electron images of the sample surface. Images were taken with accelerating voltage of 30 kV, spot size 11, and 4096 × 4096 pixel resolution, with a 3.2 μs dwell time at each pixel position. A strain gage was centrally applied to the backside of the gage section for continuous macroscopic strain monitoring. Digital image correlation was performed using commercial software VIC-2D [35], with a subset size of 21 by 21 pixels (1.85 μm × 1.85 μm) and a step size of 5 pixels (0.44 μm), using optimized 8-tap interpolation and zero-normalized squared difference correlation criterion, to calculate the displacement fields and therefore Lagrangian strain fields.

Determination of Image Tile Position

DIC was performed on individual image tiles, and the displacement/strain fields were stitched together. To avoid data loss and analysis complications, the number and magnification of image tiles, as well as the center-located feature of each image tile, remained constant at each globally applied load step. With increasing globally applied strain, the deformed images can contain smaller areas of DIC data, as features near the FOV boundary can move outside of the FOV due to the applied deformation (particularly for tensile tests). In this work, an approximately 15% overlap was applied for the reference image tiles. This overlap provided common features for image stitching, ensured that an overlap of the deformation fields could still be found in the neighboring tiles after deformation, and avoided over-collection of redundant data that would result in a significant increase in experiment time.

For each tile position, the feature located at the center of the reference image was imaged as close to the center of the deformed image as possible. The SEM stage location at which each image should be taken (image position) was determined using a bilinear interpolation, as follows: First, assume a rectangular grid of R × C images is captured to later stitch together. The stage position for an arbitrary image is a function of the row number and column number, denoted by P(r, c) = (x, y), where r ∈ [0, R] is the image row number and c ∈ [0, C] is the image column number. The image positions at the four corners of the grid, i.e., P(0, 0), P(0, C), P(R, 0), and P(R, C), were determined by inspecting surface features and used as references to interpolate all other positions. A linear interpolation was applied to the 0th row and Rth row, for an arbitrary column c:

$$ P\left(0,c\right)=\frac{C-c}{C}P\left(0,0\right)+\frac{c}{C}P\left(0,C\right) $$
(1)
$$ P\left(R,c\right)=\frac{C-c}{C}P\left(R,0\right)+\frac{c}{C}P\left(R,C\right) $$
(2)

For an image in the cth column and an arbitrary row r, a linear interpolation was applied using the image position of the 0th row and the Rth row:

$$ P\left(r,c\right)=\frac{R-r}{R}P\left(0,c\right)+\frac{r}{R}P\left(R,c\right) $$
(3)

By substituting in the expression of P(0, c) and P(R, c), an expression for P(r, c) was obtained:

$$ P\left(r,c\right)=\frac{1}{R\cdot C}\left(\left(R-r\right)\cdot \left(C-c\right)\cdot P\left(0,0\right)+\left(R-r\right)\cdot c\cdot P\left(0,C\right)+r\cdot \left(C-c\right)\cdot P\left(R,0\right)+r\cdot c\cdot P\left(R,C\right)\right) $$
(4)

The image position for an arbitrary tile was then determined based on the above relationship.

Mitigation of SEM Image Distortions

Figures 2 and 3 illustrate two examples of spatial distortions observed at different length scales. Such distortions have been observed across different SEMs, including a FEI Teneo SEM and a Tescan Mira 3 SEM, and were found to be greatly minimized with the application of the custom external scan control.

Fig. 2
figure 2

Image distortions can vary in character and can be mitigated by external scan control. (a) An illustration of the relative positions of a reference and deformed image, wherein the test specimen underwent a rigid body translation. (b) With a sample translation of 2 μm in a Tescan Mira 3 SEM with stock scan control, noise appeared as diagonal cross-hatched lines. (c) With a sample translation of 20 μm in a FEI Teneo SEM with stock scan control, noise appeared as vertical and diagonal lines. (d) The application of external scan control during a sample translation of 20 μm in a FEI Teneo SEM resulted in significant noise mitigation. Note that (d) was performed on the same area of the sample (same pattern quality) as (c), with the same pixel resolution and imaging parameters adjusted for an image capture time of approximately 45 s. The area imaged in (d) is indicated by the white square window in (c)

Full size image
Fig. 3
figure 3

Image distortion in the overlay between two FOVs can significantly affect image stitching and be mitigated by external scan control. (a) Two images with a stitching overlap region (located on the right side of image 1 and left side of image 2). Each image contributed 50% of the grayscale value in the overlap region. Four small windows in the overlap area are labeled 1–4 and magnified for observation. (b) For images captured using internal/stock scan control, the quality of overlay can vary significantly. (c) For images captured using external scan control, the overlay quality is consistently improved

Full size image

Spatial noise patterns in the strain fields were observed when using the stock scan controller to image a sample subjected to a rigid body translation, as shown in Fig. 2. Two examples are shown in Fig. 2(b) and (c), where noise appeared as crosshatches and vertical/diagonal lines respectively. Such distortion was also observed in other studies [36]. However, the use of external scan control resulted in a significant mitigation of this image noise, as evident in Fig. 2(d).

Spatial distortion also existed in another form, wherein the relative positions of the same features were not the same on different images. In Fig. 3, the sample was subjected to a rigid body translation close to the width of the FOV. Features were found to be imaged differently depending on their location in the image, with more severe distortion at the left edge of the image. This precluded a high quality overlay needed to stitch neighboring images, which significantly affected the reconstruction of the whole sample area. Thus, if this distortion is found to be present, it is advised to capture images with a large area of overlap and discard the highly distorted data near the edges, using the remaining data where the distortions are relatively small for image stitching. Alternatively, the application of external scan control produced a high quality overlay and eliminated the need to discard data near the edges, as shown in Fig. 3(c).

In the current study, a NI USB 6251 BNC M Series DAQ Device (referred to as a DAQ) was used for external scan control to customize the imaging process on a FEI Teneo SEM. A brief introduction of the setup is introduced as follows, and the reader is referred to for further details. Two analog output terminals of the DAQ were connected to the external scan control port on the SEM, sending horizontal (X-axis) and vertical (Y-axis) scan signals to the microscope. An analog input terminal was connected to the Everhart-Thornley Detector (ETD), sampling the input signal at 1.25 MHz for imaging. No line integration or frame integration was applied. As illustrated by Fig. 4, the electron beam was driven by the scan signal to sequentially reach each of the target pixel positions, where the selected dwell time was applied. At each pixel position, the sampled detector data were stored in an array. An image was constructed from the detector data arrays from each pixel and stored as a multiple image TIFF stack, where the N-th image in the stack was constructed from the N-th element in the detector data arrays. Different electron beam scan patterns, such as a raster or a snake pattern, can be applied. It was found that a few microseconds of dwell time was generally required for the beam to stabilize at each pixel dwell location. As a result, compared to higher-numbered image stacks, lower-numbered image stacks in the raster mode exhibited a systematic shift of the features imaged, and lower-numbered image stacks in the snake mode exhibited interlaced line defects, represented by ragged feature edges. A one-dimensional discrete Fourier Transform (DFT) based algorithm was developed that aligned each line of pixels in a lower-numbered image stack with the corresponding line of pixels in the last image stack. The snake scan led to lower image distortions; a dwell time > 10 μs was used to stabilize the beam at each pixel location and thereby eliminate interlaced line defects in the last image in the detector sampling TIFF stack. For raster scans, a shorter dwell time (e.g., 3.2 μs) can be applied, as the last image in the detector sampling TIFF stack contained no interlaced line defects and therefore was suitable as a reference to align the detector sampling image set as described in. However, a distinctive distortion near the left edge of the image still existed, and resulted in noise in the strain field. This is illustrated by the high absolute strain (blue) region in Fig. 2(c), and imperfection in image stitching as illustrated in Fig. 3(b). This may be due to the large jump of the beam position from the end of the previous row to the beginning of the current row, and the related error in beam position control.

Fig. 4
figure 4

Illustration of the imaging algorithm with the custom external scan controller. (a) The electron beam was driven by the scan signal to sequentially reach each of the target pixel positions. At each pixel position, the detector signal was sampled multiple times (with the data indicated by the colored squares) and stored in an array. The N-th sample in the array of each pixel was used to construct the N-th image stack. (b) Illustration of the electron beam path in the raster scan mode and snake scan mode

Full size image

To reduce distortion, a new raster scan strategy was applied wherein the beam was paused at the start position of each scan line for an additional wait time equal to 25% of the time required to scan an entire line. This allowed the beam position to stabilize, which was essential to reduce the long range image distortion shown in Figs. 2(c) and 3(b). Comparing to applying a snake scan with >10 μs dwell time, the long range image distortion was reduced to comparable level with the application of the new raster scan strategy with a dwell time of 3.2 μs.

Stage translation, focus, brightness, and contrast adjustment during image capture were automated by a script in the FEI iFast (Developer’s Kit 5.1) interface. The auto stigmation and switching external scan on/off functions were performed through a C++ automation script that sends commands directly to the FEI xT microscope interface. A lab PC was used to operate the DAQ and externally control image capture. To synchronize the imaging parameter adjustment on the microscope PC with the imaging on the lab PC, communication between the two PCs was established through wired connections using custom C++ scripts. Once imaging conditions were adjusted, the microscope PC sent a signal of set frequency with the imaging tile position encoded, for example:

$$ Frequency=1000+100\times r+4\times c $$
(5)

where r = row number and c = column number. The lab PC detected this frequency by a Fourier transform of the time-domain signal to the frequency domain. Once imaging was complete, a signal was sent from the lab PC to the microscope PC to resume stage translation and imaging condition adjustments. The setup of the automated imaging system is illustrated in Fig. 5.

Fig. 5
figure 5

Illustration of the automated imaging system

Full size image

SEM Image Stitching

The image of a 5.7 mm × 3.4 mm sample area was reconstructed by stitching individual image tiles using a custom Matlab script. Due to image distortions and the limited precision of the SEM stage translation control, the relative position of the image tiles could not be accurately determined from the stage positions recorded by the SEM (the manufacturer-stated precision of the Teneo SEM stage is 3 μm repeatability with 1 μm resolution). Instead, the positions were determined using cross-correlation for image registration. Cross-correlation is a measure of the similarity between two signals as a function of the displacement of one signal relative to the other. The cross-correlation of an M-by-N matrix, F, and a P-by-Q matrix, G, representing the grayscale value of two images, is defined as:

$$ \left(F\star G\right)\left(x,y\right)=\sum \limits_{m=0}^{M-1}\sum \limits_{n=0}^{N-1}F\left(m,n\right)G\left(m+x,n+y\right) $$
(6)

where x ∈ [−(M − 1), P − 1], and y ∈ [−(N − 1), Q − 1]. Zero padding is usually applied to elements outside of the original index range. F is referred to as the template or filter.

To demonstrate its use in image matching, cross-correlation for two images is shown in Fig. 6(a) and (b), which have an overlap region indicated by the black box in Fig. 6(a) and (b). The result is shown in Fig. 6(c). A peak found at (0, 0) did not represent the true relative position of the two images, as the cross-correlation exhibited a background noise that was related to the total number of overlapping pixels during the calculation of each element in the cross-correlation matrix [33]. This background noise was estimated by applying a first-order Savitzky-Golay smoothing filter with window size of five to the cross-correlation result. It was then subtracted from the cross-correlation; the result is shown in Fig. 6(d). A distinct peak was found at position (−3383, 11), which represents the true relative position; i.e., image (a) was 3383 pixels to the left and 11 pixels to the bottom of image (b).

Fig. 6
figure 6

Illustration of cross-correlation for image matching. (a) and (b) Two images with an overlap highlighted by the black box. c) Cross-correlation of (a) and (b), calculated according to Eq. (6). d) Cross-correlation after subtracting background noise estimated by a Savitzky-Golay filter with a window size of 5. The inset is a magnified view of a small region near the correlation peak. (e) Normalized cross-correlation according to Eq. (7). f) Normalized cross-correlation after background correction using the Savitzky-Golay filter. g) A region near the normalized cross-correlation peak for two images (not shown here) taken with the stock scan controller. A diagonal streak of high value points can be observed that corresponds to the overlay error discussed in Fig. 3b. The color bar in (c)-(g) indicates the cross-correlation values

Full size image

As discussed by Lewis [37], use of cross-correlation as defined in Eq. (6) for image matching requires the image energy to be approximately constant. This suggests that the effectiveness of SEM image stitching by cross-correlation in the current study was aided by the uniform speckle pattern of the sample and the relatively consistent image quality. Normalized cross-correlation can be used to enhance peak definition:

$$ \gamma \left(x,y\right)=\frac{\sum \limits_{m,n}\left[g\left(m,n\right)-{\overline{g}}_{x,y}\right]\left[f\left(m-x,n-y\right)-\overline{f}\right]}{{\left\{\left\{{\sum}_{m,n}{\left[g\left(m,n\right)-{\overline{g}}_{x,y}\right]}^2{\sum}_{m,n}\right[f\Big(m-x,n-y\left)-\overline{f}\right]{}^2\Big\}\right\}}^{0.5}} $$
(7)

where g is the image, f is the template positioned at (x, y), \( \overline{f} \) is the mean of the template, and \( {\overline{g}}_{x,y} \) is the mean of g(m, n) in the region under the template. The normalized cross-correlation between the images in Fig. 6(a) and (b) is shown in Fig. 6(e). The normalized cross-correlation corrected by the Savitzky-Golay filter is shown in Fig. 6(f). While both peaks were found at the correct position, the peak in the filtered normalized cross-correlation was more distinctive. The application of normalized cross-correlation was found to be effective in aligning grain boundary maps during EBSD data stitching, as shown later in Section EBSD Data Stitching.

To create a stitched image, the cross-correlation algorithm was applied to each neighboring image tile to obtain their relative positions. Taking into account tile size, the size of the stitched image and the position of each image tile in the stitched image were calculated. In regions where multiple image tiles overlapped, stitching was performed using a simple image blending algorithm that averaged the greyscale values at each point. The stitched image was only used to visually confirm the quality of the overlay; if DIC analysis had been performed on the stitched image (versus individual images), it is possible that a more advanced blending algorithm would need to be used to ensure sufficient accuracies for the stitched regions.

A few additional considerations for SEM image stitching are provided in Appendix A.

DIC Displacement and Strain Data Stitching

DIC was performed on individual image tiles, and the resultant displacement and strain fields were stitched together. A practical reason for applying this method, rather than performing DIC on the stitched images, was the large size of the stitched images (~30,000 × 70,000 pixels). The deformation data of individual image tiles were exported as matrices, where the output variables included: the initial pixel positions x and y on the reference image, the displacement values u and v (where the pixel position on the deformed image was (x + u, y + v)), and selected strain values (in this case, Lagrangian strain values εxx, εxy, εyy). Similar to image stitching, each of the deformation data fields of individual image tiles (local matrices) were stitched to form a single, global matrix that represented the deformation data of the stitched image.

To simplify data processing, the global matrices were constructed to describe deformation at evenly spaced positions on the stitched reference image, with a step size equal to that in the DIC analysis. For example, when a step size of 5 pixels was used, the global pixel position matrices were:

$$ X=\left[\begin{array}{cccc}0& 5& 10& \dots \\ {}0& 5& 10& \dots \\ {}0& 5& 10& \dots \\ {}\dots & \dots & \dots & \dots \end{array}\right]\;\mathrm{and}\;Y=\left[\begin{array}{cccc}0& 0& 0& \dots \\ {}5& 5& 5& \dots \\ {}10& 10& 10& \dots \\ {}\dots & \dots & \dots & \dots \end{array}\right] $$

The position of the local matrix in the global matrix was directly related to the position of the DIC reference image tile in the stitched image. For example, if an image tile was stitched to a region starting at pixel position (a, b), the data points at position (x, y) in the local matrix should be stitched to position (x + a, y + b) in the global matrix. However, as position (a, b) was rarely a multiple of the step size used in DIC analysis, a nearest neighbor interpolation was applied, where a and b were rounded to the nearest integer values that were multiples of the step size. For example, if a step size of 5 pixels was used, (a, b) = (3, 4) was rounded to (a, b) = (5, 5). Once the stitch position of the local matrices was determined, the values were used to construct the corresponding subset of the global matrices. The accuracy of this method increased with reduced step size; other interpolation methods are possible if needed.

Stitching of strain fields was more straightforward than stitching of displacement fields, as strain was calculated for each data point using the displacement of itself and its four nearest neighbor data points, then smoothed by a Gaussian filter of a limited size (here, a filter size of five data points was used) [35]. Theoretically, the strain fields in different local matrices should be the same in regions where the local matrices overlap, as strain is not affected by rigid motion. In practice, they are different, due to image noise and distortion. When the image distortion was larger in one of the contributing matrices, i.e., the matrix where the overlap area is on the left side, those values were discarded and the more accurate values of the other matrix were used. If image distortion was relatively consistent across the contributing matrices, the strains were averaged between multiple overlapping matrices.

Stitching displacement matrices required more consideration than stitching strain matrices. As local matrices represent the displacement of individual deformed image tiles with respect to the corresponding reference image tiles, and global matrices represent the displacement of the stitched deformed image with respect to the stitched reference image, the displacements depend on the physical displacement of the trackable features and on the origin of each image. This relationship is illustrated in Fig. 7. A trackable feature is imaged at position (xi, yi) on the reference image tile, and at position (xf, yf) on the deformed image tile. Therefore, the displacement values (u′, v′) in the local matrices are:

$$ \left({u}^{\prime },{v}^{\prime}\right)=\left({x}_f-{x}_i,{y}_f-{y}_i\right) $$
(8)
Fig. 7
figure 7

Illustration of the effect of image tile stitch position on the displacement values. A single reference image tile and the stitched reference image are represented by red solid rectangles. A single deformed image tile and the stitched deformed image are represented by blue dashed rectangles. A trackable feature is represented by a circle before (solid red) and after (dashed blue) deformation. All images have their own local coordinate system with the origin located at their upper-left corner. The position of the feature is (xi, yi) on the reference image tile, and (xf, yf) on the deformed image tile. The stitch position of the reference image tile is (xr, yr). The stitch position of the corresponding deformed image tile is (xd, yd). The relationship between the displacement values measured from the image tiles and from the stitched images is discussed in the text

Full size image

The position on the stitched image is (xr, yr) for the reference image tile, and (xd, yd) for the deformed image tile. Therefore, on the stitched image, the position of the feature is (xr + xi, yr + yi) in the reference image, and (xd + xf, yd + yf) in the deformed image. The displacement values (u, v) in the global matrices are:

$$ {\displaystyle \begin{array}{rl}\left(u,v\right)& =\left(\left({x}_d+{x}_f\right)-\left({x}_r+{x}_i\right),\left({y}_d+{y}_f\right)-\left({y}_r+{y}_i\right)\right)\\ {}& =\left({u}^{\prime }+\left({x}_d-{x}_r\right),{v}^{\prime }+\left({y}_d-{y}_r\right)\right)\end{array}} $$
(9)

Therefore,

$$ {u}^{\prime }=u-\left({x}_d-{x}_r\right) $$
(10)
$$ {v}^{\prime }=v-\left({y}_d-{y}_r\right) $$
(11)

For a given feature, the displacement measured from the stitched image is a set value. Therefore, it can be seen from Eqs. (10) and (11) that the displacement measured from a given image tile (in local matrices) is dependent on the stitch position (xr, yr) and (xd, yd). For a feature located in the overlap area, as the stitch positions of the image tiles are different, the displacements measured from different image tiles are also different. Such difference in the displacement data between overlapping image tiles is illustrated in Fig. 8(a), which plots the horizontal displacement fields of two neighboring image tiles. In the overlap area, half of the displacement field was plotted using the values of image tile 1, and the other half using the values of image tile 2. A discontinuity was observed in the middle of the overlap area.

Fig. 8
figure 8

Displacement fields of different image tiles need to be corrected during data stitching. (a) The displacement field measured from two image tiles with a discontinuity evident in the overlap area. In the overlap area, half of the displacement field was plotted using values from image tile 1 and the other half using values from image tile 2. (b) Correction minimized the discontinuity in the displacement field

Full size image

To avoid this stitching discontinuity, the values in the local matrices were corrected using Eq. (9), such that the displacement fields in different local matrices become the same in the overlap region. When stitching local matrices to the global displacement matrices, the values (xd - xr) and (yd - yr) were added to the local displacement matrices u’ and v’. As shown in Fig. 8(b), this correction minimized the discontinuity in the displacement field. Note that the discontinuity shown in Fig. 8(a) is mainly due to the fact that reference and deformed image tiles have their origins located at different positions on the stitched image. The stitching defect being corrected here is different from that in SEM image distortion [15, 16, 26,27,28].

The displacement field stitching method introduced above requires stitching both reference and deformed image tiles. An alternative approach that requires only stitching reference image tiles is also possible, as follows: As discussed, the stitch position of the local matrices only depends on the stitch position of the reference image tiles. The purpose of displacement field stitching is to equalize the local displacement matrices in the overlap region. Consider the horizontal displacement u as an example. Each local matrix needs to be corrected by adding a specific constant that represents the difference between the stitch position of the deformed image and reference image tiles (xd − xr), as discussed previously. However, this approach solves for this constant as an unknown variable. Taking into account all image tiles, a least squares method was used to solve for all constants required to correct the local displacement matrices. The sum of squared difference between the corrected displacement values in all the overlap areas was expressed as:

$$ E=\sum \limits_i\sum \limits_j\sum \limits_{k=1}^{Kij}{\left({u}_{ik}+{U}_i-{u}_{jk}-{U}_j\right)}^2 $$
(12)

where i and j are the indices referring to different local matrices, k is the element index in the overlap region of the matrices, Kij is the total number of overlap elements between matrices i and j, and Ui and Uj are the constants to be added to matrices i and j to correct the displacement value. A minimized E indicates that the corrected displacement fields from different local matrices exhibit minimal difference in the overlap region. The minimization of E is equivalent to setting the derivative of E with respect to all the Ui, Uj, ... values equivalent to zero, which results in an overdetermined system of linear equations. By solving this system, the constants used to correct the individual local matrices were obtained. Similar to stitching strain fields, the corrected displacement values can be either averaged in the local matrix overlap region or selected preferentially, depending on the distortion of the image tiles.

Figure 9 shows the DIC-obtained displacement and strain maps over a very large area of 5.7 mm × 3.4 mm, constructed by stitching 209 matrices of individual 360 μm × 360 μm tiles. The corresponding macroscopic strain was approximately −3.7%. The step size between each neighboring data point was five pixels, which is approximately 0.44 μm. Deformation mechanisms including dislocation slip and twinning are clearly resolved, as shown in the inset of Fig. 9(b), where slip traces are shown as sets of sharp, thin, and straight lines with highly localized strain, and twinned areas inside a grain are observed as a region with a relatively constant strain that is different from the parent grain, with a lenticular shape and sharp edges.

Fig. 9
figure 9

The DIC-obtained (a) displacement and (b) strain maps of the 5.7 mm × 3.4 mm area of interest of the Mg WE43 sample were constructed by stitching 209 matrices of individual 360 μm × 360 μm tiles. The step size between each neighboring data point is 0.44 μm. The region highlighted by the black box has a size equal to that of a single image tile, and is located at the intersection of four neighboring image tiles

Full size image

EBSD Data Stitching

Electron backscatter diffraction (EBSD) data was collected from the same sample area as the deformation maps by an automated multi-tile scan procedure supported by the built-in functions of the TSL OIM Data Collection software. A total of 135 (5 × 27) EBSD data tiles were collected. As previously discussed, the individual EBSD maps contain distortions, and the precision of the SEM stage control in the multi-tile EBSD scan is limited. Therefore, instead of using the stitch function in the TSL OIM Analysis 7 software, the multi-tile EBSD data was stitched using a custom script.

First, a rough cleanup by grain dilation was performed using the OIM Analysis software. A grain boundary map was generated for each tile, and the same normalized cross-correlation algorithm as that in image stitching was used to find the stitch position of each data tile in the global map. A confirmation step allowed for additional manual adjustment, if needed, of the tile position. After the stitch positions were confirmed, the raw data of all tiles were stitched together to construct a global data set, and exported as a single .ang file for analysis in the OIM Analysis software. Figure 10 shows the global grain boundary map, where averaged grayscale values were used in the overlap areas of individual tiles. As shown in Fig. 10(a), assuming a constant translation between neighboring tiles led to a misalignment of grain boundaries, particularly in the last few columns of the second row of the tiles. The EBSD stitching method described here significantly improved grain boundary alignment, demonstrated in Fig. 10(b).

Fig. 10
figure 10

Global grain boundary map constructed by stitching individual tiles. The upper left corner of each tile is indicated by a white square. Average grayscale values were used in the overlap area of neighboring tiles. (a) Assuming a constant translation between neighboring tiles led to grain boundary misalignment in the global map, as highlighted by the white box. (b) EBSD stitching resulted in improved grain boundary alignment

Full size image

The image quality (IQ) and confidence index (CI) were averaged in the overlap regions between local matrices, and set to zero in regions outside of the area of interest to facilitate data processing. Due to the time-consuming nature of averaging the crystal orientation, which requires calculation in the orientation space, the Euler angles were copied from individual data tiles to the global data set. The data in the overlap area were overwritten with the data last stitched to the global data set. The procedure to calculate the average grain orientation was to first select a seed orientation, then convert every other orientation to the nearest symmetric equivalent orientation to the seed orientation, and then calculate the average orientation in the quaternion space, and finally convert it back into Euler angles.

When determining the stitch position of individual data tiles using the normalized cross-correlation algorithm, maps other than the grain boundary map can also be used; for example, maps of the IQ, CI, or misorientation with respect to a given orientation are suitable. However, these did not provide better results than grain boundary maps in the current study.

DIC-EBSD Data Alignment

To facilitate the correlation between the deformation and the underlying microstructure, the stitched SEM-DIC data and the EBSD data were aligned as shown in Fig. 11. As the SEM images exhibited less distortion than EBSD maps, the position of the EBSD data was transformed from the EBSD to the DIC data coordinate system.

Fig. 11
figure 11

EBSD grain boundary map for the sample area projected onto the DIC data coordinate system and overlaid with the stitched εxx Lagrangian strain map at a global uniaxial compressive strain of −4.7%

Full size image

A projective transformation was used for alignment. This transformation maps lines to lines but does not necessarily preserve parallelism. N control point pairs corresponding to the same physical locations on the sample were manually selected on both the strain and EBSD maps. Assume the control points on the EBSD map have coordinates (xi, yi), where i ∈ [1, N], and the corresponding control points on the DIC map have coordinates (Xi, Yi). A projective transformation can be represented by a 3-by-3 matrix H, where

$$ H=\left[\begin{array}{ccc}{h}_1& {h}_4& {h}_7\\ {}{h}_2& {h}_5& {h}_8\\ {}{h}_3& {h}_6& {h}_9\end{array}\right] $$
(13)

where h9 = 1, and the following relationship is satisfied for each pair of control points:

$$ {X}_i=\frac{x_i{h}_1+{y}_i{h}_2+{h}_3}{x_i{h}_7+{y}_i{h}_8+{h}_9} $$
(14)
$$ {Y}_i=\frac{x_i{h}_4+{y}_i{h}_5+{h}_6}{x_i{h}_7+{y}_i{h}_8+{h}_9} $$
(15)

Rearranging the relationship for all control point pairs results in the following system of linear equations:

$$ \left[\begin{array}{c}{X}_1\\ {}{Y}_1\\ {}{X}_2\\ {}{Y}_2\\ {}\dots \\ {}{X}_n\\ {}{Y}_n\end{array}\right]=\left[\begin{array}{cccccccc}{x}_1& {y}_1& 1& 0& 0& 0& -{X}_1{x}_1& -{X}_1{y}_1\\ {}0& 0& 0& {x}_1& {y}_1& 1& -{Y}_1{x}_1& -{Y}_1{y}_1\\ {}{x}_2& {y}_2& 1& 0& 0& 0& -{X}_2{x}_2& -{X}_2{y}_2\\ {}0& 0& 0& {x}_2& {y}_2& 1& -{Y}_2{x}_2& -{Y}_2{y}_2\\ {}\dots & \dots & \dots & \dots & \dots & \dots & \dots & \dots \\ {}{x}_n& {y}_n& 1& 0& 0& 0& -{X}_n{x}_n& -{X}_n{y}_n\\ {}0& 0& 0& {x}_n& {y}_n& 1& -{Y}_n{x}_n& -{Y}_n{y}_n\end{array}\right]\left[\begin{array}{c}{h}_1\\ {}{h}_2\\ {}{h}_3\\ {}{h}_4\\ {}{h}_5\\ {}{h}_6\\ {}{h}_7\\ {}{h}_8\end{array}\right] $$
(16)

which is abbreviated as

$$ Q= PA $$
(17)

A projective transformation requires four control point pairs. If more than four control point pairs were provided, i.e., N > 4, Eq. (16) became an overdetermined system of linear equations. Therefore, matrix A was solved by the generalized inverse to average the effect of more than four control point pairs:

$$ A={\left({P}^TP\right)}^{-1}{P}^TQ $$
(18)

which provided all the required elements to define the transformation matrix H.

During an EBSD scan, it is possible for the sample to have out-of-plane misalignment. Therefore, a projective transformation was used to align a single EBSD map with a SEM image. The global EBSD scan was composed of multiple tiles (captured using SEM stage control) that could all be modeled by similar projective transformations, and the overall distortion could not be described by a single projective transformation. Performing projective transformation piecewise, i.e., separately on multiple regions of the whole EBSD map, requires a significant number of control points and would be prone to cause discontinuities in the transformed map. Therefore, the averaged projective transformation was applied. Note that the alignment was performed only to the position of the data points, and the Euler angles were not modified, as the correction could not be determined from the projective transformation.

The ability to obtain high-resolution deformations over statistically relevant fields of view enables large data analytics to examine interactions between microstructure, microscale strain localizations, and macroscopic properties. Two examples of the analyses made possible by this approach are shown in Fig. 12. Figure 12(a) shows the distribution of effective strain versus distance to the grain boundary for all data points at a globally applied uniaxial compressive strain of −3.7%. The effective strain is defined as:

$$ {\varepsilon}_{eff}=\sqrt{\frac{2}{3}\left({\varepsilon}_{xx}^2+{\varepsilon}_{yy}^2+2\times {\varepsilon}_{xy}^2\right)} $$
(19)

where εxx, εxy, and εyy are the three components of the experimental Lagrangian strain tensor, and uniaxial compressive loading was applied along the x-direction.

Fig. 12
figure 12

Box plots of the effective strain distribution versus (a) distance to the grain boundary, and (b) basal Schmid factor, for all data points in the tested sample at a globally applied compressive strain of −3.7%. The boxes indicates the 25th, 50th, and 75th percentiles of the sample distribution. The end of the whiskers indicates the ±3σ of the sample distribution

Full size image

The effective strain distribution was found to be relatively uniform inside the grains, with only a slight decrease as distance to the grain boundary increased as seen in Fig. 12(a). The basal Schmid factor was found to have a larger impact on the effective strain distribution, as shown in Fig. 12(b). Regions with high basal Schmid factor exhibited higher effective strains, as evident in the box plot of Fig. 12(b). Interestingly, regions with a small basal Schmid factor (< 0.2) also exhibited higher effective strains, which may be related to increased twinning activity.

Summary

An experimental approach to map high-resolution, sub-grain deformations across mm-scale fields of view, and to investigate their interactions with the surrounding microstructure, was presented and demonstrated on the magnesium alloy WE43 under uniaxial compressive loading. In the demonstrated test case, deformation mechanisms including dislocation slip and twinning are resolved with a spatial resolution of 0.44 μm in a 5.7 mm × 3.4 mm field of view containing 100 million (7678 × 13,004) individual data points. This was constructed by stitching 209 low-distortion micrographs of 360 μm × 360 μm each. This required the creation of new approaches for distortion mitigation, testing automation, and displacement/strain/image field stitching, which are described in detail herein.

This approach enables the investigation of the spatial evolution of microscale deformation and damage under thermo-mechanical loading across a wide range of materials. Examples of the broad range of analysis made possible by this approach are demonstrated in this paper, including the analysis of effective strain distribution versus distance to the grain boundary and basal Schmid factor of the sample when globally subjected to a − 3.7% uniaxial compressive strain, as representative examples shown in Fig. 12. The ability to obtain high-resolution deformations over statistically relevant fields of view enables the future application of large data analytics to examine interactions between microstructure, microscale strain localizations, and macroscopic properties.