1 Introduction

A complex amplitude distribution gives us important information such as surface profiles, internal structure, and the refractive index of objects. A lot of techniques such as phase shifting digital holography [1, 2], iterative algorithms [3, 4], and the transport of intensity equation (TIE) [5, 6] have been proposed to reconstruct the complex amplitude. When we observe dynamic objects or there is optical misalignment, target objects are often out of focus. Autofocusing techniques have been proposed to detect the object positions automatically. The study of the autofocusing is thriving particularly in digital holography. Many algorithms have been proposed, such as amplitude analysis [7,8,9], Fourier spectrum analysis [7, 10], and so on [11]. These algorithms are capable for other reconstruction techniques [12]. However, the techniques allows to detect a position of object(s) in only a single plane. Therefore, it is difficult to obtain an in-focus complex amplitude of an object with three-dimensional structure or ones in different depth positions. This is disadvantageous for practical applications such as cell observation, particle measurements, and inspection of industrial products.

McElhinney and coworkers have been proposed a method which allows to obtain an extended-focused image based on a complex amplitude measurement, numerical propagation, and local sharpness evaluation [13]. In this conventional method, a complex amplitude measurement is performed at first. A measured complex amplitude distribution is propagated and an amplitude stack is obtained. The sharpness of amplitude is evaluated within an area of \(n \times n\) pixels centered on a pixel of interest. The sharpness at different depth positions is compared, and an object position is determined for the pixel. The above calculation is implemented for all pixels. Therefore, object positions are locally determined and a final amplitude result focuses on all objects. However, this method can obtain only an amplitude distribution (not a complex amplitude distribution). In addition, the method chooses calculation area size n of the sharpness evaluation empirically, though it affects on the accuracy of the determination of the object positions.

We propose here adaptive autofocusing algorithm (AAA) to measure a complex amplitude distribution which focuses on three-dimensional structure or multiple objects over an entire field of view. The proposed method locally uses a numerical autofocusing technique [7] similar to the conventional method [13], except that the calculation area size n is determined automatically. “Adaptive” has double meaning that autofocusing is adaptively implemented for pixelwise and the optimal calculation area size n is adaptively chosen. Moreover, coefficient of variation (CV) is used for the sharpness evaluation instead of the conventional variance, allowing to improve the accuracy of the object plane determination. Although any methods can be used for a step of a complex amplitude measurement in the proposed method, in this paper, we use the TIE [5, 6] because of its experimental simplicity. Owing to the simplicity, the TIE is recently applied to phase retrieval in computational ghost imaging [14] and phase detection of the holographic data storage [15]. We have already reported autofocus TIE [12] for object plane detection. The autofocus TIE detects a position of an object or objects in a single plane. Thus, it can be said that the method that we describe here is extension of the autofocus TIE for detection of different depth positions of multiple objects or three-dimensional structure. The proposed method is expected to be effective for cell observation [16, 17], particle measurements [18, 19], industrial inspection [20], and so on.

This paper is organized as follows. In Sect. 2, principle of the TIE-based phase measurement and the proposed AAA is described. In Sect. 3, we demonstrate an experiment by the proposed method. Finally, we provide our conclusion in Sect. 4.

2 Principle

In the proposed method, first, a complex amplitude distribution is obtained in an image sensor plane. Then, an amplitude stack is calculated through numerical propagation of the obtained complex amplitude, and object positions are determined by local sharpness evaluation with adaptive area size n.

2.1 Complex amplitude measurement by transport of intensity equation

An experimental setup shown in Fig. 1 is used for the phase measurement by the TIE. Here, we briefly review the TIE according to the literature [21]. The TIE is given by the following:

$$\begin{aligned} \nabla _{\perp } \cdot [I_{z}(x,y)\nabla _{\perp } \phi _{z}(x,y)] = -\dfrac{2\pi }{\lambda } \dfrac{\partial I_{z}(x,y)}{\partial z}, \end{aligned}$$
(1)

where \(\lambda \) denotes a wavelength of a light source. \(I_{z}(x,y)\) and \(\phi _{z}(x,y)\) denote intensity and phase distributions at an arbitrary position of an optical axis z, respectively. The notation \(\nabla _{\perp }\) denotes a gradient operator in a transverse plane (x, y). An image sensor plane is set to \(z=0\). We solve the TIE for the phase distribution using the Fourier transforms [21] and the following equation is given by the following:

$$\begin{aligned} \phi _{0}(x,y)= & {} -\dfrac{2\pi }{\lambda }\ \mathrm{IFT}\Bigg [\dfrac{1}{4\pi ^{2}(u^{2}+v^{2})}\ \mathrm{FT}\Bigg [\nabla _{\perp }\cdot \dfrac{\nabla _{\perp }}{I_{0}(x,y)}\nonumber \\&\times \mathrm{IFT} \Bigg [\dfrac{1}{4\pi ^{2}(u^{2}+v^{2})}\ \mathrm{FT}\Bigg [\dfrac{\partial I_{0}(x,y)}{\partial z}\Bigg ]\Bigg ]\Bigg ]\Bigg ], \,\ \ \ \end{aligned}$$
(2)

where u and v denote spatial frequencies corresponding to x- and y-directions, respectively. The operators FT[...] and IFT[...] denote Fourier and inverse Fourier transforms, respectively. The intensity derivative with respect to an optical axis \(\partial I_{0}(x,y)/\partial z\) in the right-hand side of Eq. (2) can be approximated by the finite difference method [22] using two intensity distributions captured at different depth positions \(z = \pm \varDelta z\):

$$\begin{aligned} \dfrac{\partial I_{0}(x,y)}{\partial z} \approx \dfrac{I_{\varDelta z}(x,y)-I_{-\varDelta z}(x,y)}{2\varDelta z}. \end{aligned}$$
(3)
Fig. 1
figure 1

Optical setup for a complex amplitude measurement

Full size image

The intensity distributions \(I_{0}(x,y)\), \(I_{\varDelta z}(x,y)\), and \(I_{-\varDelta z}(x,y)\) are captured by scanning an image sensor along the z-axis, as shown in Fig. 1. Using the captured intensity distributions, Eqs. (2), and (3), the phase distribution \(\phi _{0}(x,y)\) is retrieved. Finally, the complex amplitude distribution \(\sqrt{I_{0}(x,y)}\exp \{i\phi _{0}(x,y)\}\) in the sensor plane is calculated from the captured intensity distribution \(I_{0}(x,y)\) and the retrieved phase distribution \(\phi _{0}(x,y)\). This complex amplitude distribution is used for the input to the proposed algorithm.

2.2 Adaptive autofocusing algorithm

The schematic diagram of AAA is shown in Fig. 2. First, (i) the complex amplitude in the sensor plane is propagated by numerical diffraction integral based on the angular spectrum [23], and a propagated amplitude stack \(a_{z}(x,y)\) is obtained. Here, the suffix z denotes the different depth positions. Therefore, \(a_{z}(x,y)\) includes the propagated amplitude distributions at different depth positions. Next, (ii) the sharpness is evaluated within an \(n \times n\) [pixels] area centered on an interested pixel (x, y) by the focus value of the CV:

$$\begin{aligned} C_{z}(x,y) = \dfrac{1}{\overline{a_{z}}} \sqrt{ \dfrac{1}{n^{2}} \sum _{i,j=-(n-1)/2}^{+(n-1)/2} \left[ a_{z} ( x+i,y+j )-\overline{a_{z}} \right] ^{2} }, \end{aligned}$$
(4)

where \(\overline{a_{z}}\) is the mean value of the amplitude in the area. The calculated values are compared along with the z direction. The depth position where a maximum or minimum focus value appears is determined as the object position. Based on a priori knowledge of objects, we decide which focus value should be chosen. If we measure objects with uniform amplitude or no amplitude information (i.e., pure phase objects), the minimum focus value is adopted. In the case where objects have fine structure of amplitude, the maximum value is adopted. (iii) A depth map is obtained by repeating the steps (i) and (ii) for all pixels. Finally, the complex amplitude distribution is obtained from the depth map by stitching corresponding complex values together. Figure 3 shows behavior of the focus value (such as variance) along the optical axis regarding a pixel of interest. In the case of fine-structured amplitude objects, we find the maximum focus value along the z axis. The area size n is fixed and determined empirically in the conventional method [13], as shown in Fig. 3a. The graph of the conventional method does not have an axis of n. The size depends on the object itself, and thus, it is difficult to determine optimal one. Moreover, the optimal size is inconstant if the object has a complicated shape [13, 24]. In the proposed method, n is determined adaptively and automatically, as shown in Fig. 3b. The focus value is a function of not only the propagation distance but also the calculation area size, and we find the maximum value in the distribution. The CV is used in this study as a focus value instead of the conventional variance [13], because the divisor \(\overline{a_{z}}\) can normalize the CV value at each area size.

Fig. 2
figure 2

The procedure of the proposed AAA

Full size image
Fig. 3
figure 3

The behavior of the focus value of sharpness evaluation at a pixel along the optical axis. The determination of a depth position through sharpness evaluation with a the conventional [13] and b the proposed methods. This is the example of finding the maximum focus value

Full size image

The final complex amplitude distribution contains in-focus information over the field of view. Note that bias phase values (background terms) are subtracted from corresponding phase distributions in each plane of different depth positions before the complex amplitude distribution is reconstructed; otherwise, the proper phase distribution cannot be obtained in the final result. This point is lacked in the conventional method [13].

3 Experimental results

The optical experiment was carried out by the proposed method. The complex amplitude of living leaves of the moss Physcomitrella patens (Physcomitrella) was measured. Physcomitrella is one of the model plants suitable for molecular, cellular, and developmental biology because of its sequenced genome, high efficiency of gene targeting, clear cell identity, and strong ability of stem cell formation [25,26,27,28]. Therefore, we chose Physcomitrella as initial observation target. Two leaves of the moss were set on a glass slide. To give difference of depth positions, the slide was located at an angle of 60\(^\circ \) with respect to the optical axis, as shown in Fig. 4. The laser with the wavelength of 532 nm was used as a light source. The distance between the center of the slide and the image sensor plane was 38 mm. The three intensity distributions were captured by scanning the image sensor with pixel size of \(4.65 \,\upmu \hbox {m} \times 4.65 \upmu \hbox {m}\). These intensity distributions are shown in Fig. 5a. Note that the defocus distance \(\varDelta z\) was determined as 1 mm by considering the noise condition [29]. Namely, the effect of the noise is expected to be suppressed at the defocus distance.

Fig. 4
figure 4

The experimental setup. The glass slide was located at an angle of 60\(^\circ \). with respect to the optical axis. The yellow-broken-lined area denotes the field of view of this experiment, where two living Physcomitrella leaves were placed with water

Full size image
Fig. 5
figure 5

Experimental results: a the captured intensity distributions, b the phase distribution at the sensor plane retrieved by the TIE, c the area size n map, and d the depth map. The color bars of c and d denote the area size n and the distance from the image sensor plane, respectively

Full size image

The phase distribution at the sensor plane is retrieved by the TIE, as shown in Fig. 5b. Using the proposed AAA with the strategy of finding the maximum CV value, the object positions are determined pixel by pixel. Here, the maximum n is set to 155 pixels which can cover the thick region of the objects. The depth resolution of the numerical propagation is set to 0.1 mm with reference to the depth of field of the optical setup. For our experimental condition, the calculation cost is 2 days with Intel Core i5 2.8 GHz. The computational cost depends on the range of n (3–155) and the propagation distance (36–40 mm). These values are changed in the for-loop. In this study, the for-loop is calculated sequentially and spends much time. However, the calculation cost can be considerably reduced by parallel computing with GPU (graphics processing unit). Figure 5c shows the map of adopted area size n, and the depth map is obtained as shown in Fig. 5d. As shown in Fig. 5c, n is different pixel by pixel. In particular, smaller n is adopted in the head region (smaller region) of the object as shown in the magnified image. We estimate the tilt angle of the objects with respect to z from Fig. 5d. The difference between the determined depth positions of the left edge of Leaf 1 and the right edge of Leaf 2 is 1.9 mm. The distance between the edges along x axis is 3.3 mm. From these distances, the tilt angle is calculated as \(\arctan (3.3/1.9) = 60^\circ \). which corresponds to the set angle. Therefore, the depth determination is achieved by the proposed method. We obtain the focused complex amplitude by stitching the corresponding complex values with reference to the depth map.

Fig. 6
figure 6

Complex amplitude distributions obtained by a the proposed method and b direct propagation of 38 mm

Full size image

At first, the complex amplitude is compared with that of direct propagation, as shown in Fig. 6. The amplitude and phase distributions in Fig. 6b are obtained by the numerical back-propagation with distance of 38 mm. The amplitude distribution obtained by the proposed method is focused in the entire field of view, as shown in Fig. 6a, whereas the direct propagation gives a blurred amplitude distribution. Furthermore, defocus-caused phase errors appear at the edges of the objects in Fig. 6b. The errors can be reduced by the proposed method. Strong phase difference at approximately half \(\lambda \) (red dots in the leaves) is most likely caused by chloroplasts as shown previously [30].

Fig. 7
figure 7

Depth maps obtained by a the proposed method and constant area size n of b 77 and c 155 pixels

Full size image

Next, we compare the results with that of constant \(n=77\) and 155 [pixels]. Figure 7 shows the comparison of the depth maps. In the results of constant n, depth positions are not determined at edges of the images (blank areas are there), because the CV cannot be calculated for \((n-1)/2\) [pixels] at the edges of the images. For the proposed method, the minimum size of \(3\times 3\) pixels can be used. Therefore, the depth information is lost at only 1 pixel. In the left magnified depth map of 77 pixels shown in Fig. 7b, wrong positions are determined in the thick region of the object (denoted by the yellow-broken line). It is caused that the area size is not sufficiently large in the region. In contrast, the proposed method can cover the region. The right magnified map in Fig. 7a shows that the proposed method gives depth position corresponding to the shape of the leaf (denoted by the orange-broken-lined area), while almost uniform map is obtained in the area by large n of 155 pixels. Smaller n should be used in this thin region of the object. Nevertheless, the detection of the positions is failed at some points in the depth map of the proposed method (particular in the left magnified image shown in Fig. 7a). These errors appear at the fine-structured parts of the objects. In this study, we used the TIE-based phase imaging, which is based on the paraxial approximation. The failure is caused by the loss of high spatial frequency information.

Figure 8a, b shows complex amplitude distributions obtained with the area size of 77 and 155 pixels, respectively. Comparing the amplitude distributions in Figs. 6a and 8a, similar distributions are obtained. A blurred amplitude distribution is obtained by the constant n of 155 pixels, as shown in Fig. 8b. Especially, blurring appears in the head of the object, as shown in the right magnified image. The defocus-caused errors can be seen in both the left magnified phase images shown in Figs. 8a, b. The proposed method comparatively reduces the errors.

According to the experimental results, the proposed AAA successfully determine object positions. In addition, the proposed method enables us to obtain focused complex amplitude with better quality than the conventional method using constant n.

Fig. 8
figure 8

Complex amplitude distributions obtained by constant area size n of a 77 and b 155 pixels

Full size image

4 Conclusion

We have proposed Adaptive Autofocusing Algorithm, termed AAA, to obtain object positions and a fully focused complex amplitude. The optical experiment was carried out by placing the glass slide with living leaves of the moss Physcomitrella patens at a tilt with respect to a (x, y)-plane. The experimental results show that the proposed method can provide the object positions pixel by pixel. The focused complex amplitude can be obtained by the proposed method with better quality than using constant area size. Recently, a biomedical field requires performance of both a wide field of view and high resolution. High NA lenses lead to a shallow depth of field. The proposed method has possibility to overcome the problem.