1 Introduction

Archer fish (genus Toxotes) exhibit multiple sophisticated prey capture strategies. These fish combine spitting, rapid in-water pursuit, and jumping to feed in competitive environments (e.g., Bekof and Dorr 1976; Davis and Dill 2012; Rischawy et al. 2015). Of particular hydrodynamic interest is the fish’s ability to jump multiple times its body length out of the water to capture prey (Shih et al. 2017). Archer fish initiate jumps from directly below the surface, leaving limited space to accelerate before exiting the water completely. Using high-speed imaging, Shih et al. (2017) observe that jumping archer fish use oscillatory tailbeat kinematics, coupled with rapid activity of additional fins at jump onset. Shih et al. (2017) further present 2D particle image velocimetry (PIV) measurements, which suggest that multiple fins contribute upward thrust but that some fins serve more to stabilize and steer the body. Such control is crucial to enabling the fish to accurately capture its aerial prey. To understand the biomechanics of this behavior, as well as any potential for engineers to replicate these aquatic launches, it is necessary to determine the relative importance of each fin and body behavior to propelling, steering, and stabilizing the fish. Any interactions between the fins must also be considered.

Fig. 1
figure 1

Fin activity in jumping archer fish. Shaded and outlined regions show the motions of the caudal (blue dashed–dotted), anal (purple dotted), dorsal (green solid), and pectoral (red dashed) fins at jump onset. The thick gray line shows the location of the free surface. Images are shown 0.01 s apart. Background subtraction and linear contrast enhancement have been applied to the images for visibility

Fins of particular interest include the dorsal, anal, and caudal fins (i.e., the median fins) located on the aft end of the fish body, and the pair of pectoral fins, located midbody near the fish’s center of mass. Figure 1 shows four high-speed images of a jumping archer fish taken 0.01 s apart with the dorsal, anal, caudal, and pectoral fins labeled. The caudal fin is deflected laterally towards one side of the body before the jump begins. When the fish initiates a jump, the pectoral fins extend, while the caudal, anal, and dorsal fins oscillate as propulsive waves travel along the body.

Lauder (2015) summarizes extensive previous studies of these fins in other species of fish, especially in forward swimming and rapid maneuvering contexts. These studies reveal how fin use and specific hydrodynamic functions depend heavily on both fish morphology and the particular swimming scenario. For instance, Standen and Lauder (2005) find varying amounts of dorsal and anal fin activity in bluegill sunfish depending on the forward swimming speed. In a C-start acceleration, Borazjani (2013) finds that the hydrodynamic force contributions of the dorsal and anal fins are greatest at one instance between preparatory and propulsive stages and that the caudal fin contributes substantial force during the propulsive stage.

In the case of the jumping archer fish, jump height and swimming speed are closely related. The archer fish trajectory is effectively ballistic once out of the water, and faster exit velocities are therefore needed to reach higher prey heights (Shih et al. 2017). Shih et al. (2017) show that the jump height increases with the number of propulsive tailbeats executed by the fish, one mechanism for controlling swimming speed at water exit. In this previous study, propulsion from each tailbeat could not be assessed quantitatively using 2D PIV; variation of the fish’s position within the light sheet limited comparison of fin wakes with respect to jump height or prey capture success.

Volumetric particle image velocimetry techniques provide simultaneous measurements of multiple propulsors involved during locomotive behaviors. Previous studies have utilized various 3D velocimetry techniques to study novel and complex swimming strategies, including holographic particle tracking of feeding and sinking copepods (Malkiel et al. 2003), defocusing digital particle tracking velocimetry (DDPTV) of fin and jet propulsion combinations in squid (Bartol et al. 2016), and tomographic PIV of sea butterfly parapodia (Murphy et al. 2016; Adhikari et al. 2016). In a 3D study of forward bluegill sunfish swimming, Flammang et al. (2011) use DDPTV to observe assimilation of upstream vortices from the dorsal and anal fins into the caudal fin wake. Volumetric techniques also reduce artificial experimental constraints on animal behavior, as utilized by Adhikari and Longmire (2013) for the study of zebrafish prey capture. In addition, analysis of 3D data can be performed in reference frames other than a single measurement plane, as shown for fish wakes by Mendelson and Techet (2015).

Synthetic aperture particle image velocimetry (SAPIV) is a volumetric PIV technique that uses light field imaging to reconstruct fields of tracer particles in 3D. Multiple cameras are used to emulate the effects of a single camera with a wide aperture and narrow depth of field scanning through a volume; particles are localized by where they appear in focus. As originally developed, SAPIV uses a particle reconstruction procedure of warping images from multiple views using transformations that correspond to a finely-spaced range of depths (Belden et al. 2010). The transformed images at each depth are then averaged according to

$$\begin{aligned} I_{\text {SA}_k} = \frac{1}{N}\sum _{i=1}^{N}I_{{\text {FP}}_{ki}}, \end{aligned}$$
(1)

where \(I_{\text {SA}_k}\) is the averaged image on the kth focal plane, N is the number of cameras, and \(I_{\text {FP}_{ki}}\) is the transformed image from the ith camera. Image averaging, known as additive refocusing, is followed by intensity thresholding of each focal plane (collectively known as the focal stack) to remove the dim, discrete image artifacts formed when a particle’s location does not converge between multiple cameras at that specific depth (Belden et al. 2010). Belden et al. (2010) use a threshold of three standard deviations above the mean image intensity on each focal plane as the minimum brightness of a valid particle. Intensity normalization of particles within and across all images during preprocessing is crucial to retaining valid particles when thresholding. The stack of all thresholded focal planes is the final 3D particle volume for PIV processing.

The non-iterative and highly-parallelizable algorithm used for SAPIV reconstructs particle volumes faster than the iterative MART variants commonly used in tomographic PIV. In refractive media, reconstruction is also accelerated by using the homography-fit method to reduce the computational cost of image transformations for each focal plane (Bajpayee and Techet 2017). The reconstruction speed of SAPIV presents an advantage for animal studies, where a significant quantity of trials from multiple specimens is ultimately desired.

Using the additive refocusing algorithm [Eq. (1)], a large number of viewpoints (typically eight to ten) is necessary for a sufficient signal-to-noise ratio when thresholding images to identify valid particles. Belden et al. (2010) determines the necessary camera array size using the reconstruction quality factor Q, a metric that isolates the influence of particle volume reconstruction on 3D PIV measurements. However, Bajpayee and Techet (2015) show that velocity field accuracy does not follow the same trends as the particle reconstruction quality when camera spacings are varied or the number of cameras used for SAPIV is reduced. Scenarios with fewer than nine cameras can yield accurate velocity information, especially when alternate refocusing algorithms for SAPIV are also considered (Bajpayee and Techet 2015). Some specific types of reconstruction errors, however, have well-characterized detrimental effects on 3D PIV measurements. For instance, ghost particles (i.e., false particles formed by the coincidental convergence of multiple viewpoints at a 3D location where no tracer particle exists) can reduce measured velocity gradients when actual particle displacements are small (Elsinga et al. 2011). These previous reconstruction studies all consider scenarios where the measurement volume is occupied entirely by particles.

Particle reconstruction when a body is present in the flow field presents additional challenges because the measurement volume contains partially-occluded regions (i.e., regions where the body blocks visibility of tracer particles in some, but not all, viewpoints). An advantage of the additive SAPIV particle reconstruction algorithm [Eq. (1)] in these scenarios is that a particle can be localized without appearing in every camera. In contrast, multiplicative algorithms such as multiplicative algebraic reconstruction technique (MART) require nonzero source information in each viewpoint for a nonzero reconstruction (Elsinga et al. 2006). While SAPIV is well-suited for partially-occluded measurement scenarios, compared to techniques with fewer viewpoints, algorithm performance in partially-occlusion regions may differ from reconstruction in the absence of a body.

Partially-occluded regions, which typically surround a body, are of particular interest when the archer fish jumps and impose measurement requirements beyond those seen in the previous applications of SAPIV to fish wakes (Mendelson and Techet 2015). At jump onset, multiple tail strokes can occur before the fish has significant upward velocity (Shih et al. 2017); the body is therefore in close spatial proximity to the wake for this period during the jump. The wakes of upstream fins (i.e., dorsal, anal, and pectoral fins) must additionally be resolved before and during any interactions with the caudal tail. Performing SAPIV on the archer fish, therefore, relies on identification of the best particle reconstruction strategy for partially-occluded regions.

The behavior of the archer fish imposes additional experimental constraints on the measurement system. Measured wake structures must be assessed in the context of the fish’s kinematics and the jump’s outcome (e.g., if the fish successfully reaches its target and how much it overshoots the bait). Shih et al. (2017) use the aerial trajectory of the fish to estimate the maximum velocity and acceleration during a jump. For coupled understanding of the kinematics and hydrodynamics, trajectory information must be obtained in 3D simultaneous with volumetric velocimetry measurements. As a result, it is desirable to reconfigure the typical 3 \(\times\) 3 SAPIV camera array for simultaneous under- and above-water imaging. This measurement constraint influences requirements for the particle reconstruction procedure as well because there are fewer cameras viewing the flow field.

This study presents modifications to the SAPIV technique that enable time-resolved measurements on jumping archer fish. A comparison of three non-iterative particle reconstruction algorithms is used to develop a processing routine specifically for partially-occluded measurement volumes. This analysis takes into account both the missing information in occluded camera views and the overall reduced number of cameras that view the particle field in partially-occluded regions. Information already necessary for 3D PIV masking is used to map and adjust particle reconstruction in partially-occluded regions, allowing use of an algorithm that typically requires particle visibility in all cameras. The reconstruction procedure can also be implemented with fewer cameras than traditional SAPIV, allowing cameras to be distributed between simultaneous aerial and underwater imaging. Simultaneous measurements of the aerial jump trajectory, fin kinematics, and flow produced by the dorsal, anal, and caudal fins demonstrate the capabilities of this technique to elucidate propulsive strategies in archer fish jumping.

2 SAPIV experiment design

2.1 Camera array

The physical camera arrangement for viewing both above and below the water’s surface must meet requirements based on the archer fish’s behavior. At jump onset, the snout of the fish is positioned at the surface (Fig. 1); the underwater measurement volume must, therefore, be located directly below the free surface. Position requirements for the aerial cameras are based on the finding of Shih et al. (2017) that the peak jump acceleration occurs immediately after jump onset. Aerial-viewing cameras must, therefore, begin to capture the fish trajectory as soon as the snout breaks the surface. Based on peak 2D jump height measurements, the field of view for aerial imaging must span vertically from the surface to 2.5 times the fish’s standard length (approximately 18 cm). Separate aerial and underwater cameras are desirable to avoid multiple calibrations for each camera and to have full camera sensor resolution in each fluid media.

The camera configuration meeting these requirements contains two rails of cameras (Fig. 2), with three underwater viewpoints on the top rail and four underwater viewpoints on the bottom rail. Two rows of cameras viewing underwater are used instead of three because of limited vertical space viewing the measurement volume without reflections or occlusions at the free surface. The top row of cameras is mounted directly on the rail. This row includes a central camera for aligning the 3D coordinate system during camera calibration and locating the fish within the experiment field of view. The bottom row of cameras is attached by ball-head camera mounts to facilitate aiming the cameras \(15^{\circ }\) upward about the X-axis. A photograph of the camera configuration is shown in Fig. 2b. Two additional aerial cameras, also on ball-head camera mounts, are positioned 8.6 cm above the top underwater cameras on the top rail. This imaging configuration avoids adding additional cameras beyond the typical nine to an already hardware-intensive measurement technique. The number of cameras is not targeted for further reduction, with the goal of providing sufficient viewpoints for particle reconstruction even in partially-occluded regions.

Fig. 2
figure 2

Camera configuration for simultaneous SAPIV and 3D jump trajectory tracking. a Schematic of the camera placements for the jumping archer fish experiment. The free surface is located at approximately half the height of the tank. The shaded regions show the fields of view for the seven SAPIV cameras (red dotted line) and the two aerial trajectory cameras (green dashed line). The coordinate system is defined with the X-axis parallel to the long sides of the tank, the Y-axis vertical, and the Z-axis normal to the front tank wall. b Photograph of the camera setup showing the physical implementation of the design in a alongside a 38 L tank

2.2 Characterization of partial occlusion locations

When SAPIV is implemented around a body, occlusion of a tracer particle can be caused by either another particle or the body. When a particle is occluded by another particle in a single camera view, the particle will still reconstruct in 3D when refocused. Additive refocusing does not divide intensity contributions between multiple sources along the same line of sight; therefore, the occluding particle in the source image will count towards reconstruction at both depths.

Fig. 3
figure 3

Visual hull and six focal planes with regions partially-occluded by the fish body, both determined from SAPIV measurements of a jumping archer fish obtained using a seven camera array. a Reference image of the fish body from the center camera of the array. b The corresponding 3D visual hull reconstructed by refocusing binary body images. The visual hull is shown at a resolution of 8 voxels. c Partial occlusion locations at six depths in the measurement volume. Shading represents the number of cameras in which a given voxel is obscured by the body at each focal plane. Regions occupied by the body in all seven cameras correspond to the visual hull necessary for PIV masking. All Z coordinates are relative to the position of bait behind the tank wall

The more detrimental category of occlusions is when a region of particles is blocked from view by the body in a subset of cameras. If the body is masked (i.e., set to zero source intensity) in individual camera images before 3D reconstruction, the occluded particles will refocus, using Eq. (1), at a weaker intensity than particles visible in all cameras. If the body is left unmasked, bright or dark patches of the body will influence the final position and brightness of the reconstructed particles. A particle field reconstruction routine with the ability to identify and compensate for partial occlusions could avoid either of these scenarios.

The visual hull method (Adhikari and Longmire 2012) is commonly used for body masking in tomographic and synthetic aperture PIV; this method projects binary images of the body along each camera’s line of sight to determine the 3D regions where all cameras contain the body. These regions, where no cameras view particles, are then excluded during PIV processing. Figure 3a, b shows a sample 2D image of an archer fish body during one timestep of a jump sequence and the corresponding visual hull determined from seven camera viewpoints (cameras arranged as in Fig. 2). The visual hull (Fig. 3b) distinctly shows the pelvic, anal, and caudal fins. In the Z-direction, the reconstructed fins and body taper to a point; the size of the intersecting regions between all binary images decreases the farther a given depth is from the true location of a body feature. The elongation of the visual hull beyond its true depth in the viewing direction is a function of camera placement and is characterized in detail by Adhikari and Longmire (2012).

The information used to identify the visual hull can also be used to map partially-occluded regions in the flow field. If Eq. (1) is applied to the individual 2D binary masks used to create the visual hull, the result is a focal stack where intensity indicates how many cameras contribute to partial occlusion of the measurement volume. For this mapping of partially-occluded regions, points in front of and behind the body are both treated as occlusions. It is common for a bright body to wash out particles located in front of it, leaving them effectively still occluded.

Figure 3c shows the locations and severities of partial occlusions at six depths in the measurement volume. At depths towards the edges of the measurement volume (e.g., Fig. 3c, Z = 7 mm and Z = − 38 mm), most partially-occluded regions (59–64% in the examples shown) are occluded by the body in two or fewer cameras. In these regions, there are still five or six viewpoints that can contribute to particle reconstruction. The finite viewing angle between cameras causes image regions towards the center of the body to have worse visibility, even at the front and back of the measurement volume (Fig. 3c, Z = 7 mm and Z = − 38 mm). The camera viewing angle similarly causes the Z-direction elongation of the visual hull (Fig. 3b). In regions towards the center of the measurement volume, the visual hull (occluded in all seven cameras) is identifiable, including the pelvic fins and anal fin at Z = − 2 mm and the caudal fin at Z = \(-20\) mm. The regions surrounding the visual hull at these depths are nearly fully occluded (i.e., particles are visible in only one or two cameras). However, regions where body features found at other Z-coordinates prevent visibility of surrounding particles (e.g., the pelvic fin projections at Z = − 20 mm) have fewer occluded viewpoints. While few near-body regions are fully visible in all cameras, regions where a majority of cameras view particles are found in much of the measurement volume. Visualizing partially-occluded regions suggests that reconstruction in these regions is feasible and necessary for the jumping archer fish experiment.

2.3 Refocusing with partial occlusions and reduced cameras

Particle reconstruction must be performed with an algorithm that performs in partially-occluded regions with a reduced number of camera viewpoints and in regions with full visibility, ideally in one processing routine. The reduced overall number of SAPIV cameras, implemented in response to limited optical access near the surface and the need for simultaneous aerial measurements, adds an additional constraint on the reconstruction procedure. This section considers the performance of three non-iterative algorithms in the presence of partial occlusions and in the overall seven camera setup.

The additive refocusing algorithm traditionally used for SAPIV [Eq. (1)] is described extensively in the introduction. Two additional non-iterative particle reconstruction algorithms are the multiplicative line of sight (MLOS) (Atkinson and Soria 2009), also described as multiplicative refocusing when used in synthetic aperture imaging (Belden et al. 2012), and the minimum line of sight (minLOS) (Maas et al. 2009; Michaelis et al. 2010). These algorithms differ from additive refocusing [Eq. (1)] at the processing step where warped images from all cameras are combined. The MLOS algorithm takes the product of all transformed camera images as the value at a voxel:

$$\begin{aligned} I_{{\text {SA}}_k}=\prod _{i=1}^{N} (I_{{\text {FP}}_{ki}})^n. \end{aligned}$$
(2)

The exponent \(n = 1/N\) cameras preserves the original intensity scale of a particle image through the multiplication operations, but n can be specified otherwise to modify the size and signal-to-noise ratio of refocused features (e.g., Belden et al. 2012). The minLOS algorithm takes the minimum pixel value from all cameras mapping to a voxel:

$$\begin{aligned} I_{{\text {SA}}_k} = \min _{i=1}^{N}I_{{\text {FP}}_{ki}}. \end{aligned}$$
(3)
Fig. 4
figure 4

Relative reconstructed intensities of fully visible and partially-occluded particles shown using three sample particles of uniform intensity. All particles are shown with inverted intensity (darker particles are brighter) for visibility. Particle 1 is in focus at depth \(Z_1\), while particles 2 (fully visible) and 3 (partially-occluded) form dim ghost particles patterned in the shape of the camera array (also known as discrete blur). At depth \(Z_2\), the discrete blur patterns from particles 1 and 2 overlap to form a brighter ghost particle, and particle 3 is in focus at reduced intensity (compared to particle 1 at \(Z_1\)) due to its limited visibility

Figure 4 shows the effects of partial occlusion on additive refocusing for a simplified set of three particles: two visible in all cameras within a 3 \(\times\) 3 array (particles 1 and 2) and one visible in only three cameras of the array (particle 3). At depth \(Z_1\), particle 1 is in focus, while particle 2 forms a discrete blur pattern of one ghost particle per camera, arranged in the shape of the camera array. Particle 3 also forms a discrete blur pattern, containing one ghost particle from each of the three cameras in which it is visible. At depth \(Z_2\), particle 3 is in focus, and the other two particles each form the discrete ghost particle pattern. The coincidental overlap of the ghost particles from the two nine camera particles (particles 1 and 2) at depth \(Z_2\) is not significantly dimmer than particle 3, the in-focus particle visible in only three cameras at the same depth.

Partial occlusion also effectively reduces the number of source cameras used for reconstruction. Belden et al. (2010) show that reducing the number of cameras, either by design or as a consequence of partial occlusions, reduces the reconstruction quality of a particle field, as there is less intensity contrast between true (e.g., Fig. 4, depth \(Z_1\), particle 1) and ghost particles (e.g., Fig. 4, particle 2). Belden et al. (2010) also report that reconstruction qualities are lower for higher seeding densities. For densely-seeded images, the likelihood of two or more individual camera images converging without being a true particle location increases. Since additive refocusing is an averaging algorithm, the intensity of a ghost particle increases linearly with the number of cameras contributing to it. In some densely-seeded scenarios, most ghost particles may be as bright as true particles.

To evaluate use of Eq. (1) with partially-occluded measurements further, the probabilities of ghost particles with varying brightness forming are evaluated with respect to image source density (N \(_\mathrm{{s}}\)) and the number of array cameras (N). Probability-based analysis is also used by Elsinga et al. (2011) to study ghost particle formation in tomographic PIV, examining cases where source particles randomly converge (i.e., assuming no correlation between viewpoints). The source density (N \(_\mathrm{{s}}\)) is the product of the particle seeding density per pixel (ppp) and the area (in pixels) of an individual particle (A \(_\mathrm{{p}}\)). This quantity essentially describes the probability that a given pixel in an image is occupied by a particle. The inverse probability (\(1-N_{\text {s}}\)) is the likelihood that the corresponding pixel in another camera is not a particle. Binomial probabilities are used to calculate the probability (\(N_{\text {g}}\)) of a camera subgroup (size GC) in the N camera array overlapping to form a ghost particle during refocusing:

$$\begin{aligned} N_{\text {g}} = \frac{N!}{{\text {GC}}!(N-{\text {GC}})!} N_{\text {s}}^{{\text {GC}}}(1-N_{\text {s}})^{N-{\text {GC}}}. \end{aligned}$$
(4)

Figure 5 shows the probabilities of ghost particle formation from a camera subset of size GC for varying source density in 4–10 camera SAPIV systems. The quantity \(N_{\text {g}}\), the likelihood that a given pixel on a focal plane is occupied by a ghost particle of a particular brightness, can also be interpreted as the density of ghost particles in the reconstructed images. \(N_{\text {g}}\) is normalized by the source density (\(N_{\text {s}}\)) to compare the probability of a ghost particle occupying a pixel in a refocused image to the probability of a true particle occupying that pixel.

Fig. 5
figure 5

Probabilities of ghost particle formation (\(N_{\text {g}}\)) from a quantity of cameras GC (GC \(\le\) N), for reconstruction through additive refocusing (Eq. (1)) in 4, 6, 8, and 10 camera SAPIV systems. Ghost particle probabilities are normalized by the probability of a pixel being occupied by a true particle (source density \(N_{\text {s}}\)). Color is cutoff in locations where the probability of ghost particles forming from a given number of cameras is below \(10\%\) of the probability of a true particle

Ghost particles formed by an individual camera (GC \(=\) 1) have a high probability of occurrence at low source density. Increasing the total number of cameras (e.g., \(N=10\) versus \(N=4\)) also increases the quantity of low-brightness ghost particles relative to the number of true particles. At higher source densities, there is a nontrivial, and in many cases higher, likelihood of ghost particles forming from multiple cameras instead of a single camera. Increased camera array size improves the maximum source density where the probability of ghost particle formation is low.

The probabilities in Fig. 5 apply to cases in which intensity thresholding will appropriately segment the maximum brightnesses of true and ghost particles. The intensity distribution within an individual particle must also be considered when assessing the effectiveness of additive refocusing and thresholding. To prevent single-voxel particles and peak locking (e.g., Huang et al. 1997), intensity thresholding must remove the brightest ghost particles while preserving the dimmest regions of true particles (i.e., the minimum intensity of a true particle must be greater than the maximum intensity of the ghost particles). The appropriate threshold for separating particles from reconstruction artifacts is, therefore, also a function of the intensity distribution within an imaged particle.

The intensity distributions of true and ghost particles are compared on one focal plane of a refocused image stack (i.e., one 2D slice through the voxel volume). A true particle with perfect reconstruction located on that plane post-refocusing is modeled as a \(3\times 3\) Gaussian kernel with variance \(\sigma ^2\) and intensity ranges from \(I_\mathrm{min}\) to \(I_\mathrm{max}\):

$$\begin{aligned} I_\mathrm{min} = I_\mathrm{max}e^{-\frac{1}{\sigma ^2}}. \end{aligned}$$
(5)

If a higher intensity threshold than \(I_\mathrm{min}\) is applied, the number of single-voxel particles, and consequently the likelihood of peak locking, increases. In comparison, the maximum intensity of a ghost particle created by a single camera during refocusing is \(\frac{I_\mathrm{max}}{N}\), where N is the total number of cameras in the array. Intensity thresholding to remove ghost particles created by a single camera in a 3D focal stack will remove information regarding true particles unless

$$\begin{aligned} e^{-\frac{1}{\sigma ^2}}> \frac{1}{N}. \end{aligned}$$
(6)

In general, the maximum intensity in a ghost particle formed from a subset of cameras with size GC is \(\frac{I_\mathrm{max}GC}{N}\). Figure 6 shows how the maximum intensity in ghost particles formed from one to four cameras compares to the minimum intensity in a true particle [Eq. (5)] for varying \(\sigma\) and camera array size. Ghost particle intensities above the dashed lines representing each \(\sigma\) are retained if the noise-removal threshold is set such that it preserves all true particle intensities (threshold \(<I_\mathrm{min}\) for a given \(\sigma\)). At \(\sigma =0.5\), all ghost particles would remain after thresholding, even when there is a 1:15 ratio in brightness between ghost particles created by a single camera and true particles. With a seven camera array, ghost particles created by three or four cameras are retained for \(\sigma =1\), but the peak brightness of a ghost particle created by one or two cameras is still eliminated. If \(\sigma =1.25\), only ghost particles created by four cameras are retained, and all ghost particles are successfully eliminated if \(\sigma =1.5\).

Fig. 6
figure 6

Intensity relationships between true particles of varying Gaussian profile and ghost particles for additive refocusing [Eq. (1)] with a 1–15 camera array. All intensities are normalized by the maximum intensity of a true particle reconstructed from all cameras [\(I_\mathrm{max}\) in Eq. (5)]. The maximum intensity of a ghost particle formed by 1–4 cameras decreases as the total number of cameras increases. Dashed lines represent the minimum true particle intensity for five different Gaussian particle profiles of varying \(\sigma\). In many scenarios, ghost particles are brighter than the minimum intensity of a true particle on one focal plane within the refocused volume

Particle size and brightness are controlled in an experiment by the illumination and lens \(f\#\), which in turn are driven by the required thickness of the measurement volume. For volumetric experiments, depth-of-field requirements typically necessitate a high \(f\#\) and resultantly small particles with a low \(\sigma\). While the particle intensity profile can be modified through image preprocessing operations, the particles that can be segmented using intensity thresholding are the least similar to the intensity profiles of actual particles in volumetric measurements.

The thresholding process does not exist with use of either the MLOS [Eq. (2)] or minLOS [Eq. (3)] algorithms. In contrast to the additive refocusing algorithm, with both the MLOS and minLOS algorithms, ghost particle formation requires nonzero source intensity in all cameras. The likelihood of ghost particle formation (\(N_{\text {g}}=N_{\text {s}}^N\)) drops with each additional camera added to the array (Fig. 7). However, ghost particles formed by either of these algorithms have the same intensity scale as true particles.

Fig. 7
figure 7

Probabilities of ghost particle formation (\(N_{\text {g}}=N_{\text {s}}^N\)) on one focal plane for increasing source density and total number of array cameras using MLOS or minLOS reconstruction. Color is cutoff for a ghost particle density less than 10\(\%\) of the source density

2.4 Comparison of reconstruction algorithms

The main advantage of the additive refocusing algorithm is that particles can be reconstructed without appearing in all cameras. However, the analysis of additive refocusing shows that in partially-occluded measurement scenarios, and in many fully-visible situations, intensity is insufficient to segment real particles from reconstruction artifacts in SAPIV. Regardless of the number of cameras, intensity thresholding for particle segmentation is only effective at low source densities, where most ghost particles are actually dimmer than the true particles (Fig. 5). When the source density is high enough that many ghost particles form from more than one camera, these false particles become comparable in brightness to partially-occluded true particles.

The smaller a particle is (lower \(\sigma\)), the harder it is to segment, even with a large number of cameras (Fig. 6). Small particles are frequently a consequence of the high \(f\#\) required for depth of field in volumetric PIV experiments, though this limitation can be mitigated by blurring and re-normalizing particle intensities during image preprocessing. Even when these intensity segmentation constraints are satisfied, partially-occluded regions introduce additional intensity variation. Additive refocusing can reconstruct partially-occluded volumes with no additional information or modification of the reconstruction algorithm. However, the limitations to threshold definition and ghost particle removal with partial occlusions suggest that it is not the optimal particle reconstruction method for studies with bodies in the flow field.

As typically implemented, agreement between all cameras is required to reconstruct particles with either of the minLOS [Eq. (3)] and MLOS [Eq. (2)] algorithms. Additional information about occlusion locations is needed to implement reconstruction in the extensive partially-occluded regions surrounding a body (e.g., Fig. 3c). This limitation is not unique to non-iterative particle reconstruction algorithms; Adhikari and Longmire (2012) suggest that the accuracy of tomographic PIV in partially-obscured regions could be improved by running MART reconstruction in subsets of cameras corresponding to where particles are visible around a body.

Of the MLOS [Eq. (2)] and minLOS [Eq. (3)] algorithms, the minLOS reconstruction is more punitive, as it requires a bright particle in all cameras for a high image intensity reconstruction. Particle brightness determined via the MLOS algorithm can be inaccurately increased from the product of bright regions in some cameras and any nonzero value in others. The binary images of the body from each camera, already required for the visual hull method, also provide the information necessary for efficient camera subgroup handling using minLOS [Eq. (3)] reconstruction. If image regions corresponding to the body are set to the maximum brightness, the resultant minimum is obtained from valid particle viewpoints, except in regions that are occupied by the body in all cameras. Separate reconstructions for each combination of cameras are not required using this routine, and a cutoff for how many viewpoints are needed to consider a particle reconstruction valid can be determined from Fig. 7. Use of the MLOS algorithm [Eq. (2)] instead of minLOS requires that the additional parameter n be varied depending on the number of contributing viewpoints, complicating the processing routine.

Figure 8 shows the minLOS refocusing process using SAPIV measurements of flow generated by the dorsal, anal, and caudal fins of an archer fish, including two example slices of the 3D volume along the body. Raw images from each camera (Fig. 8a) are used to obtain binary masks of the fish body (Fig. 8b). During image preprocessing (before refocusing), regions corresponding to the body, identified using the binary masks, are set to the maximum intensity value (Fig. 8c); the minLOS algorithm [Eq. (3)] can then be applied globally. The value of the combined image pixels at each focal plane is the minimum of the non-body viewpoints; regions occupied by the body in all cameras have maximum intensity (Fig. 8d, g). The additive-refocused binary body images (Fig. 8e, h) are then used to mask the focal stack in regions partially-occluded in more than a prescribed minimum number of cameras (Fig. 8f, i). Refocused body masks (Fig. 8e, h) are also used to identify the visual hull; regions occupied by the body in all cameras have the maximum possible brightness.

The two depths shown in Fig. 8 correspond to regions occupied by the anal fin (Z = − 1.6 mm, Fig. 8d–f) and body and caudal fin (Z = 17 mm, Fig. 8g–i). In Fig. 8f, the final occluded region to the right of the anal fin is smaller than the entire shaded region in Fig. 8e. Similarly, in Fig. 8i, the near-body region occluded by the anal fin is much smaller than the regions where any cameras are occluded by the anal fin (Fig. 8h). In partially-occluded measurement volumes, the minLOS algorithm is both simple to implement and provides improved performance over additive refocusing by eliminating thresholding operations.

Fig. 8
figure 8

Reconstruction steps using a minLOS particle reconstruction coupled with image averaging to determine partially-occluded regions. a Raw image from the center camera of the array. b Binary mask corresponding to the body in a. c Preprocessed 2D SAPIV image created by combining a, b and performing preprocessing operations to enhance particle visibility. d, g Two slices through the focal stack (Z \(= -\) 1.6 mm and Z \(=\) 17 mm) reconstructed using minLOS refocusing. e, h Occlusion maps obtained from additive refocusing of binary masks at the same depths as d, g. Brightness in the occlusion maps is proportional to the number of occluded cameras. f, i Refocused images after masking regions occluded in greater than four cameras

3 Experiment implementation

The SAPIV system designed to provide aerial and underwater measurements (Fig. 2) is implemented to obtain high-resolution wake measurements of the dorsal, caudal, and anal fins immediately following jump onset. For this particular experiment, the pectoral fins are not included in the measurement volume. Experiments are performed in a 38 L aquarium (51 cm \(\times\) 25 cm \(\times\) 30 cm) filled halfway (15 cm from the bottom). The experiment tank is filled using water from the archer fish’s home tank to ensure consistent brackish salinity. The tank is heated to match the home tank temperature using a 50 W aquarium heater. These procedures reduce stress on the fish during experiments. The experiment tank is seeded with 50 \(\upmu\)m polyamid particles. The seeding density of 0.04 particles pixel\(^{-1}\) corresponds to a source density \(N_{\text {s}} = 0.4\). Bait (dried plankton) is suspended from a thread running through a hole in the aquarium hood. The bait is located 8 cm behind the front tank wall. Fish position in the measurement volume is controlled by bait placement. All results shown herein are from a smallscale archer fish (Toxotes microlepis) with a standard length of 7.0 cm and weight of 7.5 g. All animal use protocols are approved by the Massachusetts Institute of Technology Committee on Animal Care (protocol number 0315-026-18). Fish training procedures and husbandry details are discussed in detail in Shih et al. (2017).

Nine high-speed cameras (Vision Research Miro 310, 1280 \(\times\) 800 pixel resolution), seven for SAPIV and two for 3D aerial body tracking, are configured to image above and below the free surface, as shown in Fig. 2. The upper three cameras are spaced 170 mm horizontally, and the lower four cameras are spaced 130 mm horizontally. The vertical spacing of the cameras is 125 mm. The array is positioned 390 mm outside the front tank wall. For a high magnification view of the median (i.e., dorsal, anal, and caudal) fins, the SAPIV cameras use 105 mm Sigma macro lenses (f/16). The resultant measurement volume size is 70 \(\times\) 40 \(\times\) 35 mm. The aerial cameras are equipped with 35 mm Nikon lenses (f/11). All cameras are synchronized at 750 frames s\(^{-1}\).

Near-infrared illumination is provided using an Oxford Lasers Firefly 1000 W volumetric laser synchronized with the cameras at a \(1\%\) duty cycle. This wavelength is invisible to archer fish and is used to prevent any influence of PIV illumination on the fish’s behavior and aiming strategy. As in Mendelson and Techet (2015), a first surface mirror is used to reflect the laser volume back into the tank for additional light. Illumination for aerial imaging is provided by ambient room lighting and overhead LEDs in the aquarium hood.

SAPIV cameras are calibrated with a bundle adjustment model accounting for planar refractive interfaces (Belden 2013). The aerial cameras are calibrated by direct linear transformation using the custom MATLAB programs DLTcal5 and DLTdv5 developed by Hedrick (2008). The DLTdv5 program is also used to automatically track the fish snout in 3D using the aerial camera data. Snout trajectories are used to measure the jump height of the fish; snout position data are fit to quintic splines to evaluate overall body velocity and acceleration over time.

Underwater fin kinematics are determined by using DLTdv5 to manually digitize marker points in the top center, bottom left, and bottom right cameras. Body points tracked over time are the tips of the caudal fin, the three spines of the anal fin, and the dark spot at the tip of the dorsal fin. Tracked points are triangulated using the same camera calibration used for particle volume reconstruction. Marker trajectories are smoothed over time in X, Y, and Z using cubic splines. Eight additional points along the edges of the caudal and anal fin are used to describe the curvature of these fins at each timestep. These points correspond between cameras but not over time; marker locations are redistributed as fins partially leave the field of view. Fin edge outlines at each time are smoothed by fitting fourth-order polynomials to the tracked points.

The binary masks necessary to construct the visual hull and map partially-occluded regions are generated using a semi-automated routine that implements the GrabCut algorithm available in the OpenCV library (Rother et al. 2004; Bradski 2000). The algorithm is initialized for each camera with a bounding box around the fish body at the first timestep selected for SAPIV processing. After running an initial segmentation, the user either identifies over- or under-masked regions of the fish body and runs another segmentation iteration or saves the mask. The mask from the previous timestep is used to initialize the mask at the next time. The semi-automated approach is able to adapt to changes in body lighting and shadow locations throughout a jump sequence. Once the binary body masks are identified for each camera, particle image regions outside the body are preprocessed by subtracting a \(5 \times 5\) median-filtered background image, convolving with a \(3 \times 3\) Gaussian blur kernel (\(\sigma =1\)), performing local intensity normalization (sliding \(5 \times 5\) windows), and applying a low-intensity threshold to the 2D source images to remove any noise amplified during intensity normalization. Body regions within the mask are set to the maximum image intensity to eliminate their contributions when using the minLOS algorithm (Fig. 8c).

The homography-fit method developed by Bajpayee and Techet (2017) is used to warp particle images from each camera to each focal plane. At the experiment source density (\(N_{\text {s}} = 0.4\)), the likelihood of ghost particle formation in a given voxel is less than one-tenth of the likelihood of a true particle existing at that location when four or more cameras are used for reconstruction (Fig. 7). Four non-occluded viewpoints are, therefore, required for a refocused region to be considered valid. Refocused image regions with fewer than four viewpoints are masked along with the visual hull determined from all seven cameras (e.g., Fig. 8f, i). The particle fields are processed by multi-pass cross correlation using a modified 3D version of the MatPIV code originally developed by Sveen (2004). This code is also used in Mendelson and Techet (2015). The final vector spacing using \(64^3\) voxel windows at \(50\%\) overlap is 1.79 \(\times\) 1.79 \(\times\) 1.92 mm. Velocity fields are post-processed using the ratio between the first and second cross-correlation peaks, a \(3\times 3\times 3\) local median filter (threshold of two standard deviations from the median), and smoothing at each timestep using the algorithm of Garcia (2011). Vorticity (\(\omega\)) is calculated from the smoothed data using a second-order centered difference.

Momentum transfer in the fish wake is also assessed through the hydrodynamic impulse (I), which in 3D vector form is calculated from the vorticity field as

$$\begin{aligned} \mathbf {I} = \frac{1}{2}\rho \int _V \mathbf {x} \times \mathbf {\omega } {\text {d}}V, \end{aligned}$$
(7)

where x is a position vector and \(\rho\) is the fluid density (1.0 g cm\(^{-3}\) at experiment temperature and salinity). The archer fish wake contains close proximity, interacting vortex structures, which Mendelson and Techet (2015) show must be avoided for wake impulse models using the geometry and circulation of an isolated vortex ring. Therefore, the hydrodynamic impulse is instead calculated directly from the vorticity field. Equation (7) is sensitive to the choice of origin for the position vector (Rival and Oudheusden 2017); these effects are minimized by using an origin determined from the fish body position. Specifically, the centroid of the visual hull at t = 0 s is used as the origin for all impulse calculations.

Fig. 9
figure 9

Aerial trajectory, fin kinematics, and wake measurements during a 1.7 body length jump (bait height 1.2 body lengths). a XYZ positions, vertical velocity, and vertical acceleration of the snout from jump onset (t = 0 s) until the fish reaches its maximum height above the water. The snout becomes visible above the surface at t = 0.005 s. b Caudal, anal, and dorsal fin kinematics, using the markers shown in the photograph, over time in the SAPIV measurement volume. The solid line denotes the edge of the caudal fin, the dashed line denotes the edge of the anal fin, and the circle denotes the posterior lobe of the dorsal fin. c Wake measurements at three times during the first three peak-to-peak tail strokes. Flow structures are visualized by vorticity magnitude (isosurface at 100 s\(^{-1}\)). The gray isosurface shows the location of the visual hull at a resolution of 8 voxels, and the dashed black lines show the tail tip trajectories

4 Results and discussion

Figure 9 presents simultaneous measurements of the aerial trajectory (a), underwater fin kinematics (b), and volumetric flow field (c) during a 1.7 body length jump. The bait height for this trial is 1.2 body lengths. Figure 9a shows the 3D position, vertical (Y) velocity, and vertical (Y) acceleration over time. The time t = 0 s is when the fish initiates propulsive tailbeats for the jump. The time intervals of each peak-to-peak tail stroke and the gliding stage (i.e., when the fish is completely out of the water) are also shown. Tail stroke timings are determined from the underwater caudal fin kinematics. The start of the gliding stage is identified from the aerial trajectory as when the snout height is greater than one body length above the surface. Figure 9b shows XY and XZ projections of the dorsal, anal, and caudal fin kinematics within the underwater measurement volume. Kinematic marker locations are shown in the accompanying photograph, which also shows the position of the fish in the measurement volume. Vortex wake structures during the first three propulsive tail strokes are presented along with the tail tip trajectories, as shown in Fig. 9c.

The measurements of the snout position (Fig. 9a) obtained from aerial imaging show that it moves in the same direction and approximately in phase with the tail from jump onset to the end of the second tail stroke (t = \(0-0.03\) s). The snout does not move laterally after the initial two peak-to-peak tailbeats, indicating a change in undulation waveform. Motion is isolated towards the aft end of the fish once more of the body has left the water. At jump onset, the snout also moves backwards in X, again only until the conclusion of the second tail stroke. The next major snout motion occurs when the mouth opens (t = 0.1 s). The fish is completely out of the water (Y \(> 0.07\) m, the standard length of the fish) by this time. The full-body (i.e., snout to tail) propulsive motions observed when the entire body is submerged, in addition to the fin behaviors observed in Fig. 1, may be crucial to producing the high acceleration observed at jump onset (Fig. 9a).

Fig. 10
figure 10

Dorsoventral (X), vertical (Y), and lateral (Z) velocity profiles in the dorsal, anal, and caudal fin wakes. a Velocity profile locations relative to fin kinematics; all profiles are taken along the X-axis. The triangle (anal fin), circle (dorsal fin), and square (caudal fin) markers show the locations of the velocity profiles at the conclusion of the first tail stroke (hollow markers, t = 0.015 s) and the conclusion of the second tail stroke (filled markers, t = 0.024 s). b, d Velocity profiles in the caudal fin wake at t = 0.015 s and t = 0.024 s. c, e Velocity profiles in the dorsal and anal fin wakes at t = 0.015 s and t = 0.024 s. The flat center region is the location of the caudal peduncle

Shih et al. (2017) find that velocity fields slicing through the caudal fin wake during jumping resemble the reverse Kármán street of steady forward fish locomotion, with one vortex core appearing to shed per peak-to-peak tail motion. The vorticity contours over time (Fig. 9c) show this vortex ring structure in 3D for the first three peak-to-peak tail strokes. Each stroke produces a coherent vortex ring that links with the wake of the previous tailbeats. The first tail stroke produces a smooth vortex ring (Fig. 9c, t = 0.016 s); the tail does not encounter upstream fin wakes during its initial motion. The first and second vortex rings are spatially closer together than the second and third vortex rings. The much higher vertical velocity of the fish during the third tail stroke (t = 0.031–0.040 s) results in greater spacing between subsequent wake structures than is seen between the vortices shed shortly after jump onset. The waveforms traced by the dorsal and ventral tail tips also show the increased vertical distance traveled during the third tail stroke. Additional tubes of vorticity appear to connect the vortex rings from the second and third tail strokes (Fig. 9c, t = 0.031 s, t = 0.040 s).

Underwater fin kinematics and SAPIV measurements are combined to determine the velocity profiles in the wake of each fin at jump onset and during later propulsive undulations. Figure 10 shows profiles of the dorsoventral (X), vertical (Y), and lateral (Z) velocity components at the conclusions of the first and second tail strokes. The velocity profile locations (Fig. 10a) are determined by finding the Y-velocity extrema closest to each fin’s location at each time. After the first tail stroke (Fig. 10b), the peak Y and Z velocities in the caudal fin wake are of comparable magnitude (600 mm s\(^{-1}\)). The Z-velocity is negative, following the direction of motion during the preceding tail stroke. Flow in the dorsoventral (X) direction is directed towards the center of the tail on both sides of the body, but has higher velocity (− 400 mm s\(^{-1}\)) on the ventral side of the body. This significant dorsoventral momentum transfer by the tail may be responsible for rotating the body (as also evidenced by the snout motion in − X at jump onset) towards a more vertical posture before subsequent tail strokes.

After the second tail stroke (Fig. 10d), the peak vertical velocity immediately behind the tail has a comparable profile to the first tail stroke (peak velocity approximately 500 mm s\(^{-1}\)). The lateral (Z) velocity, however, is of much lower magnitude and changes direction along the dorsoventral span of the body. The first two tail strokes occur before the fish has traveled significantly upward, and the tail passes directly through the earlier paths of the dorsal and anal fins (Fig. 10a). The low lateral wake velocity may be the result of the second tail stroke reversing momentum that was shed into the wake during the first tail stroke.

Separate propulsive jets behind the dorsal and anal fins are observed at the conclusion of each tail stroke (Fig. 10c, e). Following the first tail stroke (Fig. 10c), the peak velocities in jets generated by the dorsal and anal fins are much lower than those observed behind the caudal fin (200 versus 600 mm s\(^{-1}\)). The jets generated by the dorsal and anal fins are also not as wide as those generated by the caudal fin. The combination of these factors suggests that the caudal fin transfers more momentum to the water at jump onset. The direction of the wake jets is the same between all three fins at jump onset; the dorsal and anal fins do not move opposite the tail to counteract its lateral forces. As with the caudal fin, the velocities measured in the Z-direction are comparable to those measured in Y and follow the direction of caudal fin motion. In the dorsoventral (X) direction, the wakes of both the dorsal and anal fins are directed towards the caudal peduncle and the center of the caudal fin.

At t = 0.024 s (Fig. 10e), flow velocities in the dorsal and anal fin wakes have higher overall magnitude and are directed more laterally than vertically. Peak velocities match those observed from the caudal fin at jump onset, especially from the dorsal fin. Unlike the minimal lateral velocity in the caudal fin wake, there is flow produced in the direction of the propulsive stroke from the dorsal and anal fins. The measurements of the kinematics and velocity profiles from the dorsal and anal fins suggest that these fins have independent capabilities that vary between jump onset and later propulsive motions, but contribute less overall thrust than the caudal fin. Kinematic tracking of these fins also highlights their ability to interact with the tail to propel, stabilize, and provide upstream momentum for later tail strokes to exploit.

Fig. 11
figure 11

Hydrodynamic impulse calculated using Eq. (7) in the measurement volume over time. Time intervals correspond to each peak-to-peak propulsive tail stroke

The overall impulse in the flow field (Fig. 11), calculated using Eq. (7) shows the three-dimensional momentum in the wake over time. The impulse helps quantify the variations between the vortex rings shown in Fig. 9 and the net propulsive effects of the jets measured in Fig. 10. At the conclusion of the first tail stroke, the Y and Z components of the impulse vector have similar magnitude. During subsequent tail strokes the total impulse in the Y direction increases. The rate of change in vertical impulse during the second and third tail strokes is greater than during the first tail stroke. While Shih et al. (2017) found that propulsive tail strokes can be considered a discrete unit of propulsion that correlates with the final jump height, the volumetric measurements of the impulse during each tail stroke show that there are hydrodynamic differences between propulsion at jump onset and during subsequent tailbeats.

The velocity profiles from the caudal, dorsal, and anal fins at the conclusion of the first tail stroke are consistent with the distribution of impulse between lateral and vertical directions. In the lateral direction, the impulse oscillates with each tail stroke; the velocity profiles from the dorsal and anal fins during the second tail stroke suggest that this oscillation is caused by momentum contributions from all three fins. The net impulse in the X-direction during the first tail stroke is close to zero, suggesting that the strong dorsoventral jet from the first tail stroke is counterbalanced by additional momentum.

5 Conclusions

Synthetic aperture PIV, performed with the near-body particle reconstruction method presented in this work, provides both volumetric, three-component flow fields (for quantification of vertical thrust, dorsoventral, and lateral force production by each fin) and measurements of multiple propulsors during a single experimental trial. The vortices generated by each tail stroke are resolved despite the three-dimensional motion of the fish, revealing a linked chain with one vortex ring shed per tail stroke. These measurements highlight the interactions between subsequent tailbeats and changes in the orientation and spacing of wake structures as a jump progresses. Velocity profiles show that the orientation and strength of the propulsive jets produced by each fin also vary between jump onset and subsequent tail strokes. The velocity profiles observed at the conclusion of each tail stroke are consistent with the overall changes in wake momentum as quantified by the hydrodynamic impulse.

The camera system design and occlusion-compensated particle reconstruction techniques presented in this study are promising tools to elucidate the complex hydrodynamics of archer fish jumping. The experiment procedures developed in this study can facilitate assessment of how wake structures and fin interactions vary with jump height. Since archer fish start from rest at the surface, it is also feasible to capture the entire wake generation process in measurement volume sizes appropriate for 3D PIV experiments. With coupled information about the aerial trajectory, methods for force and energy prediction can be compared between PIV and the aerial kinematics of the fish. The measurements presented in this study characterize the wakes of three fins immediately following jump onset, but the same techniques can be used to characterize the use of the pectoral fins or the wake structure immediately before the fish leaves the water.

In applications beyond the jumping archer fish, this work demonstrates that synthetic aperture particle image velocimetry can physically and algorithmically adapt to partial occlusions and other optical access constraints. By analyzing reconstruction algorithm performance for varying camera array size and seeding density, this study identifies that the minLOS algorithm, coupled with binary masking to identify occluded viewpoints, provides a better signal-to-noise ratio than additive refocusing in partially-occluded regions. This algorithm enables physical redesign of the SAPIV camera array to include asymmetric camera spacings and a reduced numbers of cameras. With a large number of viewpoints that can contribute to particle field reconstruction, SAPIV is uniquely well-suited to measurement scenarios where partial occlusions are present.