Fourier Multispectral Imaging: Measuring Spectra, One Sinusoid at a Time

Abstract

We recently introduced the notion of Fourier Multispectral Imaging (Fourier MSI), a novel technique for undersampling spectral images without significant sacrifices to the spectral features that may be useful for material identification. The idea originated in Fourier transform spectroscopy, where multiple interferometric measurements are used to take spectral samples while varying optical path difference (OPD). Since interference is equivalent to spectral modulation by a filter with sinusoidally shaped transmittance function, we designed a sinusoidal filter using thin film Fabry-Perot to acquire OPD samples at every pixel. Owing to the rapid decay of OPD samples, Fourier MSI is an ideal multispectral imaging modality for preserving spectral features with the fewest spectral samples. We detail our prototyping efforts and demonstrate the advantages of such multispectral imaging configurations.

1 Introduction

In spectroscopy, the interaction between light and matter is used to study the properties of the reflecting surface materials. The absorption of electromagnetic radiation depends on the material composition, penetration depth, photon energy conversion to heat, etc. Spectrally narrowband electromagnetic radiations called emission lines also correspond to the difference in energy between two active atomic/molecular states. Hence, the spectral attenuation of reflected, transmitted, or scattered light (or the presence of spectral emission lines) can be useful for detecting the composition of chemicals or inferring the state of the materials. The applications are wide ranging, including material identification, biopsy, geophysical surveying, and target tracking/detection/recognition.

In human visual system, a light in the visible range is observed by three types of photon receptors known as cones. The spectral responses of the three photon receptor types are different, giving rise to the notion of perceived “color.” Hence one may interpret human color vision as a type of undersampling on the light spectra by the three spectral samples summarizing the spectral characteristics of the material. However, it is impossible to recover the full light spectra from the color tristimulus values (e.g. red, green, blue). As such, utility of color imaging as means of material/chemical identification is severely limited.

Full spectral description of light can also be useful in color imaging. An accurate color rendering and material appearance require modeling of light and reflectance spectra. For example, consider the problem of computational “relighting”—rendering a scene captured under one illuminant as if it were lit by another illuminant. The relighting is a function of a full spectrum, meaning it cannot be performed with only tristimulus values. An accurate imaging and color rendering of objects containing fluorescent agents (or ordinary objects illuminated by fluorescent illumination) is difficult also because the underlying spectral features are extremely narrow. Hence, most color imaging applications involving these difficult scenarios rely on crude approximations in a three dimensional subspace that clearly have limitations.

Fig. 1.

Conventional multispectral imaging leverages narrowband filtering to sample spectral radiance. Basis vectors are linearly combined to reconstruct the spectrum. (Color figure online)

The complexity of hyperspectral imaging (dense spectral sampling at every pixel) hardware is very high and its acquisition speed is very slow. The color imaging sensors on the other hand are fast and simple, but they do not have the discriminating features for detection/recognition tasks. As an alternative to hyperspectral imaging, multispectral imaging (MSI) systems make N finite spectral measurements using N predetermined filters to reduce the hardware complexity. Conventional MSI configurations use narrowband filters whose center wavelengths were chosen to satisfy the needs of the imaging applications (example shown in Fig. 1(b)) [1,2,3,4]. Although this approach indeed helps reduce the hardware complexity, a given sensor configuration is optimized for the detection of one or two specific materials only (e.g. a filter transmittance is aligned with the emission line). Undersampling with narrowband filters does not yield a “general-puropose” MSI hardware that is application-agnostic or multipurpose. MSI-based surveillance/security/biometry systems are particularly prone to spoofing attacks if there is an overreliance on certain spectral features or filter center wavelengths.

Our goal for MSI is to develop a novel foundation for undersampling spectra that best preserves the spectral features of materials. Towards this goal, we recently introduced the notion of Fourier multispectral imaging [7]. We leverage the direct correspondences between Fourier sampling theories and the Fabry-Perot thin film filters that have sinusoidal shaped spectral transmittance functions. As we detail below, the sinusoidally shaped filters have advantages over the narrowband filtering in preserving spectral features that are critical for tracking/detection/recognition tasks. Fourier MSI is also general purpose, allowing the detection of multiple materials/chemicals/objects by a single detector. In this paper, we review key details of the Fourier MSI and describe the ongoing prototyping efforts at the University of Dayton towards verifying its viability.

2 Fourier Multispectral Imaging

2.1 Interferometric Spectroscopy

By “spectrum of light” we refer to the spectral radiance, or surface radiance (watts per steradian per square meter) as a function of wavelengths per unit of wavelengths (meter). In physics and optics, the spectral radiance is equivalently be expressed in terms of wavenumber (radians per meter), which describes the number of periodic wave cycles per distance (reciprocal of wavelengths, which is the distance per cycle). Among techniques available to measure the spectral radiance is the Fourier transform spectroscopy.

Fig. 2.

Michelson interferometer. The two light paths (via the mirrors A and B) are combined at the detector. The difference of the path lengths determines the phase difference of the combined beams that interfere destructively/constructively. (Color figure online)

Consider measurements in the Fourier transform spectroscopy made by a well-known device known as Michelson interferometer, shown in Fig. 2. The two light paths (via the mirrors A and B, as indicated by green and blue arrows) are combined at the detector with the help of the half mirror. The difference in the path lengths (referred to as the optical path difference, or OPD) determines the phase difference of the combined beams that interfere destructively/constructively, which can be inferred from the light intensity measured by the detector. The spectral radiance of the light can be recovered by taking a Fourier transform of the interferogram, acquired from repeating the measurements with varying OPDs.

Fig. 3.

Fourier multispectral imaging samples interferogram, which is effectively same as filtering with a sinusoidal transmittance. Forward Fourier transform recovers the spectra in wavenumber domain, which may subsequently be converted to wavelength.

It is clear from the Fourier transform relations that the interferogram and the spectral radiance represent equivalent information. We proposed in [5,6,7] the notion of Fourier MSI, or the undersampling of the interferogram, whose workflow is summarized in Fig. 3(a). The example in Fig. 3(c) shows five interferometric measurements taken by the Michelson interferometer in Fig. 2. Each of these five measurements corresponds to the coherence strength at varying optical path lengths. The acquired measurements are also equivalent to filtering the light with filters whose spectral transmittance function are sinusoidally shaped—sinusoidal in wavenumber domain, which looks like a skewed sinusoid in the wavelength domain as shown by Fig. 3(b). Unlike the narrowband filter counterparts, the sinusoidal “filters” have broadband transmittance.

To recover the spectrum from the sampled interferogram, we take its Fourier transform by linearly combining five cosine waves together. The cosine’s periodicity is reciprocal of the OPD, and its amplitude is scaled by the measured values—see Fig. 3(d). The combined sinusoids reconstruct the light spectrum in the wavenumber domain (Fig. 3(e)), which may also be converted into wavelength in Fig. 3(f). The recovered spectrum closely approximates the light spectrum, albeit slightly smoother than the actual spectrum. With only five interference measurements, spectral features such as peak heights and locations are accurately preserved.

2.2 Narrowband MSI v.s. Fourier MSI

We now contrast the proposed Fourier MSI to the conventional MSI approach of using narrowband filters, which is effectively an undersampling of the spectral radiance. We compare Figs. 1 and 3 to illustrate the profound differences between undersampling interferogram and spectral radiance. As illustrated in Fig. 1(b), the center wavelength of the five narrowband filters may or may not coincide with the spectral peak of the light (by chance). As a result, the peak of the recovered spectrum in Fig. 1(e) coincides with the one of five filter transmittance functions (and not with the light spectrum). Mitigating this problem by increasing the number of spectral samples also increases the hardware complexity; in an application-specific MSI hardware, we may also tune the center wavelengths of the filter transmittance to coincide with the spectral features of interests.

Appealing to the fact typical light spectra lives in a low dimensional subspace, one may improve the performance of the conventional MSI by projecting the measurements to a vector spaces (e.g. principal component analysis) that best models the light—see Fig. 4. Fourier MSI is expected to benefit from such approach also. However, although model-based recovery is useful for imaging natural scenes, it is less effective in applications such as anomaly detection or chemical/material analysis, and less desirable for target detection where the underlying models confound the detection rates. Clearly, an accurate multispectral imaging without invoking models (the so-called “nonparametric” techniques) are more desirable in many applications.

Fig. 4.

Simulated reconstruction from five spectral samples. Blue = ground truth. Red = Fourier MSI. Pink = Monno MSI with cubic interpolation. Green = Monno MSI with PCA reconstruction. (Color figure online)

From the sensing point of view, the Fourier MSI is a far more efficient than the conventional MSI. The energy of the interferogram in Fig. 3(c) decays rapidly with the increasing OPD, and the N spectral samples should correspond to short coherence lengths to maximize the captured signal energy. In this sense, interferogram is an ideal sensing modality for MSI—each successive interferometric measurement of longer OPD is a refinement to the signal reconstructed with the shorter OPD samples. Compare this to the undersampled spectral radiance—as illustrated in Fig. 1(c), the signal energy is spread evenly throughout the wavelengths ranges, meaning it is difficult to determine the N center wavelengths that maximizes the overall signal fidelity.

However, one disadvantage to the Fourier MSI is the Gibbs phenomenon. For example, the presence of narrow spectral feature near 520 nm in Fig. 4(c) cause the unwanted oscillation in 400–500 nm region. The Gibbs phenomenon can be suppressed by increasing the number of OPD measurements.

Fig. 5.

Fabry-Perot filter with varying etalon thicknesses. (b) If unfiltered, we measure the zero coherence light. (c–f) Each filtered light is a combination of discrete OPD samples that must be linearly uncombined in post-processing.

2.3 Filter Designs for Fourier MSI

In the previous sections, we showed the advantages to undersampling the interferogram. However, the hardware configuration in Fig. 2(a) does not preserve spatial location, meaning Michelson interferometer is appropriate for spectroscopy but not spectral imaging. In [5, 7], we proposed to replace the hardware design by leveraging Fabry-Perot thin film filter. As illustrated in Fig. 5(a), an etalon is a three dimensional structure where partial reflections occur at the interfaces. The detector sees a linear combination of multiple reflected light, resulting in an interference caused by the path length differences among the reflections. One major advantage to Fabry-Perot thin film filters over the Michelson interferometer is that it preserves the spatial location (i.e. appropriate for imaging).

Consider a Fourier MSI system where the same scene is imaged using multiple Fabry-Perot filters of varying thicknesses, as shown in Fig. 5(a). Since the path lengths differences are integer multiples of the etalon thicknesses, the light combined at the detector is a linear combination of equally spaced OPD samples, as shown in Fig. 5(b–f). In addition, the path differences of the light combined by multiple etalons whose thicknesses are integer multiples of each other are also integer multiples of a common OPD length. We conclude, therefore, that each etalon measurement is a linear combination of the OPD samples in Fig. 3(c) we are after. Thus by linearly uncombining the acquired data, we can recover the desired interferometric values.

3 Prototyping and Results

Prototyping efforts are being carried out at the University of Dayton to verify the effectiveness of Fourier MSI. The “Prototype #1” configuration is shown in Fig. 6(a) [7]. The selected operating range is 450–900 nm, which is one octave in the wavenumber. We used a grayscale image sensor (OptiTrack V120 Slim), with a CMOS detector responding from 380–1000 nm. A 450 nm longpass filter and a 900 nm shortpass filter were added to restrict the range of incoming light that the detector responds to.

Fig. 6.

(a) Bulk filter-based Fourier multispectral imaging prototype system. (b) Spectral transmittance of the Zinc Sulfide filters of varying thicknesses.

For the sinusoidal filters, we fabricated six Fabry-Perot thin-film bulk filters on a glass substrate, using Zinc Sulfide thin film [7, 8]. Zinc Sulfide is ideal for our application because of relatively small dispersion over 450–900 nm. Dielectric filters form almost perfect sinusoids, as shown in Fig. 6(b). The cavity thicknesses were chosen so that the sinusoid spans 0.5, 1.0, 1.5, 2.0, 2.5, and 3.0 periods over 450 nm–900 nm. Images were taken by time multiplexing using filter wheels to exchange the fabricated filters.

Fig. 7.

Narrowband LED spectra reconstructed from by the Prototype #1. The reconstructed peaks (blue) coincide with the LED emission wavelengths (red) fairly accurately. (Color figure online)

Fig. 8.

Example of the Prototype #1 measurements. (a)–(g) Raw sensor data. (i–k) Recovered spectra. (h) RGB rendering of the recovered spectral scene. The ground truth spectra shown here is the combination of the spectral reflectance (measured by a contact spectrometer), illuminant and the quantum efficiency (measured using a monochrometer). (Color figure online)

Fig. 9.

(a) Spectral filter array-based single-shot Fourier multispectral imaging prototype system. (b) Filter array is comprised of four types of Zinc Sulfide thin-film filters of varying thicknesses. (c) Raw sensor data captures the spatially multiplexed OPD measurements. Post-processing “demosaicking” step recovers a full four-filtered interferogram.

Figure 7 shows examples of LED spectra imaged and recovered by Prototype #1. LEDs emit very narrowband light, which are precisely the type of spectra that is difficult to capture with conventional MSI systems. The spectral peaks recovered by the Fourier MSI are at (or very near) the emission wavelengths, albeit the presence of the Gibbs phenomenon and the recovered spectral peaks that are wider than the actual spectra. This confirms the claim made in [7] that the Fourier MSI systems have near-infinite resolution in detecting the wavelengths of a single wavelength peak.

Shown in Fig. 8 are example OPD measurements from the Prototype #1. The actual spectral measurements in Fig. 8(a–g) are linearly combined to yield the spectral reconstruction in Fig. 8(i–k), yielding a per-pixel recovery of the spectrum. The RGB rendering of the recovered spectra is shown in Fig. 8(h). The ground truth spectra in Fig. 8(i–k) was rendered by taking the reflectance data measured by a field contact spectrometer, which was combined with the spectral radiance of the illuminant and the quantum efficiency of the sensor, both measured by a monochrometer. While the reconstructions are not perfect, the recovered spectra accurately represented the features such as the narrowband spectral peaks and sharp transitions.

Another ongoing prototyping effort at the University of Dayton include fabrication of pixelized Fabry-Perot Zinc Sulfide filters [9]. As shown in Fig. 9(b), we fabricated a spectral filter array comprised of four types of filters (0.5–2.0 period filters) spatially multiplexed over the substrater [5, 11]. In our “Prototype #2,” the spectral filter array is attached to the CCD sensor surface directly (Fig. 9(a)), where each filter is designed to occupy multiple pixels so that the filter-pixel misalignments can be handled in post-processing (Fig. 9(c)). The spectral filter array configuration enabling a single-shot Fourier MSI, where the “demosaicking” is used to interpolate the missing filtered values to recover a full four-filtered interferogram from its spatially multiplexed version [6, 10, 11].

4 Conclusion

In this paper, we reviewed the novel concept of Fourier multispectral imaging—undersampling of interferogram intended to reduce the complexity of the spectral imaging—and the practical implementation using Fabry-Perot thin-film filters that yield periodic spectral transmittance functions while preserving the pixel/spatial integrity. We showed by simulated examples that the Fourier MSI captures spectral features such as peaks far more accurately than the conventional MSI approach of using narrowband filtering. We described the ongoing prototyping efforts at the University of Dayton to verify the feasibility of Fourier MSI. Our Prototype #1 demonstrated the ability to recover very narrowband spectral features such as LED emission, and yield acceptable reconstruction of broadband spectra. Our Prototype #2 enables a single-shot capability to the Fourier MSI.