Hardware of Near-Infrared Spectroscopy

Abstract

The hardware of near-infrared (NIR) spectroscopy is almost the same as UV-VIS and infrared spectrometer except the wavelength area. However, the high SN ratio and stability of the instruments are required for a quantitative analysis by NIR spectroscopy, because of the smooth and dull absorption peaks of the NIR spectral shapes. These are realized by the hardware and computer technologies and are special features of the hardware of NIR spectroscopy. It is important to understand what they are when you use or design a near-infrared spectrometer. These aspects of the technologies are described. Instrumental difference also is an important problem in NIR spectroscopy where a calibration is used to predict contents of the matter. In this Chapter, not only the method to avoid the instrumental difference, but also the sources of the instrumental are described. To decrease the instrumental difference, it is crucial to understand why and how the instrumental difference is generated. The information described in this chapter will help you design a new NIR instrument, and a designing process with less effort is also described.

1 Noise Reduction Technology of the NIR Spectrometer

Karl Norris successfully evaluated agricultural products using near-infrared (NIR) spectroscopy around 1970 [1]. Near-infrared absorption was first discovered by Abney and Festing in 1881. However, it took 100 years to develop practical NIR applications.

It is said that the electronics and computer technologies played an important role in Karl Norris’ success. However, it actually was a fight against the noise in the spectra.

Karl Norris specialized in the field of electronics and was unfamiliar with spectroscopy, which is based on the wavelength and height of the absorption peak. His background enabled him to easily adopt a new scanning method and statistics to eliminate the noise from the spectra, thus enabling the detection of subtle spectral changes. This would have been difficult to achieve using an infrared spectrometer, which generated the spectra on a chart paper recorder using a pen. The experiments performed to eliminate noise were successful, and using statistics, information on the ingredients could be retrieved from the NIR spectra [2].

1.1 Noise and NIR Spectroscopy

Figure 10.1 shows the reflectance spectra of beef meat obtained using the Foss XDS analyzer. In Fig. 10.1, abs refers to the absorption, which is the logarithm of the reflectance R (%), as shown below in Eq. 10.1.

Fig. 10.1

Reflectance spectra of beef meat

$$ {\text{abs}} = - { \log }\left( {R/100} \right) $$

(10.1)

The intensity of the 928 nm fat absorption peak increases with increasing fat. Although it is difficult to confirm fat absorption peak in the lean spectra (Fig. 10.2a), the second derivative spectrum clearly indicates the presence of the fat peak (Fig. 10.2b).

Fig. 10.2

Lean reflectance of beef (a) and its second derivative (b)

The vertical scale of the spectrum shown in Fig. 10.2a is in (abs), while that of the spectrum in Fig. 10.2b corresponding to the second derivative is in (µabs).

Figure 10.3 shows the spectrum to which 20 µabs noise was added in the simulations. It is difficult to differentiate between Figs. 10.2a and 10.3a. However, the second derivative spectra shown in Figs. 10.2b and 10.3b are completely different. Even a low noise of 20 μabs, which is not easily discernible, deteriorated the second derivative spectrum illustrated Fig. 10.3b.

Fig. 10.3

Lean reflectance of beef with 20 µabs noise (a) and its second derivative (b)

The presence of noise affects the estimation of the ingredients, which depends on the sample characteristics. The effects of noise on the ratio of performance to deviation (RPD) [3] of three samples are shown on Fig. 10.4.

Fig. 10.4

RPD versus noise corresponding to different samples

RPD of approximately 2.5 is obtained when the noise level is 20 mabs for the sugar in a tomato, 500 µabs for the protein in polished rice, and 50 µabs for the protein in unhulled rice. The required noise limits of the instrument are different depending on the nature of the target of measurements. A typical high-grade NIR spectrometer has a low noise level of less than 20 µabs. Such a low noise level can be realized using high-grade A/D conversion instead of a pen recorder, and the new fast repeat scanning (FRS) method developed by Karl Norris.

1.2 Noise Reduction Using the FRS Method

The noise reduction technology used for an instrument, before the advent of digital technology until 1970, involved slow speed measurements and noise reduction using the time constant of the amplifier circuit. For example, the UV-VIS or Raman spectrometers employed in that era took a long time (>10 min or sometimes hours) to measure the sample, and the spectra were generated using a long time constant.

White noise with flat frequency characteristics can be reduced by employing slow scanning speeds and longer time constants. However, the actual noise is composed of white noise and 1/f noise [4], as shown in Fig. 10.5, which has larger amplitudes at low wavelengths. The sources of the 1/f noise include temperature and time, which influence many factors such as the sensor, optical parts, mechanism, and light source of the NIR spectrometer. The actual noise has a larger amplitude at lower frequencies than the 1/f noise.

Fig. 10.5

Noise—frequency characteristics

Karl Norris employed a method with FRS to avoid the effect of high noise levels at low frequencies. For example, when the wavelength scanning speed is 100× faster, the signal frequency is 100× higher, which reduces the noise due to the 1/f noise. However, the frequency bandwidth of the signal is then 100× wider, which would require 100 × averages to obtain the same noise level as white noise.

The noise can be reduced by using a large number of averages. This method is called FRS, which cannot be realized using an analog instrument. The NIR spectrometers available in the market have a high-speed scanning mechanism of around 0.2 s per scan.

A typical waveform of the NIR spectrum is shown in Fig. 10.6a. The noise and signals measured using the slow scan and FRS methods over a 1 min duration are shown in Fig. 10.6b.

Fig. 10.6

The NIR spectral shape (a) and measured signals (b)

The signals and noise measured using the slow single scan and FRS methods are shown in Fig. 10.7.

Fig. 10.7

The signals and noise measured using the slow single scan (a) and FRS (b) methods

In the spectrum obtained with the slow single scan, as shown in Fig. 10.7a, the noise is reduced by smoothing, similar to the time constant circuit. In Fig. 10.7b, nine repeated data with noise measured using the FRS method are averaged. These results are shown in Fig. 10.8, which indicate that the FRS scan method has a lower noise level than the slow single scan.

Fig. 10.8

Spectra measured using the slow single scan and FRS methods

Karl Norris succeeded in retrieving the spectra with low noise using the FRS method and could estimate the ingredients of a material using statistics. This method can be realized only using digitalized technology. Almost all the wavelength scanning NIR spectrometers available in the market employ the above-discussed FRS method.

1.3 Noise Reduction in a Linear Array Spectrometer

Many NIR linear array spectrometers containing a linear array and grating spectrometer have been widely used for onsite ingredient measurements. A linear array spectrometer measures all the wavelengths simultaneously to ensure low noise levels. Most of the noise sources in a linear array spectrometer are intrinsic to the sensor.

The photoelectrons produced by light in a pixel of the linear array sensor are accumulated in a capacitor connected to each pixel. The number of accumulated photoelectrons in the capacitor is proportional to the incident light intensity. The noise in the linear array includes the circuit and quantum noises. Quantum noise is caused by the photoelectric effect. The number of photoelectrons produced by the incident light is proportional to the light intensity, and the noise is $\sqrt N$ when the averaged photoelectron number is $N$. The signal-to-noise ratio (SNR) (Eq. 10.2) can then be defined as follows [5].

$$ {\text{SNR}} = \frac{N}{\sqrt N } = \sqrt N $$

(10.2)

Although the noise $\sqrt N$ increases at large N, the SNR improves. This indicates that high intensity light is required to achieve a good SNR in a NIR spectrometer.

The maximum number of photoelectrons is limited by the size of the capacitor. This is called saturation exposure, which is expressed in terms of the number of electrons (Me-). As shown in Fig. 10.9, when the saturation exposure is large, the capacitance of the linear array is large along with an improved SNR. A linear array with a large saturation exposure is called a “deep well.”

Fig. 10.9

Signal and saturation exposure of a linear array

The saturation exposure ranges from 0.03 × 10⁶ to 1000 × 10⁶ (Table 10.1). In a NIR spectrometer, a saturation exposure of 500–1000 × 10⁶ is preferred to obtain a good SNR. As shown in Table 10.1, the saturation exposure of a CCD linear array is smaller, while that of the NMOS linear arrays is larger. Recently, CMOS linear arrays with large saturation exposures of around 1000 × 10⁶ have been developed.

Table 10.1 Saturation exposure of a linear array

The SNR can be improved by averaging repeated measurements even when a linear array with a small saturation exposure is used. However, in this case, noise from the circuit is added with each measurement. Hence, a linear array with a large saturation exposure is preferred, and the exposure time should be selected such that a signal close to the saturation exposure is obtained.

When a “deep well” linear array is used, the light intensity should be high to obtain sufficient signal. Therefore, the optical design that supplies light of high intensity to the linear array detector should be considered.

1.4 Noise Caused by Wavelength Accuracy and Repeatability

In the IR or Raman spectroscopic techniques, where the peak position and peak height of spectral absorption are evaluated, the wavelength accuracy is approximately a third or fifth of the wavelength resolution. In NIR spectroscopy, though all the peaks are not sharp and the wavelength resolution is around ten nm, a high wavelength accuracy is still required to observe small subtle changes in the spectrum.

The spectrum shown in Fig. 10.10a (reflectance spectrum of a leaf) has a small absorption peak at 1728 nm, with its magnified plot shown in Fig. 10.10b. The peak height of the second derivative is approximately 20 µabs. The original spectrum has a slope of 1000 µabs/nm around 1728 nm. A 0.005 nm change in the wavelength around the 1728 nm position will cause a 5 ($1000 \times 0.005$) µabs change. A higher wavelength accuracy is required where the spectrum has a large slope.

Fig. 10.10

The effect of the spectral slope on the wavelength accuracy, the reflectance of a leaf and its second derivative (a), and magnified graph around 1728 nm (b)

A laser is incorporated to calibrate wavelength with high accuracy in a Fourier-type spectrometer. A high-resolution rotary encoder is used to detect the grating rotation angle with high accuracy in a wavelength scanning-type grating spectrometer.

2 Grating Spectrometer

When considering the in-plane optics of a spectrometer, the off-plane angle δ is set to zero with the first order of diffraction, as shown in Fig. 10.11.

Fig. 10.11

The principle behind diffraction grating

The relationship between the incident angle α, diffracted angle β, and wavelength λ can be expressed by Eq. 10.3 using the groove density N of the grating (see 5.2) [6].

$$ {{\lambda }} = \frac{{10^{6} }}{N} \cdot \left( {{ \sin }\alpha + { \sin }\beta } \right) $$

(10.3)

${{\lambda }}$ :

Wavelength (${\text{nm}}$)

$N$ :

Groove density (${\text{mm}}^{ - 1}$)

$\alpha$ :

Incident angle

$\beta$ :

Diffraction angle.

The optics of the grating spectrometers can be examined based on Eq. 10.3.

2.1 Wavelength Scanning Grating Spectrometer

In a plane grating spectrometer, the parallel light beam on a grating is diffracted along the direction determined by Eq. 10.3. The diffracted parallel light is collimated into the exit slit. The wavelength can be scanned by rotating the grating.

• (a) Optical Mount of the Plane Grating Spectrometer

The Czerny-Turner mount spectrometer shown in Fig. 10.12 is the commonly used optical mount [7]. The incident light from the entrance slit is reflected by the spherical mirror to form a parallel beam, which enters the plane grating. The light diffracted from the grating surface is collimated into the exit slit by the second spherical mirror. The wavelength of the spectrometer can be calculated using Eq. 10.3, producing Eq. 10.4 as shown below.

Fig. 10.12

The Czerny-Turner mount spectrometer

$$ {\lambda } = \frac{2}{N} \cdot {\text{cos}}\gamma \cdot { \sin }\theta \cdot 10^{6} $$

(10.4)

where

λ :

Wavelength (nm)

N :

Groove density of the grating (mm⁻¹)

γ :

Deviation angle from the center axis (rad)

θ :

Rotation angle of the grating (rad).

Various other types of optical mounts have been developed, as shown in Fig. 10.13.

Fig. 10.13

Various optical mounts of a spectrometer a Ebert, b Littrow, and c Fastie Ebert

A grating spectrometer has stray light depending on the location of the optical axis on the principal plane [8]. The spectrometer should be designed such that any stray light is effectively eliminated.

• (b) Concave Grating with Constant Interval Grooves—Seya-Namioka Mount

A spectrometer using a concave grating with constant interval grooves does not require any other additional optics (Fig. 10.14). This results in a simple instrument with minimized optical loss. In the Seya-Namioka mount [6, 9], the optical configuration can be calculated using Eqs. 10.5 and 10.6. The wavelength can be determined using Eq. 10.4.

Fig. 10.14

Seya-Namioka mount

$$ 2\gamma = 70.5^\circ $$

(10.5)

$$ R = R^{\prime} = r \cdot { \cos }\,35.25^\circ $$

(10.6)

r :

Radius of curvature of the concave grating (mm)

R :

Entrance slit position (mm)

R′:

Exit slit position (mm).

The wavelength is calculated using Eq. 10.4.

• (c) Concave Grating of Constant Deviation

Using a concave grating with grooves at unequal intervals, a spectrometer with a constant deviation angle of 2γ can be realized (Fig. 10.15). The specific angle γ, and focal points R and R′ are determined based on the design of this grating [6]. Although the groove interval is not constant, the nominal groove density of the grating N is used in Eq. 10.4 to determine the wavelength.

Fig. 10.15

Constant deviation mount using a concave grating

2.2 Spectrometer with a Linear Array Detector

A spectrum can be measured by deploying a linear array detector at the exit slit position of a grating spectrometer [6]. This type of spectrometer has a compact size and enables rapid measurement. The NIR spectrometer can be realized using this type of spectrometer for specific samples.

• (a) Linear Array Spectrometer with a Plane Grating

The focal point of each wavelength at the exit slit of a grating spectrometer should be on a flat plane to detect the spectrum without blurring, as the detector plane of a linear array is flat.

In a grating spectrometer where all the optical axes are on the principle plane, such as the Czerny-Turner, Ebert, or Littrow mounts shown in Fig. 10.13, the positions of the grating collimating spherical mirror, and linear array determine the flatness of the focal plane. The grating position shown in Fig. 10.16, which is $1/\sqrt 3 \cdot R$ from the center of the radius curvature of the collimating spherical mirror, is known as a super flat condition [7] that achieves the flattest focal plane.

Fig. 10.16

The super flat condition for a linear array grating spectrometer

Astigmatism perpendicular to the slit image is generated in the Fastie-Ebert spectrometer (Fig. 10.13c) where the optical axis is out of the principle plane. However, by deploying the linear array just above the grating, the image plane becomes approximately flat in a narrow wavelength band without any stray light. Thus, this mount is appropriate for specific NIR applications.

The configuration parameters of the Czerny-Turner linear array spectrometer (Fig. 10.17), i.e., the focal length, grating position, angle γ, and angle ϑ are determined by Eq. 10.4 using the center wavelength $\lambda_{0}$, as shown in Eq. 10.7.

Fig. 10.17

Czerny-Turner linear array spectrometer

$$ \lambda_{0} = 2/N \cdot { \cos }\,{{\gamma }} \cdot { \sin }\theta \cdot 10^{6} $$

(10.7)

The angles $\beta_{1}$ and $\beta_{2}$ corresponding to $\lambda_{ \text{min} }$ and $\lambda_{ \text{max} }$, respectively, can be determined using Eqs. 10.8 and 10.9.

$$ \lambda_{\text{min} } = \frac{{10^{6} }}{N} \cdot \left( {\sin \left( {\theta - \gamma } \right) + \sin \beta_{1} } \right) $$

(10.8)

$$ \lambda_{\text{max} } = \frac{{10^{6} }}{N} \cdot \left( {\sin \left( {\theta - \gamma } \right) + \sin \beta_{2} } \right) $$

(10.9)

All configurations of the spectrometer can be determined based on the above calculations.

• (b) Concave Grating with Constant Interval Grooves

When a concave grating with constant interval grooves is used, the incident light through an entrance slit on the Rowland circle [6] is diffracted and focused back onto the same Rowland circle (Fig. 10.18). The Rowland circle has a diameter R, which is the radius of curvature of the concave grating and is in contact with its surface.

Fig. 10.18

A concave grating mount for a linear array spectrometer

The relationship between α, β1, and β2 can be described using Eqs. 10.7–10.9.

The spectra can be measured by deploying a linear array on the Rowland circle. Some blurring is inevitable as the focal plane of the spectra is on the Rowland circle, while the linear array sensing area is on the flat plane.

• (c) Flat Field Concave Grating

By adopting grooves with uneven/unequal intervals using holographic technology, a flat field concave grating with a flat spectral focal plane can be realized [6] (Fig. 10.19).

Fig. 10.19

Flat field concave grating for a linear array spectrometer

The configuration of the spectrometer (R1, α, R4, β3) can be determined based on the grating design. The other parameters (β1, β2) can be calculated using Eqs. 10.7–10.9.

• (d) Volume Phase Holographic (VPH) Grating

A grating with high efficiency is preferred for NIR spectrometer, which requires a high intensity light signal to decrease the noise. The VPH grating, which is based on Bragg diffraction, has a high efficiency of approximately 90% in the narrow wavelength band [10].

The VPH grating is a transmissive-type grating that is fabricated using holographic technology. The interference fringes generated by a laser are recorded three-dimensionally in a photosensitive material sandwiched between glass plates. The three-dimensional periodic modulation in the material causes Bragg diffraction, which occurs when α = β. This enables an efficiency of >90% and produces the characteristics of the narrow wavelength range based on its thickness. Therefore, this grating is used around the Bragg condition. The anomaly that appears in a plane grating is not observed.

The Bragg condition is calculated using Eqs. 10.10 and 10.11. Angles $\beta_{1}$ and $\beta_{2}$ corresponding to $\lambda_{\text{min} }$ and $\lambda_{\text{max} }$, respectively, are calculated using Eqs. 10.12 and 10.13 (Fig. 10.20).

Fig. 10.20

Optics of a VPH grating

$$ \alpha = \beta $$

(10.10)

$$ \lambda_{0} = 2 \cdot \frac{{10^{6} }}{N} \cdot \sin \alpha $$

(10.11)

$$ \lambda_{\text{min} } = \frac{{10^{6} }}{N} \cdot \left( {\sin \alpha + \sin \beta_{1} } \right) $$

(10.12)

$$ \lambda_{\text{max} } = \frac{{10^{6} }}{N} \cdot \left( {\sin \alpha + \sin \beta_{2} } \right) $$

(10.13)

Higher-order diffraction does not exist in Bragg diffraction. Therefore, a cut filter is not necessary in a VPH spectrometer to eliminate second and higher order diffraction. The VPH grating is easy to handle due to its sandwich structure. The surface of the VPH grating can be wiped or polished. The typical optical alignment of a VPH grating spectrometer is shown in Fig. 10.21.

Fig. 10.21

Optical mount of a VPH grating spectrometer

2.3 Hadamard Spectrometer

In a Hadamard spectrometer, an image of the spectrum is focused onto the exit slit position of a grating spectrometer. The light passing through a multi-aperture plate deployed at the exit slit position is detected by a single detector (Fig. 10.22). This measurement is repeated more than n times for different multi-aperture plates. The signal intensity of the n signals corresponding to the different multi-aperture plates can be expressed in terms of Eq. 10.14.

Fig. 10.22

Hadamard spectrometer

$$ I_{k} = \mathop \sum \limits_{i = 1}^{n} P_{i}^{k} \cdot I_{i} $$

(10.14)

where

$I_{k}$:

n-summed detector signal corresponding to the different multi-apertures

$P_{i}^{k}$:

Multi-aperture pattern

$I_{i}$:

Spectral component.

Equation 10.14 can be rewritten as Eq. 10.15. The spectral shape can be calculated from the n signals and n aperture patterns [11]. This is called the Hadamard spectrometer.

$$ \left( {\begin{array}{*{20}c} {M_{1} } \\ \vdots \\ {M_{k} } \\ \end{array} } \right) = \left( {\begin{array}{*{20}c} {P_{1}^{1} } & \cdots & {P_{1}^{k} } \\ \vdots & \ddots & \vdots \\ {P_{n}^{1} } & \cdots & {P_{n}^{1} } \\ \end{array} } \right) \cdot \left( {\begin{array}{*{20}c} {I_{1} } \\ \vdots \\ {I_{k} } \\ \end{array} } \right)\,{\text{or}}\,\varvec{M} = \varvec{P} \times \varvec{I} $$

(10.15)

In Eq. 10.15, matrix $\varvec{I}$ can be calculated using $\varvec{P}^{ - 1}$, which is the inverse matrix of s $\varvec{P}$ (Eq. 10.16).

$$ \varvec{I} = \varvec{P}^{ - 1} \times \varvec{M} $$

(10.16)

When noise $\varvec{e}$ is present in the measurements, the signal can be expressed in terms of Eq. 10.17.

$$ \varvec{M} = \varvec{P} \times \varvec{I} + \varvec{e} $$

(10.17)

The result of Eq. 10.15 would then be Eq. 10.18, as shown below.

$$ \varvec{I} = \varvec{P}^{ - 1} \times \left( {\varvec{M} - \varvec{e}} \right) = \varvec{P}^{ - 1} \times \varvec{M} - \varvec{P}^{ - 1} \times \varvec{e} $$

(10.18)

Finally, the noise in the spectrum would be $\varvec{P}^{ - 1} \times \varvec{e}$, which can be minimized using a multi-aperture pattern. This is known as Hadamard transform spectroscopy. Although this method was proposed around 1970, it did not become popular because it was difficult to realize a dynamic n multi-aperture mechanism. However, the MEMS DLP element has been used as a multi-aperture since the advent of MEMS technology around 1990. This type of spectrometer has been available since 2015 [12]. Using the MEMS DLP element, users can design their own multi-aperture patterns for the desired applications.

2.4 Wavelength Resolution and Measurement Interval

• (a) Wavelength Resolution of a Grating Spectrometer

A wavelength resolution of five to ten nm is sufficient for a NIR spectrometer due to the broad absorbance peak of the NIR spectra. The light intensity attained by the grating spectrometer is proportional to the square of the wavelength resolution. The lower the wavelength resolution of a NIR spectrometer, the higher is the signal intensity, which results in a good SNR. Therefore, the minimum possible low wavelength resolution should be selected. However, as explained later in Sect. 10.4, a lower wavelength resolution might cause greater instrumental differences induced by the spectral response of the spectrometer. Thus, both of SNR and instrumental difference should be considered.

As the grating spectrometer for NIR spectroscopy has a wavelength resolution of five to ten nm, the wavelength resolution of a spectrometer is decided mainly by the wavelength dispersion of the grating. Other factors such as the aberration of the optics, optical surface accuracy, and insufficient adjustment of the focal point increase the wavelength resolution of spectrometer. These factors will cause differences in the wavelength resolution even between the same spectrometer designs, and the instrumental differences will be larger depending on the different wavelength resolutions. The difference between the wavelength resolutions of the instruments should be within 10% to avoid any instrumental differences.

The wavelength resolution resulting from the dispersion caused by the entrance and exit slits can be described as follows [6, 13].

The dependence of the position deviation on the entrance slit and wavelength can be calculated by differentiating Eq. 10.3. The relationship between the wavelength and incident angle $\alpha$ or slit position can be expressed using Eq. 10.19. The slit wavelength width corresponding to the mechanical slit width can be expressed using Eq. 10.20.

$$ \frac{{{\text{d}}\lambda }}{{{\text{d}}\alpha }} = \frac{{10^{6} }}{N} \cdot \cos \alpha \cdot \frac{{{\text{d}}\lambda }}{{{\text{d}}w_{\text{in}} }} = \frac{{10^{6} }}{N \cdot f} \cdot \cos \alpha $$

(10.19)

$$ \omega_{\text{in}} = \frac{{10^{6} }}{N \cdot f} \cdot \cos \alpha \cdot W_{\text{in}} $$

(10.20)

$\omega_{\text{in}}$ :

Entrance slit wavelength width

$W_{\text{in}}$ :

Mechanical entrance slit width

$f$ :

Focal length of the spectrometer.

At the exit slit, the relationship between the wavelength and diffracted angle or slit position can be expressed in terms of Eq. 10.21, while the exit slit wavelength width caused by the mechanical slit width can be expressed by Eq. 10.22.

$$ \frac{{{\text{d}}\lambda }}{{{\text{d}}\beta }} = \frac{{10^{6} }}{N} \cdot \cos \beta \cdot \frac{{{\text{d}}\lambda }}{{{\text{d}}w_{\text{exit}} }} = \frac{{10^{6} d \cdot \cos \alpha }}{N \cdot f} $$

(10.21)

$$ \omega_{\text{exit}} = \frac{{10^{6} \cdot cos\alpha }}{N \cdot f} \cdot W_{\text{exit}} $$

(10.22)

$\omega_{\text{exit}}$ :

Exit slit wavelength width

$W_{\text{exit}}$ :

Exit slit mechanical width.

The shape of the total slit function of the spectrometer will be trapezoidal, with the top side and base line given by $\omega_{\text{in}} - \omega_{\text{exit}}$ and $\omega_{\text{in}} + \omega_{\text{exit}}$, respectively (Fig. 10.23b). When $\omega_{\text{exit}}$ is small, the shape in Fig. 10.23a is obtained, while a triangle shape is obtained when $\omega_{\text{exit}} = \omega_{\text{in}}$ (Fig. 10.23c).

Fig. 10.23

Slit function of the spectrometer

In a wavelength scanning grating spectrometer, the $\omega_{\text{exit}} = \omega_{\text{in}}$ condition is selected to achieve the highest intensity at the same wavelength resolution.

In a linear array grating spectrometer, the pixel width is assumed to be the exit slit width and $\omega_{\text{exit}} \cdot \omega_{\text{in}}$. The slit function is illustrated in Fig. 10.23b.

• (b) Wavelength Interval for Measurement

Selecting the right wavelength interval for a measurement is important to obtain the right spectral shape. Thus, the spectral shape is influenced by the wavelength interval [13].

When the wavelength interval is larger than the wavelength resolution, some spectral components between the measurement points will not be measured. When the interval is too narrow, some spectral components will be overlapped in the measurement. The following important aspects should be considered to enable an even/equal measurement of all the spectral components.

• (b-1) Wavelength Scanning Grating Spectrometer

The shape of the slit function of a wavelength scanning grating spectrometer is usually triangular with a half bandwidth (HBW) of $\omega$, as shown in Fig. 10.23.

When the wavelength interval $\Delta \lambda$ is $\omega$ (Fig. 10.23b) or $\frac{\omega }{n}$ (Eq. 10.23), all the spectral components will be measured evenly.

$$ {\Delta }\lambda = \frac{\omega }{n} $$

(10.23)

${\Delta }\lambda$ :

Wavelength interval

${\omega }$ :

HBW of the slit function

$n$ :

Integer 1, 2, …

When the wavelength interval ${\Delta }\lambda \ne \frac{\omega }{n}$ (Fig. 10.23a, c), the spectral components will not be measured evenly (Fig. 10.24). However, a large $n$ in Eq. 10.23 will result in a smaller error.

Fig. 10.24

Wavelength interval and wavelength resolution

• (b-2) Linear Array Grating Spectrometer

The entrance slit wavelength width $\omega_{\text{in}}$ is always larger than the pixel width $\omega_{{{\text{pix}} - w}}$. The slit function of the signal from a pixel of the linear array in a spectrometer cannot be triangular. This is shown in Fig. 10.25, which is the same as Fig. 10.23b, where the exit slit wavelength width $\omega_{\text{exit}}$ is replaced by the pixel width wavelength $\omega_{{{\text{pix}} - w}}$.

Fig. 10.25

Slit function of a pixel

This slit function is lined-up with the interval of the pixel interval wavelength width $\omega_{{{\text{pix}} - i}}$. When the entrance slit wavelength width $\omega_{\text{in}}$ is n times the pixel interval wavelength width $\omega_{{{\text{pix}} - i}}$ (Eq. 10.24), all the spectral components are evaluated evenly. The relationship between the mechanical size of the entrance slit $W_{\text{in}}$ and pixel interval $W_{{{\text{pix}} - i}}$ is also described in Eq. 10.24.

$$ \omega_{\text{in}} = n \cdot \omega_{{{\text{pix}} - i}} \cdot W_{\text{in}} = n \cdot \frac{\cos \beta }{\cos \alpha } \cdot W_{{{\text{pix}} - i}} $$

(10.24)

However, the wavelength widths of the slit and pixel vary according to the wavelength, and it is impossible to realize the condition defined in Eq. 10.24 at all wavelengths. Therefore, this condition can be realized around the center wavelength region.

• (c) Calculation of the Absorbance Data

At the wavelength $\lambda_{j}$, which is determined by the hardware of the spectrometer, the detector signal intensity is acquired for the white reference $W\left( {\lambda_{j} } \right)$ and sample $X\left( {\lambda_{j} } \right)$. The wavelength interval of the measurement is decided by the encoder in a wavelength scanning-type spectrometer and by a pixel in a linear array spectrometer and is not a constant round number, i.e., 1 nm or 0.5 nm.

Using the data $W\left( {\lambda_{j} } \right)$ and $X\left( {\lambda_{j} } \right)$ at the wavelength determined by the hardware, the reflectance, transmittance, or absorbance values at even wavelengths should be calculated. The process used for the calculations is described in the following sub-sections. The sequence of the calculations is important for avoiding the instrumental difference.

• (c-1) Calculation of the Absorbance

Before processing the data, the transmittance or reflectance $D\left( {\lambda_{j} } \right)$ should be calculated according to Eq. 10.25.

$$ D\left( {\lambda_{j} } \right) = \frac{{X\left( {\lambda_{j} } \right)}}{{W\left( {\lambda_{j} } \right)}} $$

(10.25)

$\lambda_{j}$ :

Wavelength used for the measurement

$W\left( {\lambda_{j} } \right)$ :

Signal intensity of the white reference

$X\left( {\lambda_{j} } \right)$ :

Signal intensity of the sample

():

Reflectance or transmittance.

• (c-2) Calculation of the Wavelength Spectra

Using the calculated reflectance or transmittance $D\left( {\lambda_{j} } \right)$, the data $S\left( {\lambda_{n} } \right)$ are calculated using Eq. 10.26 at each even wavelength, i.e., 1000, 1001, and 1002 nm.

$$ S\left( {\lambda _{n} } \right) = \frac{{\sum\nolimits_{{j = f}}^{g} {\left[ {\left( {\Delta - \left| {\lambda _{n} - \lambda _{j} } \right|} \right) \cdot D\left( {\lambda _{j} } \right)} \right]} }}{{\sum\nolimits_{{j = f}}^{g} {\left( {\Delta \cdot D\left( {\lambda _{j} } \right)} \right)} }} $$

(10.26)

$S\left( {\lambda_{n} } \right)$ :

Reflectance or transmittance at any wavelength $\lambda_{n}$ (nm)

$D\left( {\lambda_{j} } \right)$ :

Reflectance or transmittance at the wavelength $\lambda_{j}$ (nm) determined by the hardware

${\Delta }$ :

Wavelength resolution used in the calculations (nm)

$j$ :

Measurement number (a smaller number corresponds to a shorter wavelength)

$f$ :

Minimum measurement number that satisfies the condition $\left( {\lambda_{n} - \lambda_{f} } \right) < {\Delta }$

$g$ :

Maximum measurement number that satisfies the condition $\left( {\lambda_{g} - \lambda_{n} } \right) < {\Delta }$.

Based on the above calculations, the reflectance or transmittance at any wavelength $\lambda_{n}$ with the wavelength resolution ${\Delta }$ can be acquired. The actual wavelength resolution following these calculations will be a convolution of the slit function with the wavelength resolution ${\omega }$, and the triangular shape with the wavelength resolution ${\Delta }$. If ${\omega }$ is very small compared to ${\Delta }$, the actual wavelength resolution will be close to ${\Delta }$. If ${\omega }$ and ${\Delta }$ are almost identical, then the total wavelength resolution will be approximately $1.43 \times {\Delta }$.

• (c-3) Preprocessing of the Spectral Data

The absorbance $A\left( {\lambda_{n} } \right)$ of the spectrum $S\left( {\lambda_{n} } \right)$ can be calculated using Eq. 10.27 if required.

$$ A\left( {\lambda_{n} } \right) = - \log \left( {S\left( {\lambda_{n} } \right)} \right) $$

(10.27)

Using the spectral data $A\left( {\lambda_{n} } \right)$ or $S\left( {\lambda_{n} } \right)$, the calibration can be established using statistics. Various data preprocessing steps, such as smoothing, can be applied prior to the statistical analysis to obtain good calibrations.

3 Designing a NIR Spectrometer for Special Materials

The main application of NIR spectroscopy is to measure the ingredients of a certain material, which is an easy and non-destructive method used onsite. An appropriate method of designing such a spectrometer is explained in the following sections. The performance of the desired instrument should be examined before commencing the design. It is particularly critical to assess the permissible noise levels with respect to the desired instrument.

When an instrument is designed without knowledge of the allowable noise levels for a particular application, the instrument should be repeatedly improved until satisfactory calibration is achieved. Generating calibrations repeatedly to evaluate the instrument is a waste of time and money.

The efficient design process of NIR spectroscopy with minimum waste is explained in Fig. 10.26. The process involves three steps, namely the first test measurement, the second step to determine the specification, and the third step involving the final manufacture of the instrument.

Fig. 10.26

Flow chart showing the design process of a NIR spectrometer

3.1 The First Step: Test Measurement

At least thirty samples, which have various ingredient values, are prepared and the spectra are measured using a high-grade laboratory-type NIR spectrometer. The true ingredient values of the samples are also analyzed using other official/established methods. The calibration is retrieved from the spectra and ingredient values using multivariate statistical analysis.

If the calibration performance is not satisfactory, the conditions (optics, wavelength range etc.) are changed and measurements are repeated until a satisfactory calibration is established.

Once satisfactory calibration is established, the second step is implemented to determine the specifications of the instrument hardware using this calibration data. The NIR method is replaced if satisfactory results are not obtained after many trials.

3.2 The Second Step: Determining the Specification

Noise of various amplitudes is added to the acquired spectral data via simulations and the calibrations are generated for each noise amplitude. The allowable noise level can be determined based on the relationship between the noise amplitude and calibration performance (R², SEP and RPD). Other performance parameters such as the wavelength resolution and wavelength accuracy can also be determined similarly.

The sampling optics should be decided in this step. Using experimental sampling optics to determine the specification, the samples are measured to check whether the optics generated the appropriate spectral shape for this application. It is important to note that the purpose of the measurement is not just to obtain the correct spectral shape but also to generate an appropriate spectral shape with good SNR, which in turn provides good calibration for estimating the ingredients.

3.3 The Third Step: Manufacturing

The NIR instrument is designed according to the determined specification and manufactured. This process ensures the development of a NIR instrument with satisfactory performance. Subsequently, the satisfactory calibration can be established using this developed instrument.

4 Instrumental Differences

4.1 Effect of Instrumental Differences

The main application of NIR spectroscopy is not to study the molecular structure but to measure the ingredients of a material. Therefore, many NIR instruments are being employed at various sites for manufacturing or agriculture. Even when the same type of instruments and same calibration are used, the measured ingredients differ due to a phenomenon known as the instrumental difference.

In UV-VIS or infrared spectroscopy, the main purpose of the measurement is to obtain the spectral waveform, and a small difference in the spectral shape does not present a major problem. However, in NIR spectroscopy, a small subtle difference in the spectral shape results in large differences in the predicted ingredient values, thus presenting a significant problem.

It is necessary to establish a calibration for each instrument when the instrumental difference is large. However, generation of the calibration requires substantial cost and time, which is a wasteful process. It is important to understand the factors causing the instrumental differences to find an effective solution. Instruments with small differences can be effectively designed by understanding the underlying causes.

4.2 Instrumental Differences Caused by the Sampling Optics

The spectral shape of the light reflected or transmitted by the sample depends on the physical configuration of the sample optics. During the manufacture of the sample optics, the angle of the optical axis and position of the optical parts should be precisely adjusted. The distance between the irradiated and observed areas should be considered to generate the identical spectra when manufacturing the interactance optics.

Furthermore, each lamp has light intensity angular characteristics due to the filament shape, and each spectrometer also has angular sensitivity dependence. Thus, when the lamp or entrance slit is collimated onto the sample, the angular characteristics of the lamp and spectrometer influence the spectral shape. The non-collimated optics and fiber optics are effective for eliminating the spectral shape difference, although the light intensity reduces.

4.3 Instrumental Differences Caused by the Spectral Sensitivity and Slit Function

Based on the spectral sensitivity of the spectrometer $H\left( \lambda \right)$ and slit function $\Delta \left( a \right)$, the measured signals for the sample $x\left( {\lambda_{0} } \right)$ and reference $w\left( {\lambda_{0} } \right)$ can be expressed by Eqs. 10.28 and 10.29 [14].

$$ x\left( {\lambda_{0} } \right) = \frac{{\mathop \int \nolimits_{ - D}^{ + D} X\left( {\lambda_{0} - a} \right) \cdot H\left( {\lambda_{0} - a} \right) \cdot \Delta \left( a \right) \cdot da}}{{\mathop \int \nolimits_{ - D}^{ + D} \cdot \Delta \left( a \right) \cdot da}} $$

(10.28)

$$ w\left( {\lambda_{0} } \right) = \frac{{\mathop \int \nolimits_{ - D}^{ + D} W\left( {\lambda_{0} - a} \right) \cdot H\left( {\lambda_{0} - a} \right) \cdot \Delta \left( a \right) \cdot da}}{{\mathop \int \nolimits_{ - D}^{ + D} \Delta \left( a \right) \cdot da}} $$

(10.29)

where

$x\left( {\lambda_{0} } \right)$ :

Sample signal intensity

$w\left( {\lambda_{0} } \right)$ :

Reference signal intensity

$\lambda_{0}$ :

Wavelength of the spectrometer

$X\left( {\lambda_{0} } \right)$ :

Sample spectrum

$W\left( {\lambda_{0} } \right)$ :

Reference spectrum

$a$ :

Wavelength shift from ${\lambda }_{0}$

$H\left( \lambda \right)$ :

Spectral sensitivity of the spectrometer

$\Delta \left( a \right)$ :

Slit function of the spectrometer

$D$ :

Integral width of the slit function.

Using Eqs. 10.28 and 10.29, the reflectance (or transmittance) spectrum ${\text{R}}\left( {\lambda_{0} } \right)$ can be calculated (Eq. 10.30) [15].

$$ R\left( {\lambda_{0} } \right) = \frac{{x\left( {\lambda_{0} } \right)}}{{w\left( {\lambda_{0} } \right)}} \cdot W\left( {\lambda_{0} } \right) = \frac{{\mathop \int \nolimits_{ - D}^{ + D} X\left( {\lambda_{0} - a} \right) \cdot H\left( {\lambda_{0} - a} \right) \cdot \Delta \left( a \right) \cdot da}}{{\mathop \int \nolimits_{ - D}^{ + D} W\left( {\lambda_{0} - a} \right) \cdot H\left( {\lambda_{0} - a} \right) \cdot \Delta \left( a \right) \cdot da}} \cdot W\left( {\lambda_{0} } \right) $$

(10.30)

Equation 10.30 can be expressed as Eq. 10.31 when the reference spectrum $W\left( {\lambda_{0} } \right)$ is constant within the integral width $D$.

$$ R\left( {\lambda_{0} } \right) = \frac{{\mathop \int \nolimits_{ - D}^{ + D} X\left( {\lambda_{0} - a} \right) \cdot H\left( {\lambda_{0} - a} \right) \cdot \Delta \left( a \right) \cdot da}}{{\mathop \int \nolimits_{ - D}^{ + D} H\left( {\lambda_{0} - a} \right) \cdot \Delta \left( a \right) \cdot da}} $$

(10.31)

It is obvious from Eq. 10.31 that the reflectance $R\left( \lambda \right)$ is influenced by the spectral sensitivity $H\left( \lambda \right)$ and slit function $\Delta \left( a \right)$ [15].

Variations in $H\left( \lambda \right)$ within the integral width $\pm D$ cause instrumental differences. Variations in $H\left( \lambda \right)$ can mainly be attributed to the detector and grating. These optical elements should not exhibit steep/sharp changes within the integral width $\pm D$. The linear array has an etaloning effect that causes periodic sensitivity variations with respect to the wavelength. This effect differs depending on the manufacturing lot and is a significant factor producing of the instrumental difference.

When the variation of $H\left( \lambda \right)$ within the integral width $\pm D$ is small and considered constant, the slit function $\Delta \left( a \right)$ will be the only source affecting the reflectance spectral shape.

The slit function is a wavelength response of the spectrometer with respect to monochromatic light and is determined by the slit width and wavelength dispersion at the entrance and exit slits in the case of a grating spectrometer. The width of a slit function, which is equal to the wavelength resolution, increases due to aberrations in the optical system, mirror surface imprecision, and insufficient adjustment of the spectrometer optics. The mirror surface precision should be within $1/4 \cdot \lambda$. The adjustments should be checked by measuring the sharp spectral lines from a mercury or argon discharge lamp. The wavelength resolution of the grating-type NIR spectrometer should be adjusted to <0.1 nm of the master instrument to avoid instrumental difference.

4.4 Considerations to Avoid Instrumental Differences

The following factors should be considered to minimize the instrumental difference [15].

• The characteristics of the sample optics should be made approximately equal via adjustments.

• The wavelength resolution of each spectrometer should be adjusted within 0.1 nm.

• The spectral sensitivity of the spectrometer should be as flat as possible.

• A wide wavelength resolution generates low noise levels; however, the instrumental difference increases.

• The sequence of calculations is important (see Sect. 10.2.4c). The reflectance should be calculated first.

4.5 Standardization Methods for the Calibration

To compensate for the instrumental difference, the calibration or predicted results should be modified. The modification process is called standardization. Several standardization methods have been reported [16].

• (a) The SBC Method

The value predicted by a slave instrument using the calibration established by a master instrument can be corrected using the slope and bias correction (SBC) method [17]. As shown in the scatter plot in Fig. 10.27, the relationship between the predicted values from the master and slave instruments can be calculated.

Fig. 10.27

The slope and bias correction (SBC) method

The spectra of about 30 samples should be measured by the master and slave instruments. Subsequently, the same calibration is applied to both the instruments to obtain the predicted value. Using the predicted values from the master and slave instruments, the relationship between these two instruments (slope and bias) is determined using regression analysis. These values are then applied to the value predicted by the slave instrument, as defined in Eq. 10.32.

$$ A_{K} = {\text{Slope}} \cdot \sum K_{{\lambda_{i} }} \cdot I_{{\lambda_{i} }} + {\text{Bias}} $$

(10.32)

A _k :

Corrected predicted value from the slave instrument

K _λi :

Calibration by the master instrument

I _λi :

Sample spectrum obtained using the slave instrument.

This method can be applied when the instrumental difference is small.

• (b) Shenk Method

Shenk and Westerhaus proposed this method in a US patent [18] in 1991 (Fig. 10.28). Though this method is old, it is nevertheless important.

Fig. 10.28

The Shenk method

The principle idea is to identify a function that modifies the spectra from the slave instrument to match that of the master instrument. Subsequently, the calibration of the master instrument can be used for the slave instrument.

A minimum of 30 samples are measured using both the master and slave instruments. It is desirable that the samples cover all the features of the target.

Based on the correlation between the absorbance at $\lambda_{i}$ of the master and $\lambda_{j}$ of the slave, the relationship between $\lambda_{i}$ of the master and $\lambda_{j}$ of the slave can be acquired. Using these results, the wavelength of the slave instrument can be corrected.

Based on the absorbance values of the master and slave instruments at each corrected wavelength $\lambda_{j}$, the correction factors for the slope and bias corresponding to the absorbance value at each wavelength can be retrieved. Using the correction matrix that includes all the corrections for the wavelength and absorbance, the spectra of the slave instrument match those of the master. Subsequently, the ingredients can be calculated using the slave spectra and calibration established by the master.

• (c) PDS Method

The piecewise direct standardization (PDS) method is similar to the Shenk method. In this method, the wavelength and absorbance are corrected using multivariate analysis [19] (Fig. 10.29).

Fig. 10.29

The PDS method

Using the absorbance values of K samples from the master and slave instruments, the $A_{{\lambda_{i} }}^{K}$ of the master at wavelength $\lambda_{i}$ can be calculated based on the slave absorbance values ($C_{{\lambda_{i} }}^{K}$) in the wavelength band $\lambda_{i - X}$ to $\lambda_{i + X}$ using the coefficient ($\beta_{n}^{{\lambda_{i} }}$). The coefficient ($\beta_{n}^{{\lambda_{i} }}$) can be calculated from the MLR, PCA, or PLS.

The direct standardization (DS) method involves using all the wavelength data of the slave instrument.

In the calculation of the above corrections, it is important to ensure the repeatability of the measurements for this standardization.