Spectral Analysis in the NIR Spectroscopy

Abstract

This chapter is concerned with the introduction to spectral analysis in the NIR spectroscopy. It consists of two major parts, conventional spectral analysis and spectra pretreatments. In the former, various conventional spectral analysis methods such as group frequency analysis, derivative spectra, difference spectra, spectral analysis based on perturbation, comparison of a NIR spectrum with the corresponding IR spectrum, and isotope exchange experiments are explained. In the latter part smoothing, derivative methods, multiplicative scatter correction (MSC), standard normal variate (SNV), centering methods, and normalization are described.

1 Introduction to Spectral Analysis in the NIR Region

As described partly in Chaps. 1 and 2 there are various kinds and various types of NIR spectra [1,2,3,4,5,6,7]. First of all, NIR spectra can be divided into electronic spectra and vibrational spectra. However, in this chapter, we treat only vibrational spectra of solids and liquids. Compared with IR spectroscopy diversity of the types of NIR spectra is quite large because NIR spectroscopy is concerned with so many kinds of materials from pure samples such as pure liquids, solutions, and crystals to bulk materials including raw materials, industrial products, and natural products. To look at the diversity of NIR spectra let us compare the spectrum of methanol (0.005 M, in CCl₄; see Fig. 2.3) with that of flour (Fig. 4.1). The former is rather simple although several bands are overlapped in the 4500–4000 cm⁻¹ region. It has very little baseline change. On the other hand, the spectrum of flour consists mainly of the spectra of water, starch, and proteins. Bands are broad, and its baseline increases with the increase in the wavelength. Like this NIR spectra show significant diversity depending on samples and conditions. Therefore, spectral analysis methods are also diverse in NIR spectroscopy [1,2,3]. In other words, one must select the best spectral analysis method for a target. The spectral analysis methods must change with samples, sample conditions, measurement methods, and the purpose of analysis. The purpose of analysis is also wide spread varies from quantitative analysis, qualitative analysis, and sample identification to studies of molecular structure and chemical reaction. Therefore, when one selects a spectral analysis method, one must consider the purpose of analysis. For many purposes, chemometrics is very useful but for some purposes it is almost meaningful. In this chapter, general introduction to the spectral analysis methods and spectral pretreatments for NIR spectra are outlined. The detailed explanations of representative spectral analysis methods such as chemometrics (Chap. 7), two-dimensional correlation spectroscopy (2D-COS, Chap. 6), and quantum chemical calculations (Chap. 5) will be given in each chapter and session.

Fig. 4.1

A NIR spectrum of flour. Measured by A. Ikehata

As in the cases of IR and Raman spectroscopy, band assignments are always the base for spectral analysis of NIR spectroscopy. However, the assignments are generally not straightforward since a number of bands originating from overtones and combinations overlap each other. In some cases, bands arising from combinations including combinations of overtones appear; for such cases, it is very difficult to make accurate band assignment. In NIR spectroscopy detailed band assignments are often not necessary, but even in such cases, one should know from which functional group a band arises.

Spectral analysis methods in NIR spectroscopy can be divided into conventional spectral analysis method, chemometrics [3], quantum chemical calculation [5, 8], and 2D-COS [1]. The conventional spectral analysis methods are, more or less, common among NIR, IR, Raman, and Terahertz/far-IR(FIR) spectroscopy. One must know that they yield also a base for chemometrics, quantum chemical calculation, and 2D-COS. Chemometrics has most often been employed to extract rich quantitative and qualitative information from NIR spectra (Chap. 7). A major part of chemometrics is multivariate data analysis such as principal component analysis/regression (PCA/PCR) and partial least squares regression (PLSR), however, self-modeling curve resolution (SMCR), which is used to predict pure component spectra and pure component concentration profiles from a set of NIR spectra, is also becoming more and more significant (Chap. 7). Using quantum chemical calculations such as density function theory (DFT) calculations, one can calculate the intensities and frequencies of overtones and combination bands (Chap. 5). Quantum chemical calculation is still not always popular in NIR spectroscopy but it has already been applied not only to simple compounds but also to rather complicated molecules such as long-chain fatty acids, nucleic acid bases, and rosemaric acid [8]. 2D-COS is not a general method but it is often useful to unravel complicated NIR spectra (Chap. 6). In addition, neural network, AI, and machine learning have been started to be used to analyze NIR spectra. They are very promising methods for the spectral analysis in NIR spectroscopy (Chap. 7). However, one should know that in the early 1990 s neural network has already been tried to be applied to NIR spectra [9].

2 Conventional Spectral Analysis Method

Various kinds of conventional spectral analysis methods are used in NIR spectroscopy [1, 2, 6, 7]. They are summarized as follows:

• (1) Spectral analysis based on group frequencies

This is a traditional method established in IR and Raman spectroscopy. Spectral analysis based on group frequencies built for the fundamentals is modified for overtones and combinations. Each functional group such as OH and CH groups shows characteristic bands in particular regions. One can find tables for group frequencies in the NIR region in a few NIR textbooks [1, 6].

• (2) Calculation of derivative spectra

Derivative methods have long been popular in various spectroscopies [1,2,3, 6]. They are useful for resolution enhancement as well as baseline correction. Figure 4.2A, B shows a good example demonstrating the usefulness of the second derivative [10]. In Fig. 4.2A, NIR spectra in the 7500–5500 cm⁻¹ region are shown for water-methanol mixtures with a methanol concentration of 0–100 wt% at increments of 5 wt% at 25 °C. Figure 4.2B, a gives an enlargement of the 6000–5700 cm⁻¹ region of the NIR spectra as shown in Fig. 4.2a. In the 6000–5700 cm⁻¹ region, many bands due to the overtones and combination of the CH stretching modes of CH₃ group of methanol are expected to appear. Figure 4.2B, b displays the second derivative of the spectra in Fig. 4.2B, a. Note that a broad feature in the 6000–5750 cm⁻¹ region can be divided into many bands which show clear concentration-dependent variations. Derivative methods will be explained in more detail in Chap. 4.3.2.

Fig. 4.2

A NIR spectra in the 7500–5500 cm⁻¹ region of water-methanol mixtures with a methanol concentration of 0–100 wt% at increments of 5 wt% at 25 °C. B a An enlargement of the 6000–5700 cm⁻¹ region of the NIR spectra shown in A and B, b the second derivative of the spectra in a. Reproduced from Ref. [10] with permission

• (3) Calculation of difference spectra

Calculation of difference spectra is also useful to unravel overlapping bands and to find out a weak feature hidden by a strong band [1, 2, 5, 7]. The difference spectrum between a spectrum of sample a and that of sample b can be calculated by subtracting the spectrum of sample a from that of sample b. The calculation of difference spectra is effective to analyze perturbation-dependent NIR spectra such as temperature-dependent, concentration-dependent, and pH-dependent spectra. To calculate accurate difference spectra one must obtain spectra with very high wavelength accuracy. Let us show very simple but important example of the calculation of difference spectra. Figure 4.3a displays NIR spectra of water collected over a temperature range of 5–85 °C [11]. From this figure, it is clear that the intensity at 7050 cm⁻¹ increases while that at 6844 cm⁻¹ decreases but it is not clear whether there is a band shift or not in the 7300–6200 cm⁻¹ region. The calculation of the difference spectra clearly answers this question. Figure 4.3b displays the difference spectra of water obtained by subtracting the spectrum at 5 °C as a reference spectrum from other spectra in Fig. 4.3a [11]. It can be seen from Fig. 4.3b that the broad water feature consists of two bands at 7089 and 6718 cm⁻¹ and that there is no significant band shift.

Fig. 4.3

a NIR spectra of water measured in a temperature range of 5–85 °C. b The difference spectra of water obtained by subtracting the spectrum at 5 °C as a reference spectrum from other spectra in a. Reproduced from Ref. [11] with permission

Generally speaking, difference spectra method is a reliable method but even so care must be taken. Using spectra of a model system, let us explain a problem of difference spectra. Figure 4.4a shows spectral changes of a system consisting of two components having Gaussian type bands with different peak wavenumbers (7300 and 7250 cm⁻¹), different intensities (10→5 and 10→20), and different band widths(59 and 118 cm⁻¹). Figure 4.4b depicts variations of each band and Fig. 4.4c displays the difference spectra calculated using the first spectrum (pink color) as a reference. Note that peak positions (7309, 7239 cm⁻¹; Fig. 4.4c) are shifted in the difference spectra compared with positions in the original spectra. Figure 4.4d exhibits an enlargement of Fig. 4.4c. A ghost peak appears near 7380 cm⁻¹ because the widths of these two bands are significantly different from each other. Therefore, a special care must be taken for feeble peaks.

Fig. 4.4

a Spectral changes of a system consisting of two components having Gaussian type bands with different peak wavenumbers (7300 and 7250 cm⁻¹), different intensities (10→5 and 10→20), and different band widths (59 and 118 cm⁻¹). b Variations of each band. c The difference spectra calculated using the first spectrum (pink color) as a reference. d An enlargement of c. e The second derivative of the spectra shown in a. Prepared by Y. Morisawa

• (4) Spectra-structure correlations

The NIR spectrum of a compound can be compared with those of similar compounds to make band assignments. For instance, NIR measurements of a series of alcohols or that of fatty acids allow one to make assignments of bands due to OH, CH₂, and CH₃ groups. Figure 4.5 compares NIR spectra in the 7500–4000 cm⁻¹ region of saturated (stearic acid, arachidic acid, and palmitic acid) and unsaturated (oleic acid, linolenic acid, and linoleic acid) long-chain fatty acids (0.05 M in CCl₄) [12]. By comparison, one can easily discriminate saturated and unsaturated long-chain fatty acids; particularly see the 5800–5670 cm⁻¹ region. A common band near 6908 cm⁻¹ can be assigned to the first overtone of OH stretching mode. The unsaturated fatty acids yield characteristic bands near 4663 and 4590 cm⁻¹. Grabska et al. have assigned the 4663 cm⁻¹ band to the combination of C=C stretching and CH stretching modes by quantum chemical calculation [12].

Fig. 4.5

NIR spectra in the 7500–4000 cm⁻¹ region of saturated (stearic acid, arachidic acid, and palmitic acid) and unsaturated (oleic acid, linolenic acid, and linoleic acid) long-chain fatty acids (0.05 M in CCl₄). Reproduced from Ref. [12] with permission

• (5) Spectral analysis based on perturbation

NIR measurements of perturbation-dependent spectral variations, such as temperature-dependent, concentration-dependent, and pH-dependent spectra variations, often give valuable information about the band assignments [1, 2, 7]. As a good example, temperature-dependent NIR spectra changes of octanoic acid in the pure liquid over a temperature range of 15–90 °C are shown in Fig. 4.6 [13]. Octanoic acid in the pure liquid forms a cyclic dimer with hydrogen bonds at room temperature but with the temperature increase the dimer dissociates gradually and a free OH group emerges. It is noted that the intensity of a band at 6920 cm⁻¹ increases as a function of temperature, while those of other bands in the 7300–7000 cm⁻¹ region are almost temperature independent. Thus, the band at 6920 cm⁻¹ is assigned to the first overtone of the OH stretching mode of the monomeric species of octanoic acid, and the rest is due to combinations of CH vibrations [13].

Fig. 4.6

Temperature-dependent NIR spectra changes of liquid octanoic acid collected in a temperature range of 15–90 °C. Reproduced from Ref. [13] with permission

The spectra of Figs. 4.2 and 4.3 are also good examples of perturbation-dependent spectra changes.

• (6) Comparison of an NIR spectrum with the corresponding IR spectrum

The significance of comparison of an NIR spectrum with the corresponding IR spectrum was pointed out in Chap. 2.1.6. Here, let us compare NIR spectra of poly(3-hydroxybutyrate) (PHB) with the corresponding IR spectra again but this time in a different region. Figure 4.7a shows time-dependent changes in the NIR spectra in the 5200–5060 cm⁻¹ region of a PHB film during the melt-crystallization process at 125 °C [14]. The corresponding IR spectra in the region of 1780–1670 cm⁻¹ are depicted in Fig. 4.7b [14]. Of note is that a band at 5127 cm⁻¹ in the NIR spectra gradually increases during the crystallization process, while a broad feature centered at 5160 cm⁻¹ decreases with time, suggesting that the former band is assigned to the crystalline band and the latter band to the amorphous one. In the corresponding IR spectra bands at 1722 and 1743 cm⁻¹ are ascribed to the crystalline and amorphous C=O bands, respectively. Of note is that the NIR spectra and the IR spectra show very clear correspondence. The comparison of the NIR spectra with the IR spectra led Hu et al. [14] to ascribe the band at 5127 cm⁻¹ to the second overtone of the C=O stretching mode of the C–H…O=C hydrogen bonding in the crystalline state and the broad feature near 5160 cm⁻¹ to the corresponding band due to the amorphous state.

Fig. 4.7

a Time-dependent NIR spectra changes in the 5200–5060 cm⁻¹ region of a PHB film during the melt-crystallization process at 125 °C. b The corresponding IR spectra variations in the 1780–1670 cm⁻¹ region. Reproduced from Ref. [14] with permission

Figure 4.8a, b gives rise to another example of comparison between a NIR spectrum and an IR spectrum. They are the IR spectrum in the 3800–3000 cm⁻¹ region and the NIR spectrum in the 7600–6000 cm⁻¹ region of diluted methanol in CCl₄. A peak at 3630 cm⁻¹ and that at 7090 cm⁻¹ are due to a fundamental and a first overtone of a stretching mode of free OH group of methanol while broad features in the regions of 3500–3200 and 6900–6100 cm⁻¹ arise from a fundamental and a first overtone of a stretching mode of hydrogen-bonded OH groups of methanol dimer, trimer, and oligomers. Thus, there is the correspondence between the NIR spectrum and the IR spectrum, but the relative intensity of bands between the free OH bands and the hydrogen-bonded OH bands is largely changed between them. Accordingly, special care must be taken in comparison with the relative intensity between a NIR spectrum and an IR spectrum.

Fig. 4.8

a An IR spectrum in the 3800–3000 cm⁻¹ region and b a NIR spectrum of the 7600–6000 cm⁻¹ region of diluted methanol in CCl₄

• (7) Spectral interpretation by polarization measurement

Polarization measurement, which is popular in IR and Raman spectroscopy, is not often used in NIR spectroscopy, but it is useful for the determination of the molecular-orientation of solid-oriented compounds such as uniaxially stretched polymers. For more details, see Ref. [14].

• (8) Isotope exchange experiments

The use of an isotope shift, which is associated with isotopic substitution, is a traditional method for band assignment in IR and Raman spectroscopy. It is also useful for NIR spectra analysis, particularly, a deuteration shift. The isotope shift provides convinced assignment of bands in a number of cases. Force constants can be assumed not to vary due to isotopic substitution, and hence, isotope shifts involve only with mass effects. Taking a diatomic molecule as an example, one can calculate the magnitude of an isotope shift. Frequency of a stretching vibration of the diatomic molecule is given by Eq. (4.1)

$$ \nu = \frac{1}{2\pi }\sqrt {\frac{k}{\mu }} $$

(4.1)

if $\nu^{\prime }$ is the frequency for replacing an atom having a mass m₁ with an isotope having a mass m₁’, the following relation holds:

$$ \frac{\nu }{{\nu^{{\mathbf{\prime }}} }} = \sqrt {\frac{{\mu^{{\mathbf{\prime }}} }}{\mu }} $$

(4.2)

where μ′ = m₁′ m₂/ (m₁′ +m₂). As Eq. (4.2) reveals, the larger the difference between m₁ and m₁′, the larger the isotope shift is. Since $\nu$/ $\nu^{{\mathbf{\prime }}}$ = 1.36 if H is replaced with D, a C–H stretching vibration of saturated hydrocarbon, which is located in the vicinity of 2900 cm⁻¹, shifts close to 2100 cm⁻¹. Figure 4.9 shows calculated three vibrational modes (Amide I’, II’, and III’) of deuterated N-methylacetamide and the corresponding modes of the nondeuterium-substituted one (Amide I, II, and III modes) are displayed in Fig. 2.11 [16]. Band shifts induced by the deuterium substitution are rather large for the Amide II and III modes since NH bending vibrations contribute to these two modes, but the Amide I mode, being principally a C=O stretching vibration, yields a very small isotope shift [15]. It is noted with respect to isotope shifts of polyatomic molecules that vibrational modes vary more or less with isotopic substitution, which can be evidently understood from the comparison of Fig. 4.9 and Fig. 2.11. In studies of IR, Raman, and NIR spectra, [15] N-substitution, [13] C-substitution, and the like are often used in addition to deuterium substitution. Although an isotope shift is small when such a heavy atom is replaced, a variation in a vibrational mode associated with the isotopic substitution is also small.

Fig. 4.9

Amide I’, Amide II’, and Amide III’ modes of deuterated N-methyacetamide. Reproduced from Ref. [16] with permission

3 Pretreatment Methods in NIR Spectroscopy

NIR spectra often encounter the problems of unwanted spectral variations and baseline shifts [1,2,3]. They come from the following sources.

• 1. Various kinds of noises such as those from a detector, an amplifier, and an AD converter.

• 2. Light scattering from cloudy liquids or solid samples.

• 3. Changes in temperature, density, and particle size of samples.

• 4. Poor reproducibility of NIR spectra caused by, for example, path length variations.

• 5. The use of optical fiber cable may cause baseline shifts.

The above interferences may become obstacles for conventional spectral analysis and 2D-COS. More importantly, they may easily violate the assumptions on which chemometrics equations are based [3]. For instance, the simple linear relationship stated by Beer’s law does not hold any more, and the additivity of individual spectral responses is not guaranteed. Accordingly, data pretreatment is often necessary [1,2,3]. (Chap. 7) Whenever one attempts to improve SN ratio or to correct baseline fluctuations, one should explore the cause of poor SN ratio and that of baseline changes. Otherwise, one cannot find proper pretreatment methods. One interesting example of studies of baseline changes was reported by Geladi et al. [17] They modeled the reflectance spectra of milk by optical effects and chemical light absorption effects. The former induces variations in the direction of the light, and the latter is concerned with light absorption. In some cases, the former brings about more prominent variations to spectra than the latter. The response of the spectral data to the physical effects is significant baseline variations. On the basis of this study, Geladi et al. [17] proposed multiplicative scattering collection (MSC) as a preprocessing tool to correct the light scattering problems in the NIR spectra.

This section explains four kinds of data pretreatment methods, noise reduction methods, baseline correction methods, resolution enhancement methods, and centering and normalization methods [1,2,3].

3.1 Noise Reduction Methods

In NIR spectroscopy, several kinds of noise are caused by a variety of interfering physical and/or chemical process [1,2,3]. The most general noise is high-frequency noise associated with the instrument’s detector and electronic circuits. There are other forms of noise as well; for example, low-frequency noise and localized noise. Low-frequency noise is induced, for instance, by instrument drift during the scanning measurements. The reduction of the low-frequency noise may be more difficult because it often resembles the real information in the data.

Most standard method to improve SN ratio in spectra is accumulation-average processing that requires to increase the accumulation number and calculate an average. This reduces the effects of high-frequency noise significantly, but technically it is not a “pretreatment” but a normal, integrated part of collecting spectra. If the noise reduction by the accumulation average is still insufficient, one can employ smoothing to remove high-frequency noise. The most commonly used smoothing methods are moving-average method and Savitzky–Golay method [1, 2].

The moving-average method is the simplest type of smoothing [1,2,3]. In this method, the reading A_i’ (A is, for instance, absorbance) at each variable i = 1, 2, ---k is replaced by a weighted average of itself and its nearest neighbors. From i−n to i+n:

$$ A_{i} = \sum\limits_{k = - n}^{n} {w_{k} } A_{i + k} $$

(4.3)

w_k, defining the smoothing, is called the convolution weights.

The Savitzky–Golay method originated from the idea that in the vicinity of a measurement point a spectrum can be fitted by low-degree polynomials [18]. Practically, w_k is determined by fitting the spectrum with low-degree polynomials using least squares regression. Savitzky and Golay calculated w_k for the different orders of polynomials and N (N = 2n+1) [18]. One can find these calculated convolution weights in a numeral table. For example, when N is equal to 5, smoothed values can be obtained by substituting w_k = −3/35, 12/35, 17/35, 12/35, −3/35 (k = −2, −1, 0, 1, 2) into Eq. (4.3). It is noted that if one tries to increase the effect of smoothing by increasing the number of the point of w_k, a band shape would be distorted. This distortion may lead to the decrease in spectral resolution and band intensity.

There are other methods for the noise reduction such as wavelets, eigenvector reconstruction, and artificial neural networks (ANN) [2, 3, 19, 20].

3.2 Baseline Correction Methods

As described above in NIR spectra baselines vary for various reasons [1,2,3]. An observed NIR spectrum, A(λ), can be represented as follows;

$$ A(\lambda { )} = \, \alpha A_{ 0} { (}\lambda { )} + \beta + {{e (}}\lambda ) $$

(4.4)

Here, A₀ (λ), α, β, and e(λ) are a real spectrum, a multiplicative scatter factor (amplification factor), an additive scatter factor (offset deviation), and noise, respectively. There are several methods to eliminate or reduce the effects of α and β. We explain three of them.

Derivative methods

Derivative methods are utilized in NIR spectra for both resolution enhancement and baseline correction [1,2,3]. (Chap. 7) A derivative spectrum is an expression of derivative values, dⁿA/dλⁿ (n = 1, 2, …), of a spectrum A (λ) as a function of λ. The second derivative, d²A/dλ², is most often encountered. The superimposed peaks in an original spectrum turn out as clearly separated downward peaks in a second-derivative spectrum. Another important property of second-derivative method is the removal of the additive and multiplicative baseline changes in an original spectrum. Figure 4.10a displays NIR spectra of 16 kinds of linear low-density polyethylene (LLDPE) and one kind of high-density polyethylene (HDPE), and Fig. 4.10b shows the second derivative obtained with the Savitzky–Golay method of the spectra as shown in Fig. 4.10a [21]. It can be seen from Fig. 4.10b that the second derivative is powerful in removing additive and multiplicative baseline variations of the spectra, and at the same time, it enables to detect a number of bands clearly. A drawback in the derivative methods is that the SN ratio deteriorates every time a spectrum is differentiated.

Fig. 4.10

a NIR spectra of 16 kinds of linear low-density polyethylene (LLDPE) and one kind of high-density polyethylene (HDPE). b The second derivative was calculated with the Savitzky–Golay method of the spectra in a. c NIR spectra shown in a after the MSC treatment. Reproduced from Ref. [21] with permission

Let us explain derivative methods using equations. If the band shape is a Gaussian shape as below;

$$ A\left( x \right) = \alpha { \exp }\left\{ { - \left( {\frac{{x - x_{0} }}{w}} \right)^{2} } \right\}, $$

(4.5)

Analytical solutions of the first and second derivatives of the band are

$$ \frac{{{\text{d}}A}}{{{\text{d}}x}} = - 2\frac{{x - x_{0} }}{{w^{2} }}A = - 2\frac{{\alpha \left( {x - x_{0} } \right)}}{{w^{2} }}{ \exp }\left\{ { - \left( {\frac{{x - x_{0} }}{w}} \right)^{2} } \right\} $$

(4.6)

$$ \frac{{{\text{d}}^{2} A}}{{{\text{d}}x^{2} }} = - 2\frac{A}{{w^{2} }}\left( {1 - 2\frac{{\left( {x - x_{0} } \right)^{2} }}{{w^{2} }}} \right) = - 2\frac{\alpha }{{w^{2} }}\left( {1 - 2\frac{{\left( {x - x_{0} } \right)^{2} }}{{w^{2} }}} \right){ \exp }\left\{ { - \left( {\frac{{x - x_{0} }}{w}} \right)^{2} } \right\} $$

(4.7)

respectively, where $\alpha$ is a peak height of the band, and w is proportional to a full width at the half maximum (FWHM) of the band (ω = FWHM/{Ln (4)}^0.5). As shown in Fig. 4.11, at the peak position (x = x₀), the first derivative coefficient is 0, and the second-derivative coefficient is −2$\alpha$/ω². The positions and intensities of the maximum and minimum of the first derivative spectra are $x_{0} \pm \omega \sqrt 2$ and $\mp \sqrt 2 \alpha {\omega}e^{0.5}$, respectively. As can be seen in these formulas, the maximum and minimum of the first- and second-derivative coefficients are proportional to an area of the band, if the width of the band is not changed.

Fig. 4.11

Original spectrum, its first and second derivatives. Prepared by Y. Morisawa

Since, in general, spectrum data take discrete values, and the calculation of derivatives with various orders is performed by algebraic differences between data taken at closely spaced wavelengths. Transformation to first- and second-derivatives is then

$$ {\text{d}}A_{i} = A_{i + k} - A_{i - k} $$

$$ \begin{aligned} {\text{d}}^{2} A_{i} & = {\text{d}}\left( {A_{i + k} - A_{i - k} } \right) \\ \quad \quad & = A_{i + 2k} - 2A_{i} + A_{i - 2k} \\ \end{aligned} $$

(4.8)

Multiplicative scatter correction (MSC)

MSC is an effective method for correcting vertical variations of the baseline (additive baseline variation) and inclination of the baseline (multiplicative baseline variation) [17]. The idea of MSC originates from the fact that light scattering has the wavelength dependence different from that of chemically based light absorbance. Thus, we can use data from a number of wavelengths to distinguish between light absorption and light scattering.

MSC corrects spectra according to a simple linear univariate fit to a standard spectrum. α and β are estimated by least squares regression using the standard spectrum. As the standard spectrum, a spectrum of a particular sample or an average spectrum is used. Figure 4.10c shows the NIR spectra as shown in Fig. 4.10a after the MSC treatment [21]. The spectra demonstrate the potential of MSC in correcting offset and amplification in the NIR spectra. Generally, MSC improves essentially the linearity in NIR spectra. While it is generally an very useful technique, care must be exercised since the use of MSC may generate unwanted artifacts.

Standard normal variate (SNV)

Standard normal variate (SNV) is also a powerful method for correcting vertical baseline drift of a set of spectra [22]. For each spectrum $A\left( \lambda \right)$, SNV is calculated as $A_{\text{SNV}} \left( \lambda \right) = \left( {A\left( \lambda \right) - \dot{A}} \right)/\sigma$, where $\dot{A}$ and $\sigma$ are mean and standard deviation of the intensities in the spectrum, respectively. Therefore, mean and standard deviation of the intensities for each spectrum after SNV are standardized as 0 and 1, respectively.

3.3 Resolution Enhancement Methods

Resolution enhancement methods are very important to unravel overlapping bands and elucidating the existence of obscured bands [2, 3]. In NIR spectroscopy derivative methods, difference spectra, mean centering, and Fourier self-deconvolution are employed as resolution enhancement methods. Note that PCA loadings plots are often effective for resolution enhancement [21]. 2D correlation spectroscopy can also make resolution enhancement, but since it is not a pretreatment method, it will be outlined in Chap. 6. Mean centering will be discussed in centering and normalization section.

Derivative methods

We already explained the usefulness of derivative method in resolution enhancement as shown in Fig. 4.10a, b. Here, a problem in the derivative methods which we often encounter is pointed out. Figure 4.4e shows the second derivative of the spectra as shown in Fig. 4.4a. It can be seen from Fig. 4.4e that the 7250 cm⁻¹ peak is much weaker than the 7300 cm⁻¹ peak. In the second-derivative spectra, a broad band is often underestimated, and thus, care must be taken for the second derivative of a broad band.

3.4 Centering and Normalization Methods

Centering and normalization are often effective in chemometrics analysis of NIR data [1,2,3]. (Chap. 7) Mean centering is simply an adjustment to a data set to reposition the centroid of the data to the origin of the coordinate system [2, 3]. Normalization is an adjustment to a data set that equalizes the magnitude of each spectrum. [2, 3]

Centering methods

Mean centering is a method where from every element of the jth spectrum (row) the column mean is subtracted:

$$ \varvec{X}_{{jcent}} = {\text{ }}\varvec{X}_{j} - \left( {\frac{1}{n}\sum\limits_{{j = 1}}^{n} {\varvec{X}_{{ij}} } } \right) $$

(4.9)

X_j and X_ij are an element of the jth spectrum and that of a data matrix X, respectively. After this step, all means are zero and variances are spread around zero. Each mean centering spectrum can be regarded as a difference spectrum between the individual spectrum and an averaged spectrum. Mean centering is often powerful in resolution enhancement. Figure 4.12a displays NIR spectra in the region of 6000–5500 cm⁻¹ of nylon 12 collected in a temperature range from 30 to 150 °C, and Fig. 4.12b exhibits their mean-centered spectra [23]. The mean-centered spectra show that the intensity of a band at 5770 cm⁻¹ arising from the first overtone of CH₂ stretching mode varies markedly with temperature. Mean centering is used also as a pretreatment for constructing 2D correlation spectra (Chap. 6).

Fig. 4.12

a NIR spectra in the 6000–5500 cm⁻¹ region of nylon 12 measured over a temperature range from 30 to 150 °C. b Mean-centered spectra of the spectra in a. Reproduced from Ref. [23] with permission

Normalization

Two popular normalization procedures have been known in common practice [2, 3]. Most normalization methods employ vectors normalized to constant Euclidean norm. That is,

$$ {\text{x}}_{{{{j}},{\text{ norm}}}} = {\text{ x}}_{j} / \, \left| {\left| {\mathbf{x}} \right|} \right| $$

(4.10)

where ||x|| is the Euclidean norm of the spectral vector x. This normalization transforms the spectral points on a unit hypersphere, and all data are approximately in the same scaling. This normalization has a good property that the similarity between two spectral vectors may be estimated by the scalar product of these two vectors. However, the normalization induces the geometric configuration of the data points, either the clustering structure or the spreading directions, substantially different from the original one, which may result in a misleading in the understanding of the data in exploratory data analysis. In addition, the variation in spreading directions has a significant effect on principal component analysis (PCA)-related analysis. Therefore, one must be careful enough in using normalization in situations where exploratory data analysis and PCA-related procedures, such as PCA, partial least squares (PLS), and so on, are concerned.

Another normalization procedure is so-called mean normalization, where all points of the jth spectrum are divided by its mean value

$$ \varvec{X}_{{j{\text{norm}}}} = \varvec{X}_{j} /\left( {\frac{1}{m}\sum\limits_{i = 1}^{m} {\varvec{X}_{ij} } } \right) $$

(4.11)

where m is a total number of spectral points. After mean normalization, all the spectra have the same area. Essentially, mean normalization is equivalent to normalize the spectral vectors to constant 1-norm, that is, the sum of spectral values (always positive) equals to a constant. This means that the geometry of mean normalization is to transform the spectral points to be contained in a convex set, and the dimensionality of the spectral space is thus decreased by 1. This transformation is very useful in self-modeling curve resolution (SMCR).