Introduction

Less than 50% of the 13 million tons of synthetic N fertilizer that is applied to U.S. agricultural soils every year is actually used by the crops to which it was applied (Environmental Protection Agency 2011), the rest is either leached from the soil as NO3−, contaminating surface and ground water, or volatilized to the atmosphere as NH3, N2, or N2O. This inefficient use of N fertilizer by crops is both a major economic loss for farmers and the cause of serious environmental problems. Precision management of N fertilizer offers a unique opportunity to simultaneously reduce the cost of agricultural inputs for farmers while helping to mitigate the negative environmental footprint of modern agriculture and thereby increase both the economic and environmental sustainability of production agriculture (Fahsi et al. 1998; Ma and Biswas 2015).

Precision farming is the integration of information technological tools, such as global navigation satellite systems (GNSS), chlorophyll meters, remote sensing of soils and crop canopies, soil electrical conductivity sensors and yield monitors with modern agricultural equipment such as fertilizer applicators, planters and sprayers to facilitate intra-field variable management of crops and soils. Yield monitors, for example, enable farmers to identify high and low productive areas of their fields, which can aid in spatially variable management. The ultimate goal of precision agriculture is to enable farmers to identify, measure and differentially manage soils within-fields in a manner that increases farm profitability and decreases environmental impact associated with intensive agriculture. Despite recent advances, farmers still lack rapid and economical means of developing nutrient management plans that accurately account for within-field variability of soil nutrient levels. Managing soil N fertility is particularly problematic because weather, soil type and management history all have a big influence on the soil N cycle. The complexity of the N cycle means that the amount of plant available N in soil varies substantially both with time during a growing season and spatially across an agricultural field.

The late spring nitrate test (LSNT), also known as the Pre-Sidedress Nitrate Test, is an established tool for N fertility management in corn production (Binford et al. 1992; Blackmer et al. 1989; Magdoff 1991). The LSNT protocol specifies that representative soil samples be collected from a field in the late spring when the corn is approximately 0.3 m tall (V6 growth stage; Pioneer 2018) and then analyzed for NO3− concentration. If the soil NO3−–N concentration is above a critical level (25 mg kg−1) then no additional N fertilizer is recommended for the cropping season; however if the concentration is below the critical level then sidredress N fertilizer is recommended. The LSNT can be used for precision N fertilizer management if enough GNSS registered soil samples are collected to represent the spatial variability of soil NO3− concentrations within the field. Although shown to be effective for improving N use efficiency (NUE) and reducing NO3− leaching losses (Jaynes et al. 2004), the LSNT is not widely used by farmers because; (1) labor for soil sampling and analysis is expensive, (2) its low spatial resolution (typically 1 composite soil sample is collected and analyzed per ha) misses substantial in-field variability, and (3) the time delay between soil sampling and the availability of a N fertilizer prescription increases risk for farmers. In recent years, development of field mobile soil analysis systems (such as SoilScan 360®, 360 Yield Center, Morton, IL USA) have allowed in-field determination of soil NO3− levels eliminating the time delay between sampling and availability of the results, but not the labor and time required for soil sampling. The bottom line for most farmers is that the LSNT is too impractical and too risky for large-scale use on production agriculture fields.

A second approach to precision N management is measuring crop canopy reflectance during the growing season and applying midseason N fertilizer only where the crop is pale green or yellow (Rorie et al. 2011; Scharf and Lory 2002; Scharf et al. 2011). Nitrogen deficient plants often turn pale green or yellow and hence measuring canopy reflectance is effective for identifying regions of a field where soils have sub-optimal levels of plant available N. However, other factors, such as deficiencies of other nutrients, water stress and disease also cause plants to have similar symptoms. Nitrogen deficiency symptoms often are not apparent in a growing crop until late in the growing season and late-season sidedress N fertilization may not be effective in preventing substantial yield losses (Binder et al. 2000). Furthermore, canopy sensors are generally not effective until after canopy closure, hence specialized high clearance equipment is generally required for canopy sensors and to apply supplemental sidedress N fertilizer. As a result of these problems, adoption of canopy sensors by farmers for precision N management is limited.

The crop-soil-climate modeling approach to precision N fertilizer management has evolved over the past two decades and is now commercially available to farmers as agricultural industry sponsored decision support services (Bendre et al. 2015). This approach, often called big data, integrates information related to agronomic management, soil analysis, climate, soil and yield maps etc., over many farms. The data is used to model the soil-crop system and predict soil N availability during the course of a growing season. These big data systems are being used by some farmers to provide within-field soil specific N fertilizer recommendations. Examples of this approach include Encirca®, developed by scientists at DuPont-Pioneer Co; Adopt-N®, developed by Agronomic Technology Corporation (Melkonian et al. 2008), and FieldView® developed by Monsanto—Climate Corp. For example, the Adapt-N® tool models relevant soil processes and crop growth and incorporates real-time field-specific weather data to generate field-specific N fertilizer recommendations (Sela et al. 2016). Though potentially useful, farmers have been slow to adopt big data tools because of concerns about data ownership, cost and the ability of such tools to accurately predict spatial variability of soil N status within their fields.

The growing interest in precision N fertilizer management has prompted the development of proximal soil sensing techniques to directly measure soil N status in real-time. Indeed, numerous proximal soil NO3− sensor systems have been developed, yet none have been widely adopted for precision N fertilizer management in production agriculture, due to various technology-specific limitations (Adamchuk et al. 2004; Sethuramasamyraja et al. 2008; Sinfield et al. 2010). The most common approach is the use of ion selective electrodes (ISE) for real-time soil NO3− sensing (Adamchuk et al. 2003; Adsett et al. 1999; Kim et al. 2007; Sibley et al. 2009). For example, Adsett et al. (1999) built and tested an automated system for on-the-go measurement of soil NO3−, which essentially brings a wet chemistry lab into the field. With this system, a soil sample is automatically collected, NO3− is extracted from the soil using deionized water (DI), and the extract is analyzed for NO3− with an ISE. While these and other similar studies have shown that ISEs can be used for proximal soil NO3− sensing, ISEs have had limited commercial success due to their inherent complexity and relatively slow response time. The typical ISE response time is 10 s or more and accuracy is affected by many factors, which present difficulties in implementation of ISE technology for real-time on-the-go proximal soil NO3− sensing (Lobsey et al. 2010; Rossel et al. 2011).

The long-term goal of the project is to develop a robust and rapid spectroscopic soil NO3− sensor system that can be attached to a fertilizer applicator and used to measure soil NO3− status and thereby modulate N fertilizer application rates in real-time. Such a sensor system would facilitate precision sidedress N fertilizer applications based on established LSNT protocols by replacing the need for hand sampling and delayed laboratory analysis of soil NO3− status with on-the-go real-time sensing. Here the use of a handheld diamond attenuated total internal reflectance Fourier transform mid-infrared (D-ATR-FTIR) spectrometer is evaluated as a sensor for rapidly and directly measuring soil NO3−–N concentrations in field moist soils. Attenuated total internal reflection (ATR) is a sampling technique used with infrared spectroscopy, which enables samples to be analyzed with little or no preparation in the solid or liquid state. ATR measurements are typically carried out through an attachment (ATR cell) to the infrared spectrometer that directs the infrared light beam toward a medium of high refractive index (often a crystal of zinc selenide, germanium or diamond). One of the crystal facets of the ATR crystal is placed in direct contact with a sample. Any light that impinges on the contact surface between the crystal and the sample from within the crystal at an angle greater than the “critical angle” will be reflected back into the crystal, if the sample material in contact with the ATR crystal has a lower refractive index. Under these conditions, a localized evanescent wave is generated in the sample material typically within the top few µm. The sample material interacts with and attenuates this interrogating evanescent wave differentially throughout the infrared frequency regions thereby generating an absorbance spectrum of the sample (Griffiths and De Haseth 2007).

ATR-FTIR spectroscopy using a ZnSe ATR crystal has previously been used to analyze NO3− concentrations in saturated soil pastes with standard errors for non-calcareous soils of 4 mg[N] kg−1 [dry soil] (Linker et al. 2004, 2005), which is more than adequate enough for the proposed application. Here a diamond ATR cell was used together with one of the new generation of small robust handheld battery-powered FTIR spectrometers; a combination which makes dynamic, in-field, on-the-go measurement of soil NO3− concentrations possible for the first time (Laird et al. 2016). The specific objective of this study was to determine whether or not a handheld D-ATR-FTIR system is capable of measuring soil NO3− concentrations with enough accuracy to be used as a sensor for modulating mineral N fertilizer application rates in real-time as fertilizer is being applied to agricultural fields.

Materials and methods

Field samples

The performance of the handheld D-ATR-FTIR spectrometer was tested using samples collected from four agricultural fields in central Iowa within the predominant Webster–Clarion–Nicollet soil association. The tests were performed to evaluate the accuracy of D-ATR-FTIR spectrometer in predicting NO3−–N concentrations in soils with varying moisture, organic matter, texture and other chemical and physical properties. Soils ranged in texture from light silty clay loam to heavy clay loam and soil pH ranged from 5.1 to 7.4 (average = 6.04). Two fields were in corn-soybean rotations (fields 1 and 3), while the other two fields were in continuous corn. For field 1, corn was following soybeans and the field received 112 kg N ha−1 at planting with an additional 56 kg N ha−1 at sidedress. For field 2, corn was following corn and the field received 134 kg N ha−1 at planting with an additional 87 kg N ha−1 at sidedress. For field 3, corn was following soybeans and the field received 78 kg N ha−1 at planting with an additional 90 kg N ha−1 at sidedress. For field 4, corn was following corn and the field received 115 kg N ha−1 at planting with an additional 112 kg N ha−1 at sidedress.

About 30 GPS registered soil samples per field (124 total samples) were collected in a grid pattern to a depth of ~ 0.10 m during late May-early June 2016 when corn plants were about 0.15 m tall. Soil sampling was performed after preplant application of N fertilizer (late April) but before sidedress application of N fertilizer and was based on protocols for the LSNT. Samples were kept in plastic bags to prevent moisture loss due to evaporation; and within 24 h of collection samples were brought to the laboratory, scanned with the D-ATR-FTIR spectrometer, extracted with 2 M KCl and the extracts were analyzed for NO3− using a standard wet-chemistry colorimetric procedure (Schnetger and Lehners 2014). Soil moisture was also measured gravimetrically for all samples.

Soil moisture effects on spectra

The effect of soil moisture content on the D-ATR-FTIR spectra was determined by using soil samples adjusted to different moisture levels. To do so, air dry soil samples with known concentrations of NO3− were divided into three sub-samples, 20 g each. Each subsample received different quantities of DI water to achieve gravimetric moisture contents of 15, 20 and 25%. The moistened soils were covered with plastic film to prevent water loss due to evaporation and allowed to equilibrate for about an hour. The soils were then scanned with the D-ATR-FTIR spectrometer.

Laboratory samples

Bulk samples (0–0.20 m) of five soils varying in texture and soil organic matter content were collected in October 2015: The soils are Hanlon fine sandy loam (Coarse-loamy, mixed, superactive, mesic Cumulic Hapludolls); Webster silty clay loam (Fine-loamy, mixed, superactive, mesic Typic Endoaquolls); Okoboji mucky silty clay loam (Fine, smectitic, mesic Cumulic Vertic Endoaquolls); Nicollet clay loam (Fine-loamy, mixed, superactive, mesic Aquic Hapludolls); and Exira silty clay loam (Fine-silty, mixed, superactive, mesic Typic Hapludolls).

The bulk soils were air-dried to 5–9% moisture, ground, sieved to pass a 2 mm screen, and divided into 100 g (oven-dry basis) samples. Each 100 g sample was amended with 0, 10, 20, 30, 40, 50, 70, 90 and 110 mg kg−1 of NO3−–N by adding various amounts of a KNO3 solution. The difference in KNO3 solution quantities were adjusted by adding DI water to bring each soil sample to specific soil moisture content. Because of differences in textural classes of selected soils, samples from each soil type were equilibrated at different moisture contents; for Nicolet and Exira, moisture content was adjusted to 30%; Hanlon to 17%; Okoboji to 34%, and Webster to 26%. Control samples (added NO3–N = 0 mg kg−1) were treated only with DI water. Soil samples were thoroughly mixed and left to equilibrate in a cold room at 4 °C for 20-24 h. Each NO3− concentration x soil type treatment was replicated three times for total of 135 samples. After equilibration, the soil samples were placed in small containers (Petri dish) and 5–7 infrared spectra were collected per soil sample. Soil samples were then extracted with 2 M KCl and analyzed for NO3− concentrations using a standard colorimetric microplate technique (Schnetger and Lehners 2014).

Spectral measurement

An Agilent 4100 ExoScan (Agilent Technologies, Santa Clara, CA, USA) handheld spectrometer with a diamond ATR crystal was used for all spectral measurements. For each sample, 5–7 individual scans were collected each at a different location within the sample, covering the 803–2499 cm−1 spectral range with a resolution of 16 cm−1. At each location, the D-ATR cell of the spectrometer was directly pressed onto the soil surface and spectrum collected in about 1 s. Good contact with the soil is essential for obtaining high quality spectra as the presence of air bubbles or gaps at the sample contact surface degrades spectra quality. Two separate datasets were collected: the field dataset contained 767 individual spectra; and the laboratory dataset included 883 individual spectra. Both, field and laboratory spectra datasets were pre-processed to identify outliers using the sum of squares for differences between each spectrum and the overall mean spectrum. This process identified 48 individual spectra out of 767 (field dataset) and 39 out of 883 (laboratory dataset) that were discarded. With the remaining 720 and 844 spectra, an average spectrum for each soil sample (typically 6 spectra per sample) was calculated. This process led to 124 (field dataset) and 135 (laboratory dataset) average raw spectra—one spectrum for each soil sample.

Chemometric analysis was performed using The Unscrambler 10.3 (CAMO Software, Inc., Magnolia, TX, USA) software separately for the field and laboratory datasets using partial least squares (PLS1) regression. The number of factors used in the PLS1 regression models was selected based on the minimum value of the root mean squared error (RMSE) for a leave-one-out cross validation procedure. Field and laboratory datasets were then combined to create a combined dataset that included all soil samples from the four fields and all of the Webster, Nicollet and Okoboji soils (205 total spectra and related NO3−–N concentrations). The Exira and Hanlon soils were excluded from the combined dataset because these soils were not represented in any of the four agricultural fields studied, whereas Webster, Nicollet and Okoboji soils were present in the agricultural fields. The combined dataset was used for a robust “leave one field/soil out” validation procedure. For the validation procedure, PLS1 calibration models were created using all of the data in the combined dataset except for data for one of the fields or soils. The calibration model was then used to predict NO3− concentrations for the field or soil that was left out of the model calibration. This procedure was repeated 7 times so that all fields and soils in the combined dataset were analyzed. Linear regression (R2) and RMSE were used to assess the ‘leave one field/soil out’ validation results. RMSE was calculated as outlined in Borenstein et al. (2006);

$${\text{RMSE }} = \,\sqrt {\frac{{\mathop \sum \nolimits_{i = 1}^{n} (y - y^{\prime})^2}}{N}}$$
(1)

where y and yʹ are the predicted and measured concentrations, and N is the number of spectra.

Geospatial analysis

Maps of measured and D-ATR-FTIR predicted soil NO3−–N concentrations for the 4 fields were prepared using 3D Analyst Spline with Barriers Raster Interpolation tool in ArcGIS 10.4.1 software (Environmental Systems Research Institute, Inc., Redlands, CA, USA). The maps are divided into management zones based on the LSNT protocols, which stipulates that areas with soil NO3−–N < 10 mg kg−1 require on average 174 kg N ha−1, areas with 10 and 15 mg NO3−–N kg−1 require 107 kg N ha−1, and areas with 15 and 25 mg NO3−–N kg−1 require an additional 40 kg of sidedress N fertilizer. Areas with soil NO3−–N concentrations > 25 mg NO3−–N kg−1—have a sufficient supply of plant available N to sustain crop growth through the remainder of the growing season and do not require additional sidedress N fertilizer (Blackmer et al. 1997).

Results

Spectral characteristics

D-ATR-FTIR spectra of a typical Iowa agricultural soil equilibrated at 15, 20 and 25% gravimetric water content are shown in Fig. 1. In these spectra, the presence of liquid water is indicated by the prominent ν2 (O–H bending) band centered at 1644 cm−1 and a much weaker ν2 + liberation band centered at 2130 cm−1 (Verma et al., 2017). Also evident in the spectra are a prominent set of at least 5 overlapping bands between 850 and 1200 cm−1. Specific identification of the absorption bands in this region is outside of the scope of this paper; however these bands are attributed to inorganic mineral phases present in the soils. In general, both symmetric and asymmetric M–O (where M is primarily Si but can also be Al, Fe or Mg) stretching bands for both tectosilicates (quartz and feldspars) and phyllosilicates (smectite, illite and kaolinite) contribute to adsorption in this region. These M–O stretching bands are superimposed on a broad water liberation band, which causes a progressive increase in the background of the spectra below 1000 cm−1. The effect of this water liberation band is evident by comparing the baseline in the 800–1000 cm−1 region for spectra of the soils equilibrated at 20 and 25% moisture with the same region for spectra of the soil equilibrated at 15% moisture (Fig. 1).

Fig. 1
figure 1

Differences in D-ATR-FTIR spectra of the same soil equilibrated with different amounts of water to achieve 15, 20 and 25% gravimetric moisture content. Major bands contributing to the spectra are identified, where M is primarily Si but can also be Al, Fe or Mg

Full size image

Nitrate has a prominent asymmetric N–O stretching band centered at 1400 cm−1. However, because NO3− concentrations are low in the studied soils, the N–O band is only barely visible as a slight positive curvature in the baseline of the spectra between the much larger ν2 (O–H bending) band for water and the M–O stretching bands for the inorganic phases. Critically, however, neither the O–H bending band nor the M–O stretching band overlap with the N–O stretching region. Increasing soil water content causes the background to increase around 1400 cm−1 but does not interfere with the shape of the spectra in this region (Fig. 1).

D-ATR-FTIR spectra of a carbonate free soil and a soil containing free carbonates are shown in Fig. 2. The spectra of the soil that contains free carbonates has a prominent asymmetric C–O stretching band centered at 1440 cm−1 and ranging from 1337 to 1537 cm−1 and a smaller asymmetric C–O deformation band centered at 877 cm−1. Because the large asymmetric C–O stretching band for carbonate interferes with the N–O stretching band for NO3− the use of D-ATR-FTIR for quantifying NO3− in calcareous soils is problematic (Linker et al. 2005).

Fig. 2
figure 2

Typical spectra of the calcareous and non-calcareous soils showing that the carbonate C–O stretching band overlaps the nitrate N–O stretching band (N–O str.). Major bands contributing to the spectra are identified, where M is primarily Si but can also be Al, Fe or Mg

Full size image

Soil water and nitrate determination

The124 field moist soil samples (gravimetric water content ranging from 7 to 38%) collected from four agricultural fields were analyzed to assess the ability of D-ATR-FTIR coupled with PLS1 modeling to predict gravimetric moisture content of soils (Fig. 3). The results, R2  = 0.85 and RMSE = 0.025 g-H2O g-soil−1, demonstrate that D-ATR-FTIR is effective for determining water content of soil. The findings are in agreement with those presented by Borenstein et al. (2006) who found that ATR-FTIR could be used to determine water content of soil pastes.

Fig. 3
figure 3

Measured versus predicted water content (g-water g-soil−1) for 124 field soil samples collected in late May—early June of 2016. The four farms are distinguished by symbols with different colors (Color figure online)

Full size image

The dataset for the 124 soil samples from the four agricultural fields and the dataset for the 135 soil samples from 5 soils amended with different NO3− levels were used to assess the ability of D-ATR-FTIR coupled with PLS1 modeling to predict soil NO3−–N concentrations (Figs. 4, 5). The results for the field samples, R2 = 0.83 and RMSE =  8.3 mg kg−1, and for the 5 soils, R2 = 0.90 and RMSE = 8.6 mg kg−1, independently demonstrate the ability of D-ATR-FTIR coupled with PLS1 modeling to predict soil NO3− concentrations in moist soils. Finally, the combined dataset (205 spectra and soil NO3−–N analyses) was used for a robust ‘leave one field/soil out’ validation test (Fig. 6). Results for the validation test of the field samples, R2 = 0.65 and RMSE = 12.5 mg kg−1, and the 5 soils, R2 = 0.82 and RMSE = 13.3 mg kg−1, provide an independent validation of the ability of D-ATR-FTIR to predict soil NO3− levels across a broad range of soil textures and soil moisture content. Although physically impossible, the PLS1 regression model occasionally predicts negative NO3−–N concentrations, these values should be considered as zero within the margin of error.

Fig. 4
figure 4

Relationship between measured and predicted soil NO3−–N concentrations for 124 samples collected from 4 fields and analyzed by D-ATR-FTIR while the samples were still field moist (soil moisture ranged from 7 to 38%)

Full size image
Fig. 5
figure 5

Relationship between measured and predicted soil NO3−–N concentrations pooled for five soil types. Various amounts of KNO3 were added to these soils to achieve a broad range in soil NO3− concentrations. The five soil types are distinguished by color of the symbols (Color figure online)

Full size image
Fig. 6
figure 6

Validation of D-ATR-FTIR predictions of soil NO3−–N concentrations. Predicted versus measured soil NO3−–N concentrations for the combined field and laboratory dataset showing; a leave one field out validation, and b leave one soil out validation. Different fields and soil types are distinguished by symbol color (Color figure online)

Full size image

The spatial distribution of soil NO3− concentrations in four agricultural fields determined using high density grid sampling and geospatial analysis are shown in Fig. 7. The spatial pattern of LSNT N-management zones based on standard laboratory and D-ATR-FTIR analysis of soil NO3− concentrations show similar spatial patterns.

Fig. 7
figure 7

Maps of four fields showing spatial distribution of soil NO3−–N concentrations based on high density grid sampling; site names with A, standard laboratory method used to measure soil NO3−–N concentrations, sampling points shown on the maps; site names with B, soil NO3−–N concentrations predicted using the D-ATR-FTIR sensor. Maps were generated using a spline with barriers—raster interpolation technique in ArcGIS software. The thin gray lines are soil mapping unit boundaries. Soil NO3−–N categories are very low (< 10 mg kg−1), low (10–15 mg kg−1), medium (15–25 mg kg−1), high (25–50 mg kg−1) and very high (> 50 mg kg−1)

Full size image

Discussion

A key goal of precision agriculture is to apply the right amount of N-fertilizer in the right place and at the right time. While GNSS and variable rate application technology have been available to farmers for over two decades, the lack of a practical and cost effective means for rapidly measuring the spatial variability of plant available N in soils has limited adoption of precision N fertilizer management.

As stated above, the purpose of this study was to determine whether or not a handheld battery powered D-ATR-FTIR spectrometer is accurate enough to be used as a soil NO3− sensor system to facilitate precision N fertilizer management. Linker et al. (2005) previously demonstrated that an ATR-FTIR system was capable of determining soil NO3−–N concentrations in non-calcareous soils with a standard error of 4 mg kg−1. Linker et al. used a laboratory grade FTIR spectrometer (Bruker Vector 22™, Bruker Optics, Inc. Billerica, MA, USA) equipped with a ZnSe-ATR sample cell and a DTGS detector that was capable of achieving 2 cm−1 resolution. Their soil samples were prepared as saturated pastes and spread uniformly across the ZnSe ATR crystal. However, the use of a laboratory grade spectrometer, which cannot tolerate vibration, high humidity or temperature extremes, a relatively soft ZnSe ATR crystal that is easily scratched or abraded by soil minerals, and saturated pastes precludes field deployment of the system used by Linker et al. (2004, 2005). In contrast, the handheld battery powered D-ATR-FTIR spectrometer with 16 cm−1 spectral resolution used here is resistant to vibration, high humidity, and temperature extremes (Agilent 4100 ExoScan FTIR Operation Manual). Furthermore, the sample presentation consisted of pressing field moist soil against the optical surface of the D-ATR cell. Thus the instrument and the analytical conditions employed in this study are amenable to field deployment. Indeed, spectra of fresh soil samples were collected in the field.

Transmission and diffuse reflectance FTIR spectroscopy generally require that soil samples be oven dried and that the optical path be purged with desiccated air or N2 gas, because the presence of any water vapor in the optical path degrades the quality of the spectra. By contrast, the presence of free water in a soil sample is essential to obtain quality spectra of soils using D-ATR-FTIR spectroscopy. When moist soil is pressed onto the optical surface of the D-ATR cell some water is extruded from the soil forming a thin film of liquid water on the optical surface of the diamond. Hence, D-ATR-FTIR spectra of field moist soils consist primarily of absorption bands for liquid water and any mineral particles that are physically touching or within ~ 2 µm (actual penetration depth depends on wavelength, refractive index and incident angle: Khoshhesab, 2012) of the optical surface of the diamond (Figs. 1, 2). D-ATR-FTIR spectra of air-dried soil samples are generally unusable because they are dominated by numerous water vapor absorption bands between 1250 and 2000 cm−1. Indeed, the lack of water vapor absorption bands and the presence of the O–H bending band for liquid water (Figs. 1, 2) is evidence that a film of liquid water formed across the surface of the diamond in this experiment. Nitrate is present in the aqueous phase of soils and electrostatically repelled from the negatively charged surfaces of soil mineral and organic particles. Hence, NO3− is effectively concentrated in the film of liquid water touching the optical surface of the diamond. For example, in a soil that has NO3−–N concentration of 25 mg kg−1 and a water content of 30%, the effective NO3−–N concentration of the aqueous phase is 83 mg l−1, and due to anion exclusion from diffuse double layers adjacent to particle surfaces, the actual concentration of NO3−–N in the water film on the diamond surface is even higher. Thus the sample presentation concentrates NO3− in the analytical zone of the ATR crystal allowing NO3− concentrations to be measured despite the low levels present in most soils.

Three different tests of the viability of using D-ATR-FITR to determine soil NO3−–N concentrations were used. First, the 124 samples collected from four agricultural fields during the late-May to early-June sidedress N fertilizer application window had a wide range in soil moisture (7 to 38%, average 21% and SD 6.6%; Figs. 3, 4). Second, the five soils amended with various amounts of KNO3 and equilibrated in the laboratory had a wide range of soil textures (sandy loam to silty clay loam) and NO3− concentrations (Fig. 5). Third, the combined field and laboratory dataset facilitated a robust “leave-one field/soil out” cross validation test (Fig. 6). All three tests indicated the ability of D-ATR-FTIR to determine soil NO3−–N concentrations with calibration R2 values ranging from 0.83 to 0.90 and validation R2 values ranging from 0.65 to 0.82. The RMSEs for the calibration and validation tests ranged from 8.3 to 8.6 mg kg−1 and from 12.5 to 13.3 mg kg−1, respectively. Collectively the results of these three tests demonstrate that the handheld D-ATR-FTIR spectrometer is capable of determining soil NO3− concentrations across broad ranges of soil moisture and texture.

The distribution of measured versus predicted data points for the four fields and five soils show no evidence of bias in the calibration tests (Figs. 4, 5). This result is anticipated as all of the data were used to make the calibration models. In the validation test, however, some bias is evident for the three soils included in the combined dataset (Fig. 6b). Specifically, most of the data points for the Nicollet soil are below the 1:1 line, while most of the data points for the Webster soil are above the 1:1 line. This result indicates the potential to improve the accuracy of calibration models by using a more diverse calibration dataset together with locally weighted chemometric model, such as that used by Chang et al. (2001).

The LSNT protocol (Blackmer et al. 1997) stipulates five N fertilizer management zones based on soil NO3−–N concentrations; very low (< 10 mg kg−1), low (10–15 mg kg−1), medium (15–25 mg kg−1), high (25–50 mg kg−1) and very high (> 50 mg kg−1). No sidedress N fertilizer is recommended for soils in the high and very high categories; while sidedress N fertilizer is recommended for soils in the very low (174 kg N ha−1), low (107 kg N ha−1) and medium (40 kg N ha−1) categories. The RMSE for NO3−–N determined by D-ATR-FTIR ranged from 8.3 to 13.3 mg kg−1 for the three tests described above. Thus the error in determining soil NO3−–N concentrations is about the same size as the range in soil NO3− concentrations specified in the very low, low and medium management zone categories. To assess the implications of this level of precision, N fertilizer management zones determined by the D-ATR-FTIR method with N fertilizer management zones determined by the standard laboratory method were compared.

The D-ATR-FTIR method was effective for identifying soil samples that fall in the high and very high categories, as 59 and 93% of samples falling in the high and very high categories, respectively, were identified correctly (Table 1). The D-ATR-FTIR method was less effective for distinguishing samples that fall in the very low, low and moderate categories, as only approximately one-third of the samples falling in each of these categories were identified correctly (Table 1). Most samples that were incorrectly categorized, however, were in an adjacent category. Considering only two N fertilizer management zones; Zone 1—soils that need sidedress N fertilizer (soil NO3−–N < 25 mg kg−1) and Zone 2—soils that do not need sidedress N fertilizer (soil NO3−–N > 25 mg kg−1), then the D-ATR-FTIR method over predicted only 5% of samples (indicated > 25 mg kg−1 while the lab test indicated < 25 mg kg−1) and under predicted only 12% of samples (indicated < 25 mg kg−1 while the lab test indicated > 25 mg kg−1). Although the accuracy reported here only supports two N fertilizer management zones; advances in D-ATR-FTIR technology, the use of larger more diverse calibration databases and advanced chemometric algorithms are anticipated to allow accurate identification of management zones in the future.

Table 1 Percent of samples in the very low, low, medium, high and very high soil N fertilizer management zone categories predicted using the D-ATR-FTIR relative categories determined by the standard laboratory method
Full size table

The maps in Fig. 7 compare the spatial arrangement of LSNT management zones for the four studied fields based on analysis of soil NO3− concentrations by D-ATR-FTIR and the standard laboratory method. Although not identical, similar spatial patterns of soil NO3− concentrations are apparent and the overall effectiveness of the D-ATR-FTIR method for distinguishing between regions that need sidedress N fertilizer (< 25 mg kg−1) versus regions that do not need N sidedress N fertilizer (> 25 mg kg−1) is apparent.

The handheld ruggedized D-ATR-FTIR instrument used in this study requires no sample preparation and is capable of measuring soil NO3− concentrations in non-calcareous soils at field moisture levels in less than one second. These abilities suggest a potential for automation of soil sampling and analysis that would facilitate real-time on-the-go soil NO3− determinations. The need for real-time soil NO3− sensing is illustrated by the maps in Fig. 7. The soil N fertilizer management zones identified by high density grid soil sampling in the four studied fields did not correspond well with soil mapping unit boundaries. This suggests that big data modeling systems that use soil survey data to predict within field variation in sidedress N fertilizer requirements would have missed much of the actual within field variability for these four fields in 2016. In the future, soil NO3− sensing systems may work in combination with big data modeling systems to optimize fertilizer management.

Conclusions

Results of this study demonstrate that the handheld D-ATR-FTIR system is capable of collecting spectra of field moist soils in less than one second and that the spectra can be used to estimate NO3−–N concentrations in non-calcareous soils with sufficient accuracy to facilitate precision N fertilizer management (validation RMSE = 13 mg kg−1). While the D-ATR-FTIR method used in this study was not accurate enough to warrant dividing a field into five N fertilizer management zones, the results were accurate enough to support two N management zones. In the future, advances in D-ATR-FTIR technology and the use of advanced chemometric techniques that leverage larger more diverse datasets may enhance both the accuracy and the precision of soil NO3− measurements. Hand soil sampling and a handheld D-ATR-FTIR spectrometer were used in this study; in the future, however, an automated D-ATR-FTIR soil NO3− sensing system could be coupled with a fertilizer applicator to facilitate modulation of sidedress N fertilizer application rates based on real-time on-the-go soil NO3− sensing.