1 Introduction

The eddy-covariance (EC) technique is the most direct, efficient and reliable method for measuring the turbulent flux of a scalar, including greenhouse gases such as carbon dioxide (Baldocchi 2003; Aubinet 2008), methane (Alberto et al. 2014; Detto et al. 2011), ammonia (Ferrara et al. 2012; Sun et al. 2015), and nitrogen oxides (Famulari et al. 2010; Christen 2014) above the Earth’s surface. The EC approach has been widely used to study the complex behaviour of aerosol fluxes in the atmospheric surface layer, including particle formation, growth, and surface exchange processes (Seinfeld and Pandis 2006). As the processes relating to vertical transport and the exchange of particles are highly relevant to air quality and climate modelling (Wesely and Hicks 2000; Kanakidou et al. 2005; Textor et al. 2006), significant efforts have been made to determine turbulent aerosol fluxes in the atmosphere over several types of canopy: in rural environments (e.g. Gallagher et al. 2002; Deventer et al. 2015a; Rannik et al. 2015), over water surfaces (e.g. Donateo et al. 2012; Bell et al. 2015), and over ice/snow (e.g. Nilsson and Rannik 2001; Grönlund et al. 2002; Contini et al. 2010). Urban environments have also been the focus of several recent studies to characterize particle emission velocities (e.g. Dorsey et al. 2002; Mårtensson et al. 2006; Nemitz et al. 2008; Schmidt and Klemm 2008; Jarvi et al. 2009; Contini et al. 2012; Deventer et al. 2015b). Eddy covariance was further used to directly measure chemically resolved turbulent fluxes of submicron aerosols using a high-resolution time-of-flight aerosol mass spectrometer, quantifying both organic and inorganic components, including ammonium, sulphates and nitrate (Nemitz et al. 2008; Farmer et al. 2011, 2013).

In the last few decades, with the proliferation of long-term worldwide networks of flux sites, such as EuroFlux (Valentini et al. 2000), AmeriFlux (Baldocchi et al. 2001), AsiaFlux (Ueyama et al. 2010; Suzuki and Ichii 2010), and global networks such as FluxNet (Baldocchi et al. 2001), real-time automated procedures have become necessary to reduce the burden of quality control, and to verify the theoretical basis for EC measurements (Finnigan et al. 2003; Moncrieff et al. 2004). Moreover, it is important that data are made available in the public domain as soon as possible after the measurements are made (Mauder et al. 2013).

Requirements for high-quality EC measurements include stationary conditions and fully developed turbulence. In general, it is assumed that most of the net vertical transfer is due to turbulent eddies, and that there are no low-frequency trends in the data. The presence of a trend, which is any frequency component with a period longer than the record length, introduces non-stationarity into the time series. In the surface layer, non-stationarity may originate from diurnal trends and boundary-layer temporal transitions, synoptic-scale or mesoscale variability (Cava et al. 2004; Andreas et al. 2008), and turbulence interaction with sub-mesoscale motions (i.e. wave-like phenomena) (Mahrt 2010; Cava et al. 2015; Sun et al. 2015).

Nowadays, the quality control of eddy-covariance data is based on the stationarity test (Mahrt 1998; Foken et al. 2012; Cava et al. 2014) and on the test of integral turbulence characteristics (Foken et al. 2012). Turbulence statistics computed in non-stationary conditions may lead to biased estimates and to an ambiguous physical interpretation due to the modulation of fluxes related to non-stationary motions (Van de Wiel et al. 2002), which can produce contamination from large time and spatial scales (i.e. mesoscale). For this reason, it is necessary to remove the long-term trends that produce distortion at the low-frequency end of the spectrum (Kaimal and Finnigan 1994; Gash and Culf 1996; Vickers and Mahrt 2003; Barnhart et al. 2012). Non-stationary contributions may occur simultaneously over a variety of time scales, are often not sharply separated from turbulence by a well-defined spectral gap, and are only partially removed by detrending and filtering. According to Kaimal and Finnigan (1994), since the nature of the detrending method determines the shape of the detrended spectrum and the location of its maximum, care is required when using trend removal. However, turbulent fluxes have recently been obtained from raw data through different detrending methodologies, both in real-time and in off-line post-processing to reduce or eliminate the non-stationary effects of long-term contributions (Finnigan et al. 2003). Our aim here is to investigate the performance of the commonly used detrending procedures for the calculation of turbulent fluxes in real-time and automated procedures by comparing the fluxes of different scalars at two different sites.

While different methods for trend removal have been proposed in the literature, general agreement on how to separate low-frequency perturbations from turbulence has not yet been reached. The most commonly applied methods in real-time procedures include linear detrending, two types of high-pass filters, namely the moving average or running mean (Moncrieff et al. 2004), and the autoregressive filter (McMillen 1988; Rannik and Vesala 1999), which are to be compared here with a spectral filter approach based on the Fourier decomposition of the time series in post-processing analyses. The spectral filter is used as the reference method because of its ability to separate turbulent and larger-scale motions in the presence of a well-defined spectral gap (Mahrt 2014).

Detrending methods are applied here to the calculation of turbulent fluxes of different scalars (temperature, ultrafine particle number concentration, carbon dioxide and water vapour concentrations). The fluxes measured at two different measurement sites (an urban and a suburban site) are compared to assess the performance of the various methodologies, and their possible dependence on site characteristics.

2 Detrending Methods

The simplest method for obtaining the fluctuating components for the calculation of covariances according to Reynolds averaging under ideal stationary conditions is the mean removal method (Monin and Yaglom 1971; Stull 1988). Here, the turbulent fluctuation \({x}'\) is obtained as a difference between the measured time series \(x\mathrm{(}t\mathrm{)}\) and the non-fluctuating part

$$\begin{aligned} x'=x-{\bar{x}}. \end{aligned}$$
(1)

In the case of mean removal, \(\bar{x}\) may be determined simply as the arithmetic mean of the time series x over a period T (or a number of points N for a discrete series) as

$$\begin{aligned} {\bar{x}}=N^{-1}\mathop \sum \limits _{j=1}^N x_j . \end{aligned}$$
(2)

While mean removal is adequate in stationary conditions, steady-state conditions rarely exist in the atmosphere.

2.1 Linear Detrending

The linear detrending of eddy-correlation data for flux calculations (Gash and Culf 1996) initially divides data into blocks of, e.g., 30 min, followed by the linear regression of the block of data as a function of time for

$$\begin{aligned} x=St+I, \end{aligned}$$
(3)

where S and I are, respectively, the slope and intercept of the line obtained from a linear regression of x for each block of time. This rapid method can be used to calculate turbulent fluxes in real-time during data acquisition. While linear detrending is a computationally efficient procedure in removing non-stationarity due to synoptic-scale variations, it is not so helpful in the case of non-stationarity due to sub-meso or mesoscale flows (Camarori et al. 1994).

2.2 Running Mean

Another method to obtain the fluctuating components for covariance calculation is to use a running mean based on the previous N data points, where N depends on the chosen time window \(t_\mathrm{w}\) (generally from 5 to 15 min). The running mean is a high-pass filter of contributions to the covariance from atmospheric motions of a time period longer than the specified window width (Bendat and Piersol 1958). A smaller time window results in the removal of higher frequency components from the time series.

2.3 Autoregressive Filter

In the autoregressive filter, \(\bar{x}\) is obtained by high-pass filtering the time series x

$$\begin{aligned} {\bar{x}}=a{\bar{x}}_{i-1} +\left( {1-a} \right) x_i , \end{aligned}$$
(4)

where \(a=\hbox {exp}\left( {-\Delta t/\tau } \right) , \Delta t\) is the time difference between two consecutive measurements and \(\tau \) is the time constant of the filter (Rannik and Vesala 1999). The filter is similar to the simple running mean, but weighted in such a way that distant samples have an exponentially decreasing weight on the current mean, while never reaching zero.

Of the four detrending methods described above in attenuating the low-frequency part of the cospectrum, the mean removal has the smallest effect on the cospectrum, whereas the autoregressive filter attenuates the cospectrum the most (Rannik and Vesala 1999). The time constant \(\tau \) has to be related to the required cut-off frequency of the filter, with McMillen (1988) suggesting a value of 200 s as adequate for his dataset. The same value was also adopted by Rannik (1998), while Rannik and Vesala (1999) showed that higher values (around 1000 s) overestimated the variances.

2.4 Spectral Filter

Spectral decomposition allows for the determination of the typical time scales contributing to turbulent energy and fluxes. The power spectrum (\(S_{x})\) of an arbitrary flow variable x is defined as the Fourier transform of the autocorrelation function of x and its integral is proportional to its variance

$$\begin{aligned} {\overline{x{'}^{2}}} =\sigma _x^2 =\int S_x \left( f \right) \mathrm{d}f, \end{aligned}$$
(5)

where f is the natural frequency in Hz. The cross spectrum between two variables (scalars) x and y is the Fourier transform of the cross-correlation function of x and y (Jenkins and Watts 1968). The real part is the cospectrum (\(C_{xy})\), the integral of which gives the covariance

$$\begin{aligned} {\overline{{x'y'}}}=\int C_{xy}(f)\mathrm{d}f. \end{aligned}$$
(6)

The spectra and cospectra of momentum and scalars exhibit a peak related to the dominant turbulent time scales, which is often separated by a spectral gap between the mesoscale and sub-mesoscale motions. The time scale of turbulent eddies and of the gap region depends on the atmospheric stability, and in near-neutral and unstable conditions the wavelength of the gap tends to naturally increase with height. In stable conditions, the gap remains almost constant (Vickers and Mahrt 2003).

The effect of mesoscale flow on turbulent statistics and fluxes is filtered by removing all low-frequency spectral contributions below a threshold as detected on the basis of the spectral gap position in different stability conditions. Spectral analysis of velocity components and scalars is executed by means of a fast Fourier transform on runs of about 3 h in length in order to minimize the red-noise effect (produced by frequencies lower than 1/T, where T is the period for calculation) on the range of frequencies of interest. The high-pass filtered time series is obtained by nullifying the Fourier coefficients relative to frequencies lower than the detected threshold, and then by applying the inverse Fourier transform. Turbulence statistics and fluxes are computed on the filtered time series using an averaging time of 30 min as for the other detrending procedures. While this procedure is often applied in post-processing analyses, it is used as a reference method here because of its ability to separate turbulent and mesoscale motions, in particular in identifying the aforementioned spectral gap.

3 Measurement Campaigns

We collected turbulence data during two different experimental campaigns over surfaces with different degrees of complexity (Table 1): suburban (Donateo and Contini 2014) and urban (Contini et al. 2012) canopies (in Italy).

Table 1 Summary of relevant properties of the two experimental sites: the measurement height (\(z_{m})\), roughness length (\(z_{0})\) and displacement height (d)
Full size table

3.1 Suburban Canopy

The measurement campaign in the suburban area was carried out between 12 and 30 July 2010 in the experimental field of the Lecce Unit of Institute for Atmospheric Sciences and Climate within the University Campus, which is located about 3.5 km south-west from the town of Lecce. The site is a rectangular field with the longer side of about 200 m; it is characterized by short vegetation, with the two contiguous sides surrounded by small trees. More details of the site can be found in Donateo and Contini (2014) and Conte et al. (2015).

3.2 Urban Canopy

The urban measurement campaign was performed between 12 March and 8 April 2010 in the urban area of Lecce (Puglia) located in the south-east of Italy. Instruments were placed on the corner of a roof of a high school on a mast about 2.5 m tall at a total height of 14 m above ground level. The school is located near a four-lane road with traffic rates of up to 2400 vehicles per hour. A more detailed discussion and characterization of this site can be found in Contini et al. (2012).

4 Instruments

At both experimental sites, micrometeorological systems based on the EC technique measured turbulent fluxes of momentum, mass and energy. The wind velocity components (u, v, w) and sonic temperature (\(T_\mathrm{s})\) were measured using an ultrasonic anemometer (Gill R3); water vapour (q) and carbon dioxide \((\hbox {CO}_{2})\) concentrations were measured by means of a \(\hbox {CO}_{2}/\hbox {H}_{2}\hbox {O}\) infrared open-path analyzer (LI-COR LI-7500) at both the urban and suburban sites. The open-path infrared gas analyzer LI-7500 has a fixed path length of 125 mm and measures both water vapour and \(\hbox {CO}_{2}\) concentrations through absorption at wavelengths centred at 4.26 and 2.59 \(\upmu \hbox {m}\) (Mauder et al. 2007). For the calculation of particle number fluxes, a condensation particle counter (CPC-Grimm Aerosol, model 5.403; Contini et al. 2012; Conte et al. 2015) measured the total particle number concentration at a sampling frequency of 1 Hz. Additionally, a slow-response thermohygrometer (Rotronic MP100A) was used at both sites for the measurement of temperature and relative humidity.

5 Data Processing

We computed turbulent fluxes for an averaging period of 30 min with standard instrumental and physical corrections of the measured time series. Discarded were spikes in the measurements and wind directions for which the tower wake contaminates data. The wind velocity components are rotated in a streamline coordinate system according to McMillen (1988); the third rotation omitted if the rotation angle exceeded \(10^\circ \) (the absolute value). Fluctuating parts of the measured time series are extracted using the different detrending approaches discussed in Sect. 2, after which turbulent fluxes were computed. Given the fast response of the \(\hbox {CO}_{2}/\hbox {H}_{2}\hbox {O}\) detector, a correction for the high-frequency losses in the calculation of latent heat and \(\hbox {CO}_{2}\) fluxes was not applied. Scalar fluxes of water vapour and \(\hbox {CO}_{2}\) are corrected for density effects following Webb et al. (1980). The effect of air density fluctuations on particle number fluxes is corrected using the approach reported in Buzorius et al. (2000), and, considering its absolute value, the correction was negligible, both for the urban (1%) and suburban (0.2%) campaigns. Due to the relatively long first-order time response of the condensation particle counter (1.3 s), the full atmospheric cospectrum between the vertical wind component and the particle number concentration is under-sampled at high frequencies. The turbulent aerosol flux is thus corrected using the analytical formulation proposed by Horst (1997). The average correction for the high-frequency losses on the particle flux is about 23% in the suburban measurements, and 33% for the urban site. For all scalar fluxes, the correction due to the separation of the scalar instruments and the anemometer (a distance of about 0.15 m) is estimated (Massman 2000) to be negligible, being between 1 and 2% for the scalars considered, and hence not applied in our calculations. The uncertainty of particle-flux estimates due to the random statistical counting errors in the condensation particle counter is estimated assuming random Poissonian counting statistics as reported in Fairall (1984). The average statistical error in the measurements in the urban and suburban campaigns is 0.33 and 1%, respectively. Finally, a stationarity test is performed according to Mahrt (1998), in which a threshold of two is chosen for the non-stationarity index. The percentage of non-stationary data does not change significantly with different detrending methods, varying generally by \({<}1\)%. The cases of non-stationary data change according to the variable studied and, especially, depending on the measurement site (Cava et al. 2014). The average percentages of non-stationary data are reported in Table 2, which also reports the number of residual points after the removal of non-stationary data from the analysis according to the particular detrending method.

Table 2 Percentage of non-stationary data for the three different measurement sites for every variable
Full size table

To obtain a quantitative comparison, fluxes calculated by applying the spectral filter and autoregressive method (at different time constants) are compared, and a series of statistical parameters computed. The slope (S) and the intercept (I) of the best-fit regression line are calculated, together with Pearson’s correlation coefficient (C), which describes the degree of co-linearity between the two datasets. However, these statistics are oversensitive to outliers and insensitive to additive and proportional differences between different datasets (Legates and McCabe 1999).

Several error indices are commonly used in data comparison, including the root-mean-square error (RMSE), a normalized version of which is used here. Specifically, a comparison evaluation statistic is calculated (Singh et al. 2004), viz. the RMSE-observations standard deviation ratio (RSR), which standardizes RMSE values using the standard deviation \(\sigma \) of the reference dataset

$$\begin{aligned} { RSR}=\frac{{ RMSE}}{\sigma }=\left[ {\frac{\sqrt{\mathop \sum \nolimits _{i=1}^N \left( {x_{SF}^i -x_d^i } \right) ^{2}}}{\sqrt{\mathop \sum \nolimits _{i=1}^N \left( {x_{SF}^i -x_{mean} } \right) ^{2}}}} \right] , \end{aligned}$$
(7)

where \(x_{SF} \) is the \(\hbox {i}\mathrm{th}\) observation for the constituent being evaluated, \(x_d \) is the \(\hbox {i}\mathrm{th}\) compared value for the method being evaluated. The RSR values vary from a large positive value down to the optimal value of zero, which indicates \({{ RMSE}} = 0\) or residual variation and therefore a perfect data fit, to a large positive value.

The Nash-Sutcliffe efficiency (NSE) is a normalized statistic that determines the relative magnitude of the residual variance (“noise”) compared with the measured data variance (Nash and Sutcliffe 1970), indicating how well the scatter plot of the data fits the 1:1 line, where

$$\begin{aligned} { NSE}=1-\left[ {\frac{\mathop \sum \nolimits _{i=1}^N \left( {x_{SF}^i -x_d^i } \right) ^{2}}{\mathop \sum \nolimits _{i=1}^N \left( {x_{SF}^i -x_{mean} } \right) ^{2}}} \right] . \end{aligned}$$
(8)

Here, \(x_{SF} \) is the \(\hbox {i}\mathrm{th}\) observation for the constituent being evaluated, \(x_d\) is the \(\hbox {i}\mathrm{th}\) compared value for the method being evaluated, and \(x_{mean} \) is the mean of observed data for the method being evaluated. Here, NSE values range between \({-}\infty \) and 1, with NSE = 1 being the optimal value. Values between zero and 1 are generally considered as acceptable, whereas values lower than zero indicate an unacceptable performance.

Other statistical parameters are calculated to establish a good comparison between different methods, in particular the fractional bias defined as

$$\begin{aligned} { FB}=2\frac{\overline{x_{SF}}-{\overline{x_d }}}{\overline{x_{SF}} +{\overline{x_d}}}, \end{aligned}$$
(9)

and the average relative difference defined as

$$\begin{aligned} \Delta = \frac{{\overline{x_{SF}}-{\overline{x_d}}}}{\overline{x_{SF}}}100, \end{aligned}$$
(10)

where \(x_{SF} \) and \(x_d \) are defined as in Eq. 7. We evaluate the running-mean method (for different time windows), mean removal and linear trending with respect to the spectral filter in a similar manner.

6 Results and Discussion

The different detrending methods described in Sects. 2.1–2.3 are tested here and compared with results obtained by the spectral-filtering method for the extraction of high-frequency turbulent fluctuations. As specified in Sect. 2.4, the spectral filter is used as the reference approach because of its ability to successfully separate turbulent and mesoscale motions, in particular in the presence of a detectable spectral gap between the different flow scales.

The thresholds separating turbulent eddies are chosen on the basis of spectral and cospectral analyses at each experimental site, and for different stability conditions. Spectra and cospectra are computed for periods of about 3 h, and averaged over the entire duration of each of the measurement campaigns. The period of 3 h is chosen to resolve both turbulent and mesoscale motions, and to locate the gap between these two flow components. Moreover, this choice appeared optimal for differentiating the different phases of the diurnal cycle. Figure 1 shows normalized mean spectra and cospectra of temperature and number particle concentration for the urban site, as an example. Spectra and absolute values of cospectra have been artificially shifted and divided into two groups (with different colours) for night (blue shading) and day (red shading) hours for clearer comparison. The 3-h time frames are grouped on the basis of the analysis of the atmospheric stability during the sampling period (not shown). Spectra of \(T_\mathrm{s}\) and cospectra involving \(T_\mathrm{s}\) exhibit clear gaps at \(f = 0.00056\, \hbox {Hz}\, ({\approx }30\, \hbox {min}\)) during daytime and at \(f = 0.0017\, \hbox {Hz}\, ({\approx }10\,\hbox {min}\)) during nighttime, and follow the inertial subrange scaling (Kolmogorov 1941). While the spectra and cospectra of momentum and other scalars exhibit a similar behaviour, the spectral behaviour of number particle concentration appears noisier in the high-frequency range (due to the limited instrumental time response), as well as in the low-frequency range. Because this behaviour has been frequently observed elsewhere (Damay et al. 2009; Held 2014), the comparison with spectra and cospectra of other scalars is often used to obtain more reliable estimates of particle turbulent fluxes (Held 2014). On the basis of these observations, the thresholds for the spectral filter were chosen as the positions of the gaps observed in scalar spectra and cospectra above the urban canopy. The spectral behaviour and, as a consequence, the thresholds obtained for the suburban site, were comparable to those of the urban canopy, probably because of the similar surface characteristics and atmospheric conditions.

Fig. 1
figure 1

Normalized mean spectra (a, b) and cospectra (c, d) of sonic temperature and particle concentration as a function of natural frequency (in Hz) relative to the urban canopy. Spectra and cospectra are artificially shifted and divided into two groups (with different colours) for nighttime (blue shading: dark [0000–0300 LT]; blue [0300–0600 LT]; green [0600–0900 LT]; cyan [1800–2100 LT]; dark-blue [2100–2400 LT]) and daytime (red shading: pink [0900–1200 LT]; red [1200–1500 LT]; orange [1500–1800 LT]). The 3-h frames are grouped on the basis of the atmospheric stability during the sampling period. Black dashed lines represent the slopes theoretically predicted in the inertial subrange for spectra and cospectra (Kolmogorov 1941). The vertical dashed lines indicate the threshold values for separating turbulent eddies from mesoscale motions during nighttime (blue \(f = 0.0017\, \hbox {Hz}\)) and daytime (red \(f= 0.00056\, \hbox {Hz}\)) chosen on the basis of the positions of the spectral and cospectral gaps

Full size image

To assess how the different detrending methodologies affect the contribution of different scales to the scalar transport, cospectra are calculated for the unfiltered (mean removal) and detrended time series (with linear detrending, autoregressive filter and running mean), taking into account the sensible heat flux in two separate daily periods [1200–1500 LT (daytime) and 2100–2400 LT (nighttime)], as reported in Fig. 2.

Fig. 2
figure 2

Cospectra for sonic temperature (\(T_\mathrm{s}\)) and the vertical velocity component (w) calculated for data filtered with different detrending methods (linear detrending, autoregressive, running mean) compared with mean removal (in black). The vertical dashed lines represent the threshold values for separating turbulent eddies from mesoscale motions during nighttime (blue \(f = 0.0017\,\hbox {Hz}\)) and daytime (red \(f= 0.00056\, \hbox {Hz}\)), as in Fig. 1

Full size image

As expected, all detrending methods alter the low-frequency range. Some detrending methods act more incisively than others, and in particular, linear detrending and a running mean are more efficient in eliminating low frequencies with respect to the autoregressive method, as also observed below in the statistical analysis.

The autoregressive filter is used with a variable time constant \(\tau \) (from 10 to 400 s, with different time steps, as reported in Fig. 3) for sensible heat, latent heat, \(\hbox {CO}_{2}\), and particle number fluxes.

Fig. 3
figure 3

The Pearson correlation coefficient of the linear fit, RSR and NSE calculated in the comparison between the spectral filter and autoregressive methods at different time constants \(\tau \) for every variable at the two measurement sites. For greater clarity in each panel, the auto-comparison values of the spectral filter are reported. On the right y-axis, the comparison values for particles (green squares) and \(\hbox {CO}_{2}\) (blue triangles) flux are reported

Full size image

As for both the spectral-filter and autoregressive methods, the distinction between daily (from 0900 to 1800 LT) and nocturnal hours (from 0000 to 0800 LT and from 1900 to 2300 LT) was investigated for the possible effects of atmospheric stability. However, no particular differences were observed between nighttime and daytime data, so that the same \(\tau \) is applied to all hours of the day, and at both sites. The relative difference \(\Delta \) and the relative ratio clearly show that the average fluxes are increasingly underestimated relative to the reference detrending procedure, for an increasing magnitude of the time constant \(\tau \) to 130 s. However, beyond this value, the fluxes are overestimated for all the variables studied. In particular, all fluxes are underestimated by between 40 and 4%, and overestimated by 2% to about 12% (as averaged over the entire day). At the same time, it is particularly evident from the RSR (Fig. 3c, d) and NSE (Fig. 3e, f) values that the best agreement, meaning the minimization of the difference between autoregressive and the spectral-filter detrending, is achieved with a time constant of about 130 ± 10 s. Therefore, particle number fluxes behaved similarly to the other variables with respect to the autoregressive filtering procedure at both measurement sites. For the two campaigns, the worst performance is obtained for fluxes of low magnitude, for example, the latent heat flux at the urban site or the particle number flux at the suburban site.

The performance of the running mean with respect to the spectral-filter method is evaluated similarly to that done for the autoregressive filter. The running-mean detrending method is used with \(t_\mathrm{w}\) from 3 to 21 min (with different timesteps, as reported in Fig. 4). Our comparison highlights the underestimation of average fluxes with respect to the reference procedure, and increasingly so with a decrease in \(t_\mathrm{w}\) from 15 to 3 min for all the variables studied. Otherwise, all fluxes are overestimated with the increasing of \(t_\mathrm{w}\) from 15 to 21 min.

Fig. 4
figure 4

The Pearson correlation coefficient of the linear fit, RSR and NSE calculated as a comparison between the spectral (reference) filter and the running mean for different time windows \(t_\mathrm{w}\) for every variable at the two measurement sites. For greater clarity in each graph, the auto-comparison values of the spectral filter are reported. On the right y-axis, the comparison values for particles (green squares) and \(\hbox {CO}_{2}\) (blue triangles) flux are reported

Full size image
Fig. 5
figure 5

Daily patterns of fluxes calculated with the different detrending methods for each variable contained in the two datasets. Error bars in the graphs represent the standard error of the mean

Full size image
Fig. 6
figure 6

The Pearson correlation coefficient of the linear fit, RSR and NSE values calculated in the comparison between the spectral filter and the other methods (x-axis) for every variable at the two measurement sites. On the right y-axis, the comparison values for particles (green squares) and \(\hbox {CO}_{2}\) (blue triangles) fluxes are reported

Full size image

Specifically, for \(t_\mathrm{w}\) increasing from 3 to 10 min, the sensible heat flux is underestimated and overestimated with respect to the spectral filter by 8 and 1%, respectively (as average over all the day); likewise, the \(\hbox {CO}_{2}\) flux is underestimated and then overestimated by 4 and 2%, respectively, while the latent heat flux is underestimated and overestimated by 16 and 1%, respectively. Similarly, the particle number flux suffered an underestimation of 9% at \(t_\mathrm{w} = 3\) min, to an overestimation of 1% for \(t_\mathrm{w} = 15\) min. The best collinearity between the running mean and the spectral filter is achieved for \(t_\mathrm{w} = 10\) and \(t_\mathrm{w} = 15\) min at the urban and suburban sites, respectively, as inferred from RSR and NSE values (Fig. 4). As for the autoregressive method, a distinction between daily and nocturnal periods is made, so that a difference between the two sets of data may be found. For the daytime, the least difference between the running mean and spectral methods is achieved for \(t_\mathrm{w} = 15\) min, while for the nighttime, the difference is minimized for \(t_\mathrm{w} = 10\) min.

The daily hourly patterns are calculated for fluxes after the application of detrending of the scalars (Fig. 5). The autoregressive method is applied for \(\tau = 130\) s for all variables at both sites. The running mean is applied for the urban and suburban sites with \(t_\mathrm{w} = 10\) and \(t_\mathrm{w} = 15\) min during night and day, respectively. In addition, the daily patterns for linear detrending and mean removal are reported for comparison. Relative differences with respect to the reference method are greater for nighttime than daytime, for sensible and latent heat, and for particle fluxes at urban and suburban sites. These differences may be due to the low values of fluxes observed during the night, so that large relative differences result even with small absolute differences. However, it is not possible to exclude the possible influence of atmospheric stability.

The quantitative inter-comparison between the fluxes calculated on a 30-min basis and the chosen reference method is based on the Pearson correlation coefficient, the RSR value, and the NSE value as shown in Fig. 6 for each variable at both sites. All detrending methods produce an overestimation with respect to the reference procedure. In general, the relative difference is most marked between the reference and mean-removal procedures, becoming less marked in passing from the mean-removal to linear-detrending, to the running-mean procedures. Specifically, the mean removal produces, on average, an overestimation of 3% for all variables for the two sites, while linear detrending gives an average difference of about 2%. From these results, it is clear that linear detrending improved the estimation of turbulent fluxes with respect to the mean-removal procedure, but the best performance is obtained with the running-mean method (with a relative difference of 0.5%). On the contrary, the autoregressive filter is markedly different from the reference method, but is not significantly different from mean-removal procedure.

7 Conclusions

We investigated the performance of common detrending methods used for removing the low-frequency contribution to turbulent fluxes, both in real-time and in off-line post-processing automated procedures. The filtering methods were tested and compared with a spectral filter based on the Fourier decomposition of the time series, used here as a reference. Band-pass filters based on Fourier analysis are often used in post-processing analyses to separate turbulent fluctuations from other mesoscale or sub-mesoscale motions (for example, turbulence and horizontal meandering or nocturnal gravity waves). Filtering methodologies appear optimized in the presence of a well-defined spectral gap, where a clear separation between turbulence and larger scales is evident (Mahrt 2014).

The spectral filter has been used here as the ‘reference method’ because a spectral analysis of data exhibited a clear spectral gap in most cases, thus guaranteeing a reliable result in the filtering process.

We compared detrending methods for the calculation of the turbulent fluxes of different scalars (temperature, ultrafine particle number concentration, carbon dioxide and water vapour concentration) using data collected at two different measurement sites: an urban site and a suburban area. The autoregressive filter was used with time constants \(\tau \) from 10 to 400 s, where average fluxes were underestimated with respect to the reference filter, and increasingly so for time constants \(\tau \) up to 130 s, and overestimated beyond this threshold for all the variables studied. The minimization of the difference between autoregressive detrending and the reference was achieved with a time constant of 130 s. No particular differences were observed between nighttime and daytime data, and a similar behaviour was observed at both measurement sites. A running mean was used with different time windows from 3 to 21 min. In general, the average fluxes were underestimated with respect to the reference procedure, for time windows \(t_\mathrm{w}\) from 15 to 3 min, for all the variables studied. Otherwise, all fluxes were overestimated with the increasing of \(t_\mathrm{w}\) from 15 to 21 min. The running mean showed differences in the best time window, related to the different stability conditions, similarly to what has been observed for the spectral filter. The best performance was obtained when the running-mean method was applied, in both the urban and suburban environments, for \(t_\mathrm{w} = 10\) and 15 min during night and day, respectively. All detrending methods overestimated fluxes with respect to the reference procedure. In general, the relative difference was most marked when comparing the reference with the mean-removal procedure, reducing when passing to the linear and running-mean procedures. From these results, it is clear that linear detrending improved the turbulent-flux estimation with respect to the mean-removal procedure, but the best performance was obtained with the running-mean detrending. In contrast, the autoregressive method showed significant differences with respect to the reference method.

Our study was based on the idea of a comparison between different detrending methods, each being a filter based on a specific temporal window. Thresholds for all temporal windows change, for example, as a function of the surface characteristics of the site and/or as a function of the atmospheric stability conditions. Therefore, threshold parameters must be calibrated accurately for every dataset. However, in conclusion, if the spectral filter method is not used, linear detrending and the running-mean method seem to be better detrending procedures for time-series data than the autoregressive filter (or mean removal), and in particular, the running-mean method is suitable for calculating the particle flux.