1 Introduction

Frequency modulation is observed in many real-life signals that include bat signals, seizure signals emitted by human brain, frequency modulated chirps emitted by radars and sonars, jamming signals, gravitational waves [4]. Detection of such non-stationary signals has become important in applications. For example, detection of seizures is important in medical diagnosis and detection of jamming signal is important in communications.

The matched filter (MF) is a common method for detecting signals in which the observed signal is correlated with a known signal. The MF-based detectors require the signal to be known and provide maximum efficiency in case of i.i.d Gaussian noise. However, in most of the radar and communication systems, noise has non-Gaussian with impulsive characteristics, which causes the MF detectors to be severely defective [5, 34, 35]. Energy detector is an effective method for signal detection in scenarios when shape of the signal is not known in advance [34, 35]. However, it fails to achieve the desired performance in many realistic scenarios where noise power is uncertain [1, 22, 28, 28, 29].

The recorded signals can have high temporal correlation due to oversampling at receiver, inherent properties of transmitted signals, and characteristic of channel [32, 33], whereas the additive noise is temporally uncorrelated. This high temporal correlation of received signals is exploited in covariance-based methods in scenarios where the noise power is unknown [1, 2, 35]. However, these methods are not applicable to non-stationary frequency modulated (FM) chirps because of lack of temporal correlation in such signals.

In our previous study [13], we extended covariance-based detectors for FM signals by using de-chirping to remove frequency modulation from non-stationary signals to obtain temporally correlated signals. The de-chirping operation requires the estimate of the instantaneous frequency (IF), which was obtained in our earlier study by a simple peak detection algorithm. Previous studies have shown that the accuracy of IF estimates depend on (a) selection of time–frequency distribution (TFD) [15], (b) IF estimation method [16]. In this study, we improve the performance of our earlier method with the following modifications:

  1. 1.

    The adaptive directional time–frequency distribution (ADTFD) is employed instead of fixed kernel methods to obtain a clear TF representation that suppresses noise along the IF curves by directional smoothing [15]. This clear representation of signal in time–frequency (TF) domain results in improved performance.

  2. 2.

    A recently developed modified Viterbi IF estimation algorithm is employed instead of simple peak detection algorithm to obtain the robust estimate of IFs [16].

The main contributions of the paper are given below:

  1. 1.

    A new signal detection method has been developed based on the combination of ADTFD and Viterbi algorithm.

  2. 2.

    Experimental results show that the improvement in IF estimates obtained using the aforementioned modifications significantly improves the detection performance.

2 The Proposed Detector

2.1 Signal Model

Let us assume that a single sensor receives a signal contaminated by Additive white Gaussian noise (AWGN) with uncertain noise power [28]. The detection model is given as:

$$\begin{aligned} \mathcal {H}_{\text {1}}: x[n]= & {} s[n]+w[n]\nonumber \\ \mathcal {H}_{0}: x[n]= & {} w[n] \end{aligned}$$
(1)

In this study, we assume that s(n) is a non-stationary frequency modulated signal given as:

$$\begin{aligned} s(n)=a(n)\exp \left( j\varphi \left( n\right) \right) , \end{aligned}$$
(2)

where \(\varphi \left( n\right) \) and a(n) are instantaneous phase and instantaneous amplitude of s(n), respectively. It is assumed that a(n) has slow variations and its spectrum does not overlap with \(\mathrm{e}^{j{\varphi }(n)}\). In subsequent section, we propose a method for the detection of such signals under noise power uncertainty.

2.2 The Proposed Detector

2.2.1 Computation of Adaptive Directional Time–Frequency Distribution

Quadratic time–frequency distributions (TFDs) are frequently employed for the analysis and parameter estimation of non-stationary chirp signals. TFDs concentrate the energy of signal along the IF curves while spreading the noise energy in the entire TF plane. The Wigner–Ville distribution (WVD) is a core TFD defined as [9]

$$\begin{aligned} W[n,k] = \sum \limits _{m = 0}^{M - 1} {x[n + m]{x^H}[n - m] {\mathrm{e}^{ - {\textstyle {{j2\pi km} \over M}}}}} . \end{aligned}$$
(3)

The quadratic nature of the WVD introduces undesired oscillatory cross-terms [9]. These cross-terms can overlap auto-terms and thus degrade the readability of signal. Cross-terms are usually suppressed by applying smoothing kernels [9]. However, the suppression of cross-terms by smoothing also deteriorates the energy concentration of auto-terms in the TF plane. Previous studies have shown that directional smoothing along the major axis of auto-terms suppresses cross-terms while preserving auto-terms [27]. However, such smoothing is not optimal in scenarios when signal components follow more than one directions in TF plane. In such scenarios, local adaptation of the smoothing kernel on point by point basis leads to better suppression of cross-terms without degrading auto-terms [21]. Such TFDs are called adaptive directional time–frequency distributions (ADTFD) and are defined as [14]:

$$\begin{aligned} \rho [n,k] = W[n,k]\mathop {*}\limits _n \mathop {*}\limits _k {\gamma _\varphi }[n,k] \end{aligned}$$
(4)

where \({\gamma _\varphi }[n,k]\) is double derivative directional filter described as [14]

$$\begin{aligned} {\gamma _\varphi }[n,k] = \frac{{ab}}{{2\pi }}{\mathrm{e}^{ - {a^2} n_\varphi ^2 - {b^2}k_\varphi ^2}}\left( 1 - {a^2}k_\varphi ^2\right) \end{aligned}$$
(5)

where \({n_\varphi } = n\cos (\varphi ) + k\sin (\varphi )\), and \({k_\varphi } = - n\sin (\varphi ) + k\cos (\varphi )\). Parameter a controls the extent of smoothing along major axis while parameter b determines the extent of smoothing along minor axis. Note that the direction of the smoothing kernel needs local adaptation, which is done on point by point basis as [14]:

$$\begin{aligned} \varphi [n,k] = \frac{{2\pi m}}{M}\arg \mathop {\max }\limits _m \left| {\left| {p[n,k} \right| * * {\gamma _m}[n,k]} \right| , \end{aligned}$$
(6)

where \({\gamma _m}[n,k]\) is the directional filter rotated at angle \(\frac{{2\pi m}}{M}\), with m from \( - M/2,\ldots 0\ldots ,M/2\). Note that Polynomial WVD [7] can also be used as these methods achieve good energy concentration for mono-component nonlinear frequency modulated signals. However, these methods produces non-oscillatory cross-terms which cannot be suppressed by smoothing unlike WVD. Therefore, these methods fail to give desired performance in case of multi-component signals. One alternative for analysis of nonlinear FM signals is the local polynomial WVD that achieves high energy concentration for nonlinear FM signals in the TF domain and can be used for accurate estimation of IF [30]. Moreover, for multi-component signals, its implementation using the S-method can significantly reduce cross-terms for well-separated signals [26]. However, the S-method is not effective for resolving closely spaced signals. The directional smoothing kernel method, presented in this section, can resolve close components as shown in earlier studies [21].

2.3 Instantaneous Frequency Estimation Using the Modified Viterbi Algorithm

A key step of the proposed approach is to accurately estimate the IF of the given signal. A number of TF methods for estimation of IFs have been developed that includes adaptive short time Fourier transform methods [17], methods based on intersection of confidence intervals [12, 18] and Viterbi-based IF estimation methods [16, 24]. Among these methods Viterbi-based IF estimation method achieves better performance in low SNR [11]. In our earlier study, we improved the performance of the Viterbi algorithm by incorporating additional constraints and using the ADTFD as an underlying TFD [16].

In order to estimate the IF, we choose the path that leads to the maximization of the following objective function [16]

$$\begin{aligned} \hat{f}(n) = \mathop {\arg \min }\limits _{f(n)} P(f(n);{n_1},{n_2}) \end{aligned}$$
(7)

where f(n) is the path/IF joining \(n_{1}\) and \(n_{2}\), P(f(n) is the penalty function of f(n). The penalty function is optimized based on the following criteria [16]:

  • The IF should pass through high energy TF points. This constraint is implemented by function h(.) that uses amplitude of frequency bins at a given time instant to sort them in descending order.

  • The IF should follow a smooth curve, i.e., there should not be abrupt transitions between two consecutive IF points. This constraint is expressed as:

    $$\begin{aligned}&g(f(n),f(n + 1)) = 0,\quad \mathrm{for}\;\;|f(n) - f(n + 1)| < \varDelta \nonumber \\&g(f(n),f(n + 1)) = c|l(n) - l(n + 1)|,\quad \mathrm{otherwise.} \end{aligned}$$
    (8)

  • The consecutive IF points should have the same direction.

    $$\begin{aligned}&h(f(n),f(n + 1)) = 0\quad \mathrm{for}\;\; |\varphi (n,f(n)) - \varphi (n,f(n + 1))| < \nabla \nonumber \\&h(f(n),f(n + 1)) = d\,|\varphi (n,f(n)) - \varphi (n,f(n + 1))|\quad \mathrm{otherwise} \end{aligned}$$
    (9)

The final penalty function is obtained as

$$\begin{aligned} P(f(n);{n_1},{n_2}) = \sum \limits _{n = {n_1}}^{{n_2}} {g(f(n),f(n + 1)) + } h(f(n),\quad f(n + 1)) + q(\rho [n,k)), \end{aligned}$$
(10)

We use the generalized Viterbi algorithm to solve the above optimization problem efficiently.

2.4 Instantaneous Phase Estimation

The relation between the instantaneous phase, i.e., \(\varphi (t)\) and the IF, i.e., f(t) can be expressed as [6].

$$\begin{aligned} \varphi \left( t\right) =\int _{0}^{t}{f(\tau )\mathrm{d}\tau } \end{aligned}$$
(11)

For the discrete signals, we can derive:

$$\begin{aligned} \hat{\varphi }\left[ n\right] =\hat{\varphi }\left[ n-1\right] +\frac{2\pi }{N}\hat{f}\left[ n\right] , \end{aligned}$$
(12)

where N stands for the number of frequency bins in a TFD. However, due to the unavailability of the initial phase, the phase can be estimated up to a constant.

2.5 De-chirping

The given FM non-stationary signal is de-chirped using the instantaneous phase estimated in the previous section to convert it into a temporally correlated stationary signal as [8, 20, 31]:

$$\begin{aligned} y\left[ n\right]= & {} x\left[ n\right] \exp \left( -j\hat{\varphi }\left[ n\right] \right) \nonumber \\= & {} s\left[ n\right] \exp \left( -j\hat{\varphi }\left[ n\right] \right) +w\left[ n\right] \exp \left( -j\hat{\varphi }\left[ n\right] \right) \nonumber \\\approx & {} \alpha \exp \left( -j\left( \hat{\varphi }\left[ n\right] -\varphi \left[ n\right] \right) \right) +\tilde{w}\left[ n\right] \nonumber \\&\alpha \exp \left( -j\phi \right) +\tilde{w}\left[ n\right] \end{aligned}$$
(13)

where \(\phi \) is a constant phase. The above equation indicates that the de-chirped signal is temporally correlated while the noise remain uncorrelated after de-chirping.

2.6 Generalized Likelihood Ratio test

The stationary signal, y[n], obtained as the results of de-chirping operation is temporally correlated. Let us solve the following generalized likelihood ratio test (GLRT) equation [3]:

(14)

where \(f\left( \mathbf {y};\varvec{\varSigma }_{1,\mathbf {y}}\right) \) is the likelihood function under hypothesis \(\mathcal {H}_{1}\) and \(f\left( \mathbf {y};\varvec{\varSigma }_{0,\mathbf {y}}\right) \) is the likelihood function under \(\mathcal {H}_{2}\). Similarly, \(\varvec{\varSigma }_{0,\mathbf {y}}\) and \(\varvec{\varSigma }_{1,\mathbf {y}}\) are covariance matrices under hypothesis \(\mathcal {H}_{0}\) and \(\mathcal {H}_{1}\), respectively. Solving (14), we get

(15)

where \(\text {det}\) stands for determinate, \(\hat{\varvec{\varSigma }}_{1,\mathbf {y}} =\text {E}\left[ \mathbf {y}\mathbf {y}^{H}\right] \) and \(\hat{\varvec{\varSigma }}_{0,\mathbf {y}} \approx \text {diag}\hat{\varvec{\varSigma }}_{1,\mathbf {y}}\) [2].

3 Performance Comparison

We assess the efficiency of the proposed method considering a non-stationary signal given as:

$$\begin{aligned} s=\mathrm{e}^{2{\pi }j(0.48n+0.000012n^{3})} \end{aligned}$$
(16)

We corrupt the signal using AWGN with uncertain noise power:

$$\begin{aligned} \sigma _{\mathrm{un}}^{2}\sim \mathcal {U}\left( \frac{\sigma ^{2}}{\alpha _{\mathrm{nu}}}, \alpha _{\mathrm{nu}}\sigma ^{2}\right) \end{aligned}$$
(17)

where \(\alpha _{\mathrm{nu}}\ge 1\) and \(\alpha _{\mathrm{nu}}=1\) indicates no noise uncertainty [28]. We have set \(\alpha _{\mathrm{nu}}=2\) for this study. The \(\sigma ^{2}\) is mean noise power of \(\sigma _{\mathrm{un}}^{2}\), provided uncertainty in the noise power knowledge which leads to severely degradation in the efficiency of the detection technique. In such scenario, we compare the efficiency of the proposed detector with our earlier work based on de-chirping [13], covariance method [35], ridge energy detector [23], and energy detector using probability of detection curves estimated by performing 10,000 simulations for each SNR. Figure 1 shows the probability of detection curve with false alarm probability 0.01 for SNR varying from \(-\,6\) to 4 dB. The proposed method achieves higher probability of detection as compared to all other methods for all SNR levels.

Fig. 1
figure 1

Performance comparison between proposed detector based on Viterbi and ADTFD versus TF detector using simple peak detection for IF estimation [13], generalized likelihood test [2, 3], t–f ridge detector [23] and energy detector. Noise power uncertainty parameter \(\alpha _{\mathrm{nu}}=2\)

Full size image

Let us now repeat the above experiment for a multi-component signal defined as:

$$\begin{aligned} s=\mathrm{e}^{2{\pi }j(0.48n-0.000012n^{3})}+\mathrm{e}^{2{\pi }j(0.02n-0.000012n^{3})} \end{aligned}$$
(18)
Fig. 2
figure 2

Performance comparison between proposed detector based on Viterbi and ADTFD versus TF detector using simple peak detection for IF estimation [13], generalized likelihood test [2, 3], t–f Ridge Detector [23] and Energy Detector. Noise power uncertainty parameter \(\alpha _{\mathrm{nu}}=2\)

Full size image

Figure 2 shows the probability of detection curve with false alarm probability 0.01 for SNR varying from \(-\,6\) to 4 dB. The proposed method achieves higher probability of detection as compared to all other methods for all SNR levels.

Conventionally, methods that use the energy along the IF curve [10, 23] have been used in applications such as radar signal processing [10]. The proposed method and ridge energy detector have a common step of estimating IF. The ridge energy detector uses the energy along the IF curve as a measure to detect the presence of a signal. The proposed method uses the IF information to de-chirp the non-stationary signal into a stationary signal. Then the temporal correlation in the stationary signal is also used as a side information for signal detection that results in improved signal detection as illustrated in Figs. 1 and 2.

4 Conclusion

A TF-based non-stationary signal detection method has been presented. The proposed method first uses a combination of adaptive directional time–frequency distributions (ADTFD) and Viterbi algorithm to obtain robust estimate of instantaneous frequency of the given signal. Then estimated IF is then exploited to remove the frequency modulation from the signal. Experimental results indicate that the use of ADTFD and Viterbi algorithm has significantly improved the performance of the detection algorithm as compared to our earlier method for low SNR[13]. In future, we will try to further improve the performance by using higher order TF methods such as polynomial WVD and local polynomial Fourier transform [7, 19, 25].