1 Introduction

Radio Frequency Interference (RFI) is a term used for man-made signals in radio bands that are inevitably picked up by radio telescopes trying to study radio signals from celestial sources. RFI limits our ability to measure cosmic signals, and thus, its mitigation is necessary. While we are building radio telescopes with unprecedented sensitivities, the RFI is also increasing due to the increasing electronic communication from terrestrial and extra-terrestrial sources (e.g., Baan 2010; Ford & Buch 2014; Thompson et al. 2017). Thus, RFI mitigation is still needed despite locating the radio telescopes in remote locations with scarce human populations. Moreover, the telescope receiver systems and other on-site devices can also generate RFI that needs to be measured and excised.

RFI is broadly classified as ‘broadband’ and ‘narrowband’ categories based on the spectral properties (e.g., Ford & Buch 2014). The effect of broadband RFI is spread across the entire observing frequency band, while that of narrowband RFI is localized to narrow ranges of frequencies within the observing band. Typical sources of broadband RFI include powerline sparking due to gap or corona discharge and automobile sparking. The sources of narrowband RFI include broadcast transmitters, navigational satellites and wireless communication from mobile phones and short-range transmitters.

RFI mitigation has been explored in many ways and implemented based on the requirement of observatories. The most widely used techniques operate on the data after it has been recorded; we refer to these as offline techniques. PIEFLAG is a python-based tool to identify RFI from a dataset using amplitude-based and RMS-based methods (Middelberg 2006). Another tool, which is widely used on Murchison Widefield Array data and LOFAR is AO-Flagger, which is effective on narrowband RFI (Offringa et al. 2015). More recently, methods based on Deep Fully Convolutional Neural (DFCN) Networks (Kerrigan et al. 2019) and Convolutional Neural Networks (CNN) (Sun et al. 2022) have also been developed to improve the speed and efficacy of RFI identification and flagging. These have been used and tested on telescopes mainly affected by narrowband RFI. For mitigating broadband RFI, methods that operate in real-time, as the data are recorded, are needed.

The Upgraded Giant Metrewave Radio Telescope (uGMRT) (Gupta et al. 2017) is a pathfinder of the Square Kilometer Array (SKA) and is located at Khodad, about 90 km, North of Pune in India. It consists of 30 dish antennas distributed roughly in a ‘Y’-shaped array (Swarup et al. 1991) with maximum and minimum baseline lengths of 25 km and 100 m, respectively. The ‘upgrade’ of the GMRT involved the installation of new receivers and a wideband correlator to allow data recording with broad instantaneous bandwidths (Gupta et al. 2017). The uGMRT is a system offering broad instantaneous bandwidths of up to 400 MHz. Furthermore, it operates in four frequency bands, namely, 120–240 MHz (band 2), 250–500 MHz (band 3), 550–850 MHz (band 4) and 1000–1450 MHz (band 5).

This paper focuses on the broadband RFI at the uGMRT, the real-time mitigation (filtering) system implementation and the effect of filtering on astronomical observations. Further, we motivate ideas for the SKA, which proposes a threshold-based RFI detection and filtering system in the signal processing chain. The review is organized as follows. We provide an overview of the broadband RFI at the uGMRT in Section 2. The real-time RFI filtering technique is described in Section 3. A review of the tests carried out is presented in Section 4, followed by the description of the impact of the filter on astronomical data in Section 5. The current usage statistics for the RFI filtering are reported in Section 6. The relevance of real-time filtering is discussed in Section 7. Conclusions are presented in Section 8.

2 uGMRT and RFI

The GMRT Wideband Backend (GWB) (Reddy et al. 2017) consists of 16 ROACH-1 boards, which carry out real-time RFI filtering and packetize the data into 10 Gigabit Ethernet (GbE) packets for further processing, using a 16-node GPU cluster. GWB operates in the correlation and beamforming modes. Further, the beamforming can be carried out using incoherent array and phased array modes (Reddy et al. 2017).

Powerline RFI at the GMRT is caused due to sparking on high-power transmission lines and distribution equipment. Many domestic power lines for residential and farming requirements are located near the array. Apart from this, 11 kV, 33 kV, HVAC and 500 kV DC lines cross the outermost arm antennas (Raybole & Sureshkumar 2010). Several potential sources of RFI are located around the central square and near the arm antennas of the GMRT array (Swarup 2008).

Powerline RFI generally manifests as a bunch of impulses in the time domain and has properties similar to gap discharge (Maruvada 2000). These bunches show a periodicity and usually repeat at multiples of powerline frequency (50 Hz in India) (Swarup 2008). The typical periodicity is 10 ms. This RFI depends on observing frequency, and its strength is significant in bands 2, 3 and 4 of the uGMRT. The strength, duration and number of impulses in the bunch depend on the moisture, wind and faults. A typical time-series plot of a signal affected by powerline RFI is shown in Figure 1.

Powerline RFI is 10–20 dB stronger than the system noise and leads to broadband RFI increase in the power level in the power spectrum of the astronomical signal. Impulses from powerline RFI are outliers and make the Gaussian distribution of the signal heavy-tailed. This type of RFI is spatially correlated and shows a strong signature on short baselines, as seen in Figure 1.

Fig. 1
figure 1

Oscilloscope output at 5 ns sampling from C05 and C06 antennas of the GMRT at band 4 is shown. This 50 ms period trace of amplitude versus time was recorded at the input of the GWB for two polarizations of C05 and C06 antennas (\(\sim \)100 m baseline). The top two subplots show C05 (dark blue color) and C06 (red color) for Pol. 1, respectively, and the bottom two subplots show the same antennas, i.e., C05 (blue color) and C06 (green color) for Pol. 2, respectively. The powerline RFI has a periodicity of 10 ms and is correlated across the antennas and polarizations.

Full size image

2.1 Need for real-time filtering

Real-time RFI filtering at the highest time resolution helps prevent data corruption in downstream signal processing with minimal loss of astronomical data. The benefits of having a real-time filtering system in the digital backend are as follows:

  1. 1.

    The energy in impulsive time-domain RFI spreads across the observing band upon Fourier transformation, making it difficult to detect and mitigate in the post-processing operation.

  2. 2.

    Since the RFI is broadband, tuned or notch filters are not helpful for mitigation, even if used in the earlier stages of the receiver system.

  3. 3.

    Mitigation in the pre-correlation domain helps to reduce the ill effects of correlated RFI.

3 Real-time RFI filtering technique

Real-time RFI filtering technique developed and implemented (Buch et al. 2019) at the uGMRT is based on detection and excision (Buch 2019) of interference. The RFI filtering operation is divided into three basic steps—estimation, detection and filtering. These operations are carried out on each antenna and polarization data. The robustness and performance of Median Absolute Deviation (MAD)-based estimators in detecting impulsive RFI are shown in Buch et al. (2016a). Detailed architecture and implementation are provided in Buch et al. (2019). The filter can also operate in detection-only mode, bypassing the filtering operation.

  1. 1.

    Estimation: MAD is computed on Nyquist-sampled digitized time series (\(\textrm{MAD} = \textrm{median} (\textrm{abs}(X_i - \textrm{median}(X)))\)) on 16k data samples. To get an unbiased estimate, the minimum window size should be twice the expected duration of RFI burst (Buch et al. 2016b). For longer bursts of RFI, the median of 16k MAD values are computed for getting Median-of-MAD (MoM), which we refer as 16k MoM. This method is a trade-off between estimating a long duration and hardware resource utilization. Further, MoM is computed on every fourth input sample out of the samples arriving at 1.25 ns intervals to meet the real-time requirements. Thus, the estimation interval for 16k MoM is 1.34 s, which can be calculated as \((16384 \ \textrm{samples}*16384 \ \textrm{MAD} \ \textrm{values}*1.25*10^{-9} \ \mathrm{sampling \ period})*4 = 1.34)\) s.

  2. 2.

    Detection: The upper and lower thresholds are computed from the MoM value as \(\textrm{median} \pm n*1.4826*\textrm{MoM}\). The factor n is programmable. The two detection techniques designed for the RFI filtering system are (a) voltage detection and (b) power detection. The voltage detection technique compares the values of the individual samples with the thresholds, whereas the power detection technique uses the squared and accumulated samples for comparison with the threshold. The threshold mentioned above is used for voltage detection, whereas an appropriate scaling is carried out to extend the threshold for power detection (based on the number of samples accumulated). At this stage, the flagging statistics are recorded using an RFI counter that keeps a record of the number of detected RFI samples and a total number of samples per antenna and polarization. These counts would provide the fraction of flagged samples over user-defined time intervals during an observing run.

  3. 3.

    Filtering: The samples detected as RFI can be replaced with digital noise, a constant value, or a threshold. The user can choose a replacement option. Samples from a standard normal distribution are generated on Field Programmable Gate Array (FPGA) (Buch et al. 2014) and are scaled to the estimated statistics of the input signal.

Fig. 2
figure 2

A simplified block diagram of a 2-element correlator/beamformer for GWB showing the location of the real-time RFI filtering system is presented.

Full size image

3.1 Implementation

The RFI filtering system is an FPGA-based implementation on a ROACH-1 boardFootnote 1 from Collaboration for Astronomy Signal Processing and Electronics Research (CASPER) (Hickish et al. 2016).

The system is implemented at the start of the signal processing chain. A simplified block diagram of the chain showing the location of the filtering system is shown in Figure 2. Each ROACH-1 board processes 4 inputs, digitized to an 8-bit precision at 800 MHz sampling frequency. Signals from four different antennas of a particular polarization are fed to a single ROACH-1 board. A total of 16 ROACH boards cover the 30 dual-polarized GMRT antenna inputs. The RFI filtering system is programmed and configured through a control computer via a 1 Gigabit Ethernet (GbE) interface. The parameters of the filtering system on each board can be configured separately through a software interface. RFI counter readout is also through the 1 GbE interface.

An earlier design version is available in the CASPER open-source library.Footnote 2 Hardware optimization and real-time operation are achieved using the histogram method (Buch et al. 2019) for median computation. Other trade-offs and optimization are described in Buch et al. (2019).

4 Review of testing schemes

The real-time RFI filtering system tests are carried out at two levels—Engineering and Astronomical Tests. A simultaneous recording of unfiltered and filtered data is carried out for a fair comparison. The inputs from 30 antennas are split to get 15 unfiltered and 15 copies that undergo filtering. We call this 1:2 digital copy mode (Buch et al. 2022). The digital backend system is appropriately programmed to treat copies from the corresponding unfiltered counterparts. A detailed description of 1:2 digital copy mode, including the data recording and acquisition process for correlator and beamformer outputs, is provided in Buch et al. (2018, 2022). Similarly, for comparing multiple filter configurations simultaneously, 28 antenna inputs are split into seven unfiltered copies and seven each with different filter configurations (different threshold or replacement options) to get 1:4 digital copy mode (Buch et al. 2022).

The digital copy modes are also used to observe the effect of threshold and replacement options on the signal-to-noise ratio (SNR) and compare the data between the beamformer and correlator modes (Buch et al. 2018). These tests help to fine-tune the threshold and replacement options to optimize the filter performance.

4.1 Engineering tests

The tests are carried out to compare the raw voltage, spectrogram, beamformer and correlator (visibility) outputs.

We provided controlled inputs to the filtering system for characterization during the initial testing phase. We could change the duration and strength of RFI and characterize the filtering in the 1:2 digital copy mode. Examples of results are shown in Buch et al. (2018). Such controlled testing allows for setting an appropriate input power level and setting a bound on the filter’s performance.

Fig. 3
figure 3

Beam output and its FFT for a single frequency channel corresponding to 431 MHz. The data is from the C09 antenna of GMRT for a 20 min observation of 3C147 in band 3, 200 MHz bandwidth, 2048 spectral channels and 1.3 ms integration time.

Full size image

More detailed tests are carried out on calibrator radio sources to analyze the effect on SNR, suppression of 50 Hz powerline signature, cross-correlation function, and closure phase. These tests were carried out for a wide range of receiver parameters like bandwidth, observing band, number of spectral channels and integration time.

4.1.1 Effect on SNR

For analyzing the effect on SNR, we carry out a simultaneous comparison between unfiltered and filtered spectrograms (magnitude spectrum over a specific time interval) for a GMRT antenna. We study the temporal behavior of a single frequency channel. A quantitative comparison is made between the theoretical SNR given by the radiometer equation, particularly, for instances affected by RFI. The ratio of SNR before filtering to the one after filtering, measures the improvement achieved through the filtering system. The improvement depends on the intensity of RFI and the filtering efficiency. Through several test observations carried out in the different uGMRT observing bands, we have observed an improvement in the SNR ranging from 0.5 to 10 dB.

4.1.2 Suppression of 50 Hz and harmonics

The presence of powerline RFI manifests as 50 Hz and harmonics in the Fourier transform of a single frequency channel computed as a function of time. The magnitude transforms of unfiltered and filtered data are compared to determine the amount of suppression in the fundamental (50 Hz) and harmonics. The ratio of power in the 50 Hz for unfiltered data to the filtered data provides the quantitative measure of the RFI suppression. We have observed a typical suppression of 6–10 dB (Buch et al. 2022).

A comparison of power from a GMRT antenna for a spectral channel, while observing a calibrator source (unresolved) in uGMRT band 3 is shown in Figure 3. The observation was carried out in 1:2 digital copy mode. The filtering of RFI is seen along with suppression in the 50 Hz and harmonics.

Fig. 4
figure 4

CCF of a single frequency channel corresponding to 651 MHz is shown for baselines involving the labeled antenna with C09. The data is from a 20 min observation of 3C468.1 in band 4, 10 antennas, 100 MHz bandwidth, 2048 spectral channels and 0.671 s integration time. The approximate baseline lengths in meters with respect to C09 antenna are C00 (654 m), C04 (678 m), C06 (110 m), C11 (674 m), E02 (3009 m), E06 (13076 m), S02 (4383 m), S04 (9378 m) and S06 (13996 m).

Full size image
Fig. 5
figure 5

Comparison of closure phase formed by C00, C02 and C04 antennas and their filtered counterparts. The plots shown are for a single spectral channel (corresponding to 431.64 MHz) versus time. The baselines lengths for C00–C02, C02–C04, and C00–C04 corresponding to spectral channel number 700 are 688 m, 581 m and 1263 m, respectively. The Y-axis of the upper subplot shows the normalized cross-correlation magnitude for the unfiltered and filtered baselines, and the middle subplot shows the cross-correlation phase. The lower subplot shows the closure phase for the unfiltered and filtered antennas.

Full size image

4.2 Effect on cross-correlation function

The uncertainty in the cross-correlation function (CCF) measurement increases in the presence of RFI. We simultaneously compare the CCF across different baselines for time-averaged visibilities (Buch et al. 2022) to observe the uncertainty. Increased power fluctuations due to RFI lead to increased uncertainty in the measurement. Such fluctuations are significant at baselines <1 km and lead to correlated RFI even when the antennas are pointing off-source (Buch et al. 2018). The amount of reduced uncertainty in the filtered data indicates an improvement in the visibility data. A comparison of CCF for a calibrator source (unresolved) observed in uGMRT band 4 is shown in Figure 4. The observation was carried out in 1:2 digital copy mode. The increased CCF on shorter baselines is due to broadband RFI, which is reduced after filtering.

4.3 Effect on closure phase

This test helps to understand the effect of broadband RFI on the closure phase relations. Since the powerline RFI is prominent on shorter baselines, we choose three antennas and corresponding filtered copies. A comparison is made between the closure phase of the baselines computed from the cross-correlation data for unfiltered and filtered data. The closure phase departs from zero for data corrupted by RFI. This measurement is independent of any instrumental effects and can help to study the effect of RFI on astronomical data. We observed a typical variation of 70–\(100^{\circ }\) in the phase in the event of RFI, which reduces to about \(15^{\circ }\) after filtering (Buch et al. 2022).

A comparison of the closure phase for a calibrator source (unresolved) observed in uGMRT band 3 is shown in Figure 5. The observation was carried out in 1:2 digital copy mode.

5 Impact on astronomical data

5.1 Continuum imaging

We expect that removing broadband RFI can lead to better data and, thus improved images. We have presented the effects on the visibilities and images in (Kale et al. 2020).Footnote 3 Here, we provide an example of one such test to illustrate the impact.

The facility of recording filtered and unfiltered data simultaneously (1:2 mode) was used to test the impact of real-time RFI mitigation on continuum imaging. Data from half of the GMRT array were recorded, and the same data after filtering were recorded in the 1:2 mode. The observation was carried out like a typical continuum observation involving scans on primary calibrator, secondary calibrator and target source. Since the broadband RFI is correlated at the short baselines, the test included imaging of an extended source. The target Abell 2744 was chosen due to extended emission (Rajpurohit et al. 2021) that would be detected even with half of the array. We carried out the test at bands 3 and 4. A summary of the observations is provided in Table 1.

5.1.1 Data analysis

For radio interferometric data analysis, Common Astronomy Software Applications (McMullin et al. 2007) of the NRAO is the standard package. To follow identical analysis steps on the unfiltered and filtered data, we used an automated pipeline. We analysed our data using an early version of CASA Pipeline-cum-Toolkit for uGMRT Data Reduction (CAPTURE) (Kale & Ishwara-Chandra 2021) that allowed exact data analysis steps to be performed on the unfiltered and the filtered data. The standard steps of flagging (removing bad data from non-working antennas and RFI), calibration and imaging were performed. Self-calibration was carried out, as it is standard practice when making continuum images, to improve the sensitivity of the image.

Table 1 Summary of uGMRT observations.
Full size table
Fig. 6
figure 6

Flagging percentages versus uv-distance bins are plotted. The uv-distance bins in units of kilolambda are labeled in the plot. The solid lines are for band 3, and the dashed ones are for band 4. The unfiltered data are shown in red and the filtered in blue.

Full size image

5.1.2 Results

Here, we present the results that illustrate the impact of real-time filtering on the data and images. We compared the percentage flagging over uv-distance bins in the visibilities at bands 3 and 4. The uv-range was divided into 10 bins, and in each bin, the flagging percentage was obtained using the CASA task ‘visstat’. The result is presented in Figure 6. The flagging percentage is the highest in the shortest baselines. For both the bands, the filtered data resulted in \(\sim \)15% less flagging. We also studied the impact on continuum images at bands 3 and 4. Here, we present radio images that are made only using short baselines (\({<}2 \)klambda) at band 3 (Figure 7). The root mean square (RMS) values in the images from the unfiltered data is 1.1 mJy beam\(^{-1}\) and that from the filtered data is 0.7 mJy beam\(^{-1}\). On comparing, the images show that the filtered image can recover the extended central source with a better SNR. Compared to the RMS noise level in the filtered image, the unfiltered image has artefacts (Figure 7).

5.2 Spectral-line observations

We carried out spectral-line observations to test the effect of real-time RFI filtering. The target source was the HI 21cm absorption line of the \(z=2.192\) galaxy towards TXS2039\(+\)187 (Kanekar et al. 2013), observed simultaneously with the GMRT Software Backend (GSB; 32 MHz backend at GMRT) (Roy et al. 2010) and the GWB, with \(\sim \)40 min of on-source time. The GSB used a bandwidth of 2.067 MHz with 512 channels, while the GWB used a bandwidth of 3.125 MHz and 4096 channels; the bands were centered on the red-shifted 21 cm line frequency, 445.0 MHz. Figure 8 shows an overlay plot from the GSB and GWB observations. The narrow absorption line is clearly seen in both spectra. However, the GWB spectral baseline appears cleaner than the GSB spectral baseline. The RMS noise of the GWB spectrum is 20% better than that of the GSB spectrum after smoothing the spectra to the same spectral resolution.

Fig. 7
figure 7

Image of A2744 made with the use of baselines \({<}2\)klambda at band 3 using unfiltered data (left) and filtered data (right). The contours are drawn at identical levels in the two panels. The green contours are at \(-0.0014\); blue is at 0.0014 Jy beam\(^{-1}\). This represents a level of 2\(\sigma \), where \(\sigma \) is the RMS in the filtered image. The white contours are at 0.0021 and 0.0042 Jy beam\(^{-1}\) and represent 3 and 6\(\sigma \) levels. The coordinates and the color scale (Jy beam\(^{-1}\)) are matched in the two panels. The beam size of \(97.8''\times 44.3''\), position angle of \(13.3^{\circ }\) is shown as a filled ellipse at the top left in the panels.

Full size image

5.3 Time-domain astronomy

We conducted tests at multiple epochs to investigate the online RFI filtering system’s performance for beamformer observations using the uGMRT. We recorded filtered and unfiltered data from the beamformer observations in this experiment. We also recorded both incoherently phased array (IA) and coherently phased array (PA) data. We observed multiple pulsars at uGMRT bands 2 and 3 in this setup at multiple observing epochs using the total intensity mode. Thus, we record four simultaneous data files for each target pulsar for (a) IA-without filter, (b) IA-with filter, (c) PA-without filter and (d) PA-with filter. The detection of pulsar J0418–4154, a pulsar discovered in the GMRT High-Resolution Southern Sky (GHRSS) survey (Bhattacharyya et al. 2016), for these four scenarios is shown in Figure 9 and the test details are mentioned in Table 2. Significant improvement of SNR is seen with online filters in both IA and PA mode observations. To summarize, we observed improved detection in IA mode with the RFI filtering for the observations at uGMRT band 3. However, we observe comparable or better SNR detection in PA mode with RFI filtering. We note that the observed improvement in the detection significance will depend on the RFI condition of the particular epoch and the threshold chosen for the RFI filtering. We observe marginal or no improvement in the SNR in the case of our band 2 observations, which could be because a significant fraction of data gets flagged for band 2, which was severely affected by RFI.

The RFI filtering system was also used for Fast Radio Bursts (FRB) observations. Data from a typical FRB observation undergoes de-dispersion, similar to pulsar observations, followed by detection and selection of candidates. RFI produces many spurious candidates and the RFI filter dramatically reduces the number of candidates. In a particular FRB search observation using uGMRT band 4 and 14 antennas, the RFI filter resulted 20 times fewer candidates (i.e., from 10,000 to 500), significantly improving the chances of detecting an actual FRB event in the presence of RFI, which would otherwise spawn mostly spurious candidates (Figure 10).

Fig. 8
figure 8

Overlay plot of spectral line observations using the GSB and GWB.

Full size image
Fig. 9
figure 9

Prepfold (part of standard pulsar analysis software PRESTO (Ransom et al. 2002)) detection plot for PSR J0418−4154: (a) IA-without filter, (b) IA-with filter, (c) PA-without filter and (d) PA-with filter. Significant improvement in detection significance while using real-time RFI filtering is observed.

Full size image
Fig. 10
figure 10

Plot showing FRB candidates in observing runs with (blue circles) and without (red squares) using the RFI filtering.

Full size image
Table 2 Details of the observing setup for PSR J0418–4154.
Full size table

6 Current usage statistics

The RFI filtering system is released on a shared-risk basis from the GMRT Time Allocation Committee (GTAC) observing cycle 38 (April–October 2020). Since then, several observations in uGMRT bands 2, 3 and 4 have used the RFI filtering system. Observing continuum radio sources and detection of weak pulsars and FRBs have primarily used RFI filtering. We present band-wise statistics for the GTAC cycle 40 in Table 3.

The filtering usage has increased in the last two cycles, reaching almost 50–80% in bands 2 and 3. A recent publication of Schellenberger et al. (2022) cited the RFI filtering system. The filter was also used in the GHRSS survey.Footnote 4

RFI filtering system will be released as a regular feature for bands 3 and 4 from April 2022 (GTAC cycle 42), and band 2 will continue on shared risk. New features based on the uGMRT user community feedback are being introduced to the filtering system, while improving our understanding of the effects on astronomical data.

The percentage of RFI detected and filtered for a given threshold depends on the number of outliers in the data. For a pure noise signal, about 0.03% of data are filtered for a 3\(\sigma \) threshold. Table 4 provides the typical average percentage RFI filtered for uGMRT bands at 3 and 4 sigma thresholds. These percentage values are likely to change in the case of a very strong RFI and are dependent on the RFI environment. We recommend the filtering threshold based on the tradeoff from the benefits of filtering to the artefacts introduced due to excessive filtering.

User-programmable features of the RFI filtering system are the selection of threshold factor (n value in the threshold equation shown in Section 3), replacement options, bypass mode and applying filtering to either or both the polarization. The replacement options include replacement by digital noise, constant value and threshold value. The recommended settings of threshold and replacement are provided in the GTAC user document.

The typical flagging percentages from various tests and GTAC observations are 4–5% for band 2, 1–2% for band 3, 0.5–1.5% for band 4 and \({<}0.5\)% for band 5.

Table 3 Real-time RFI filtering usage during GTAC observations in cycle 40 (April–October 2021) for uGMRT band 2 (120–240 MHz) and band 3 (250–500 MHz). The table shows the total approved GTAC duration and RFI filtering usage (in hours and percentage of total duration) along with the type of observation.
Full size table
Table 4 Comparison of average percentage flagging for different uGMRT bands and noise at 3 and 4\(\sigma \) thresholds.
Full size table

The facility of recording the flagging counter, as mentioned in Section 3, can be used during the observations. The commands for recording the RFI flagging count can be added to the command file used for GTAC observations. Command file generation using online tool,Footnote 5 if the online RFI filtering option is activated, adds the commands every 5 min interval in the scan period.

The recorded RFI flagging data can be converted to percentage flagging for each counter epoch. We have developed a utilityFootnote 6 to plot the percentage flagging of individual counter epochs, the average for all the antennas, and a single antenna for multiple epochs. An example of percentage flagging for GMRT antennas is shown in Figure 11.

Fig. 11
figure 11

Antenna-wise average percentage of RFI flagging for an observing run.

Full size image

7 Relevance to the SKA

SKA is expected to receive RFI from various terrestrial and extra-terrestrial sources (like communication satellites) for SKA-Low and SKA-Mid telescopes even though the location is remote with low-population density and radio-quiet zones (Otto et al. 2016; Tingay et al. 2020) characterizing. The RFI environment at the SKA sites is mainly dominated by narrowband RFI though there is a possibility that in future, there may be broadband RFI at the SKA antennas that are far from the core (SKA-TEL-SKO-0000492Footnote 7). The report also deals with signal chain design considerations and has proposed using threshold-based detection of RFI. The report also mentions the need to excise strong RFI in the pre-correlation domain and generate a flag from the detector. Detection of RFI in high-resolution data through antenna-level buffering is also proposed in Nelles et al. (2019) for the SKA-Low.

The current implementation of the broadband RFI filtering at the uGMRT can be extended for usage with other observatories, such as the SKA. As mentioned in Section 3, some trade-offs were considered, while designing and testing the RFI filtering. These were specific to the FPGA and the hardware board used in the GWB. The most computationally challenging aspect is the real-time implementation of the median calculation, which was carried out using the histogram method (Buch et al. 2019). An alternative to the sorting required for computing the median can be achieved through the histogram method, optimal for integer arithmetic. The complexity of this technique primarily depends on the input bit-width and window size (Buch et al. 2019). For example, for a q-bit input bit-width, the complexity would scale as \(2^q\). The histogram method is suitable for input bit-width less than or equal to eight and a window size of up to 65,536 samples.

The technique operates on raw voltage and is an antenna-based detection and filtering scheme making it convenient to scale the computational and resource requirements to the SKA level. The design is not directly dependent on the receiver’s bandwidth; the only constraint is the real-time operation. The system architecture is modular, and the filtering parameters are user-configurable.

The SKA-1 signal processing system is planned to be implemented on an FPGA platform (van Es et al. 2020). The following are the possible trade-offs for implementing the real-time system on hardware or software platforms planned to be used for the SKA.

  • Computing standard deviation on discrete intervals: In this option, the standard deviation is computed over a particular window and applied to the following several windows. The computation is carried out at discrete intervals. This trade-off would provide relaxation in the real-time constraint. The interval between the standard deviation computation can be decided on the variability in the RFI environment and on the real-time constraints provided by the digital signal processing system.

  • Filtering-only mode: Here, the robust standard deviation can be calculated externally or at the start of the observation and supplied to the filtering system. The filtering operates on real-time data. This scheme would ease the resource and timing constraints on the filtering system. The interval at which the standard deviation can be applied depends on the variability in the RFI environment and other system-related and external parameters affecting the input power levels.

  • Computing standard deviation in time-multiplexed mode: In the case of multiple antennas on a single computing platform, the standard deviation can be computed in a time-multiplexed fashion, i.e., while the computation is going on for one antenna, the other antennas hold the previous value of standard deviation. Filtering happens in real time for all the antennas. This scheme helps to meet resource requirements as there is only one standard deviation computation block for each block of N antennas and being processed on a single FPGA or a GPU node.

7.1 Real-time RFI monitoring

RFI filtering system can work in a piggyback mode with only the threshold-based detection to get the RFI statistics per antenna during each observation. The filtering operation can be bypassed if required. The RFI counter can still operate and provide the flagging statistics. The statistics obtained can be used for long-term monitoring of the RFI environment and understanding the levels of RFI in different bands, RFI direction and the effect of other parameters on varying levels of interference. Depending on the system’s location, it can be used to monitor broadband or narrowband RFI.

7.2 Extension to mitigate narrowband RFI

The concept of real-time RFI filtering developed for uGMRT can be extended for mitigating narrowband signals in the spectral domain and on integrated visibilities or beam data. The algorithm is generic with user-defined parameters and can be used in other telescopes, particularly the SKA. The previous sections showed that the system could filter impulsive RFI at the highest time resolution and improve the quality of astronomical results with minimal data loss. This section provides recommendations on the usage of the filtering system for the SKA.

Depending on the type of RFI and the Interference-to-Noise Ratio (INR) (Series 2013), parameters like the estimation interval or the detection domain (voltage or power) need to be modified to achieve optimum performance. Although the system at uGMRT has been tested and released for broadband RFI, a similar algorithm by Buch et al. (2016a) has been applied to offline time-frequency data in the post-correlation domain for mitigation of narrowband RFI. The algorithm can also be applied to beamformer data for mitigating broadband and narrowband RFI for time-domain astronomy observations (Buch et al. 2016a). Thus, the real-time technique can be used on raw and processed signals at different signal-processing systems. As an example, a possible approach for implementing the technique in the spectral domain on the CPU platform is shown in Buch et al. (2016b).

8 Summary and conclusions

RFI mitigation is essential for all radio band observatories due to the unprecedented sensitivities that are being achieved and the rising interference levels. We reviewed the real-time RFI filtering implemented at the uGMRT. The technique, implementation, testing methods and impact on continuum, spectral-line and time-domain data were described. The release of this system for the users of the uGMRT and the growing usage were presented. The relevance of such schemes in the context of the SKA was also presented. A summary of these sections is provided as follows:

  • The real-time RFI filtering technique is implemented on FPGAs and uses a MAD-based scheme for identifying impulsive RFI. It allows replacing the samples identified as RFI with digital noise, a constant or zero. The user can choose the threshold for RFI identification and the replacement option.

  • The testing of the real-time RFI filtering on the uGMRT has been possible using the specially developed modes of recording filtered and unfiltered data simultaneously.

  • The engineering tests showed that suppression of RFI improved SNR, cross-correlation function and closure phase relations.

  • The astronomical data in continuum mode show a dramatic improvement in the short baseline data and can increase the fidelity of images of extended sources. Spectral line data show an improved estimation of the noise in line-free channels.

  • In the time domain astronomy, real-time RFI filtering has been tested for observing pulsars and FRBs and used in major projects like the GHRSS survey.

  • The real-time RFI filtering at the uGMRT has been released regularly for bands 3, 4 and 5 since April 2022. During the last observing cycle (April–October 2021), \(\sim \)50% observations in band 3 and 80% observed in band 2 used the real-time RFI filtering system.

  • There is an overlap in the observing frequencies of uGMRT and SKA-Low. The mitigation scheme developed for uGMRT is similar to that proposed for RFI mitigation in the SKA signal processing chain.

  • uGMRT RFI mitigation system is modular and amenable for implementation on hardware and software platforms. Various tradeoffs were provided to take care of real-time computational constraints and operate in piggyback mode.

In light of growing RFI levels, mitigation methods that will result in minimal data loss will be necessary. Implementing the real-time RFI filtering scheme at the uGMRT will serve as a pathfinder of schemes for the SKA.