The search for fast transients with CZTI

Abstract

The Cadmium–Zinc–Telluride Imager on AstroSat has proven to be a very effective All-Sky monitor in the hard X-ray regime, detecting over three hundred GRBs and putting highly competitive upper limits on X-ray emissions from gravitational wave sources and fast radio bursts. We present the algorithms used for searching for such transient sources in CZTI data, and for calculating upper limits in case of non-detections. We introduce CIFT: the CZTI Interface for Fast Transients, a framework used to streamline these processes. We present details of 87 new GRBs detected by this framework that were previously not detected in CZTI.

1 Introduction

The Cadmium–Zinc–Telluride Imager (CZTI, Bhalerao et al. 2017b) is a high-energy coded aperture mask instrument on board AstroSat (Singh et al. 2014) CZTI comprises of four independent, identical quadrants giving a total physical area of 976 cm$^2$. Each quadrant consists of a $4\times 4$ array of 5-mm thick Cadmium–Zinc–Telluride detectors, giving good sensitivity in the 20–200 keV energy range and an energy resolution of 11% at 60 keV. In nominal operations, all incident photons are saved in event-mode with 20 $\mu$s resolution. While primary coded field-of-view of CZTI is $4.6^{\circ }\times 4.6^{\circ }$, the collimators and support structure of CZTI become increasingly transparent to radiation at energies above $\sim$100 keV, making it sensitive to sources all over the sky. The off-axis sensitivity depends on the effective area, which in turn is a strong function of energy and direction. Details of the effective area calculations are presented in Mate et al. (2021). As there are very few bright sources in this energy range, the net contribution of off-axis sources is small and simply manifests itself as a slightly elevated background.

A special exception to this are bright, short-duration transient sources like gamma ray bursts (GRBs). GRBs with their high brightness and short durations (seconds to minutes) manifest themselves as an increase in the count rates in CZTI. Starting from the first GRB detection on the day the instrument was powered on (GRB 151006A; Bhalerao et al. 2015; Rao et al. 2016), CZTI has detected 325 GRBs in the five years since launch. On the other hand, the lack of a measurable change in count rates corresponding to a transient event can be mapped to an upper limit on the flux of the transient. With this technique, we have obtained stringent upper limits on X-ray emission from Fast Radio Bursts (Anumarlapudi et al. 2020), as well as from gravitational wave sources (Bhalerao et al. 2017a).

In this paper, we describe the methods used for searching for such sources (called fast transients hereafter). In Section 2, we discuss the pre-processing of data for our searches. In Section 3, we discuss the search for “known” transients, where the time and possibly location are known from other sources. We also discuss methods for putting upper limits on the flux from such transients in case they are not detected in data. In Section 4, we discuss in detail the algorithms, software, and the interface developed for searching for transients in all of CZTI data. In Section 5, we discuss the performance of our software, and present the 87 transients detected in our searches. We conclude by discussing future improvements in Section 6.

2 Preparing the data

The CZTI data reduction pipeline¹ is designed for imaging and spectroscopy of sources in the primary field of view. There are two particular operations in the pipeline that are detrimental to the search and analysis of fast transients. First, the pipeline discards data from time intervals when the on-axis source being targeted by AstroSat is occulted behind earth — though CZTI might still detect fast transients that are located elsewhere in the sky. Second, sections of data where the count rates in detectors rise above a certain value are discarded as noisy: thus suppressing bright transients. For fast transient searches, we overcome these issues by changing a few pipeline parameters — thus ensuring that final data products are still compatible with any post-processing software. We follow the standard procedure to obtain Level-2 “bunch cleaned” data created by cztbunchclean. Next, when selecting good time intervals with cztgtigen, we change the config file mkfThresholds.txt to remove the earth occult condition (the ELV parameter), which would have discarded data when the on-axis target was occulted by the earth. The next stage is to reject noisy sections of data using cztpixclean. The default settings of cztpixclean discard intervals where a single pixel has more than 2 counts per second, or where a module has more than 35 counts per second. To ensure that this step does not discard bright transients, we raise the detector count threshold to 1000 and the pixel count threshold to 100. Finally, we run cztevtclean to obtain cleaned event files. Since our processing is done independently for each quadrant, we use the _quad_clean.evt files.

The next stage is to create light curves for each quadrant. Here we have to carefully correct for various sources of dead time in the instrument: for instance quadrant-level dead time (0.3 s dead time for collecting housekeeping data every 100 s), and module-wise dead time (arising from discarding particle–induced photon bunches). We use the pipeline module cztbindata to consider all these factors to correctly calculate the dead time for each time bin used. For certain searches, we also limit select the photon energy ranges in this step.

The final step in data preparation is to remove the orbit-induced trends in the background. As AstroSat is in low earth orbit, the satellite sees a variable background count rate over different parts of the earth, rising near the South Atlantic Anomaly (SAA). We see that the background variations are relatively smooth, over timescales of hundreds of seconds. But, if a transient event were to evolve on comparable or longer timescales, we would not be able to distinguish it from background variations. Fortuitously, most transients of interest have timescales of tens of seconds or shorter. Hence, we can fit a smooth trend to the data and subtract it, effectively making the data “background-free” and greatly simplifying the task of transient detection. We have tested two methods for de-trending the data: in the first method, the trend is estimated by using a running median filter of 100 second width. In the second method, we estimate the background using a second order Savitzky-Golay (savgol) filter of 100 second width (for details see Anumarlapudi et al. 2020). Both trend estimates work well, and hence both are coded into our software. In preliminary testing, the savgol filter yielded better results for transient searches, hence it is set as the default filter.

3 Triggered searches

In CZTI data analysis, searches for fast transients are broadly categorised into two types: “triggered” and “blind”. Triggered searches are cases where the time of a transient, and possibly its position, are already known. For such cases, a qualitative search is carried out by pre-processing the data followed by visual examination. Blind searches, that are more quantitative, are discussed in Section 4.

3.1 Method

Triggered searches start with pre-processing the data as discussed in Section 2, up to the creation of cleaned event files. We then create “spectrograms” or “time-energy plots”: two-dimensional histograms of the event data, and visually examine them for the transient (Fig. 1(a)). By default, the energy axis is binned in 10 keV bins from 20–200 keV. Searches are carried out by binning the time axis in 0.1 s, 1 s, and 10 s bins. We also calculate two further variants of this spectrogram to aid visual searches: we calculate the mean spectrum and subtract it from each time bin, thus highlighting any transient variations (Fig. 1(b)). In the third step, we take these mean-subtracted spectrograms and normalise the light curve in each energy bin by its standard deviation (Fig. 1(c)). This de-weights noisy energy bands, and gives a rough idea of the statistical significance of any transient.

Light curves from a single quadrant occasionally show noise spikes which look similar to astrophysical transients. These events — often caused by charged particles or electronic noise — typically occur at low energies ($\lesssim$50 keV). Since the four quadrants of CZTI are electronically independent, the electronic noise events are always caused in just a single quadrant. Such noise candidates are readily rejected by requiring that any transient is considered “detected” only if it is detected across multiple energy bins, and seen in more than one of the four independent quadrants of CZTI. Track-like events created by charged particles can sometimes be simultaneously seen in multiple quadrants. Such cases are always of short duration (<1 s), and can be discarded based on their track-like count distributions in the detector plane. Overall, four quadrant detections of transients are most unambiguous, but detections coincident in three or two quadrants are also considered acceptable if they pass the above cuts, are bright and broadband.

Figure 1

Spectrograms for quadrant C data for GRB 200306A, utilised in visual inspection of transient candidates (Section 3.1). Panel (a): The upper left frame shows raw data, binned in 1 s and 10 keV bins along the X and Y axes respectively. The upper right frame shows the spectrum, obtained by summing the spectrogram along the X axis. The lower left frame shows the light curve, obtained by summing the spectrogram along the Y axis. The lower right frame shows the distribution of count rates in the light curve. Panel (b): Mean-subtracted spectrogram, obtained by subtracting the average spectrum from each time bin. The four frames are analogous to Panel (a). Panel (c): Mean subtracted and sigma-normalised spectrogram. Note that the transient is brightest at the lowest energy bins (Panel (a)), but since those energies also have a higher sigma, the transient is statistically most significant around 60 keV (Panel (c)).

Figure 2

Calculation of transient properties, illustrated with the light curve of GRB 200306A. Panel (a): The raw 20–200 keV light curve summed across four quadrants, with the transient region marked in green. An initial background trend (orange) is fit to the background outside the transient region, and refined (purple) with sigma-clipping outlier rejection. Outliers are marked with red circles. The mean value of the refined trend is reported as the background count rate, $R_b$. Panel (b): De-trended light curve obtained by subtracting the background trend. The peak count rate ($R_p$) and total counts ($C_\mathrm {tot}$) are measured from this de-trended light curve. Panel (c): A cumulative light curve calculated from Panel (b), normalised such that the median pre- and post-transient values are 0 and 1 respectively. Dashed lines indicate the points where data cross the 5% and 95% levels, which is used to calculate ${T}_{90}$. Panel (d): Multiple light curves are generated from Panel (a) by assuming Poisson noise distribution, and the four parameters are measured for each of these. The four frames, clockwise from upper left, show distributions of $R_b$, $R_p$, $T_{90}$, and $C_\mathrm {tot}$ obtained from these light curves. These distributions are used to define 90% confidence error bars for the parameters actually measured in Panels (b) and (c). For GRB 200306A, we get $R_b = 495_{-3}^{+4}$ counts s$^{-1}$, $R_p = 289_{-19}^{+51}$ counts s$^{-1}$, ${T}_{90}= 32^{+4}_{-7}$ s, and $C_\mathrm {tot} = 5444_{-1023}^{+449}$ counts.

CZTI also has caesium iodide scintillators as anti-coincidence “Veto” detectors, to reject particle events. Veto detector spectra are sampled once per second, and downlinked along with CZT data. We generate similar spectrograms and light curves for Veto data and repeat the transient search. Since data are intrinsically binned at 1 s, the default searches are carried out only at 1 s and 10 s timescales.

These searches are typically run by the Payload Operations Centre (POC) at IUCAA. Transients detected thus are reported in GCN circulars (see for instance, Gupta et al. 2020; Bhalerao et al. 2016) and announced on the CZTI GRB page at http://astrosat.iucaa.in/czti/?q=grb, along with the associated spectrograms.

3.2 Transient properties

For every detected transient, we estimate its duration (${T}_{90}$), peak rate ($R_p$) above background ($R_b$), and the total counts ($C_\mathrm {tot}$). We create a combined 20–200 keV light curve from all quadrants that show a clear detection of the transient. “Pre-transient” and “post-transient” sections of the light curve are visually identified, and the background is estimated by fitting a quadratic to these. The best-fit quadratic is subtracted from the data to obtain a background-free light curve, and counts are summed to create a cumulative light curve. The post-transient part of this curve gives a measure of the total counts in the transient. The time taken for the cumulative curve to rise from 5% to 95% of the total counts is the $T_{90}$ duration of the transient (Fig. 2). These details are included in the published GCN circulars.

We use a Monte-Carlo approach to estimate the uncertainties in the GRB parameters. We assume that observed photons follow a Poisson distribution, and for simplicity use the observed number of photons in each bin as the rate ($\lambda$) parameter for the Poisson distribution in that bin. We create 5000 simulated light curves by drawing photons from such Poisson distributions for each bin, and measure the four parameters $T_{90}$, $R_p$, $R_b$ and $C_\mathrm {tot}$ for each simulated light curve. We use 5–95% range in the histograms of these parameters (Fig. 2(d)) as the 90% credible intervals. For instance, the observed light curve of GRB 200306A yields $T_{90} = 32$ s (Fig. 2(c)), while the central 90% credible region is from 25 s to 36 s (Fig. 2(d)). We report this as $T_{90} = 32_{-7}^{+4}$ s.

3.3 Count rate limits for non-detections

In cases where no transient is seen, we can place upper limits on the maximum counts received from the transient that would be consistent with noise. Since the mean background level varies through the orbit, we cannot use a direct rate. Instead, we de-trend the data as discussed in Section 2. In addition, due to the noise spikes discussed in Section 3.1, the distribution of count rates deviates significantly from a simple Poisson or Normal distribution. In particular, there is a large tail of positive counts with respect to the mean rate which can mimic transient signals. To overcome this hurdle of an unmodeled count rate distribution, we estimate the upper limits (hereafter referred to as cutoff rates) using data from nearby orbits. The method is based on the assumption that the rate of astrophysical transients detectable by CZTI is low enough that nearby orbits are unlikely to have a large number of transients.

We first decide the width of the window used for transient search, say $t_w = 100$ s, and an acceptable false positives probability (FPP, $\mathcal {F}$). We typically set $\mathcal {F}=0.1$ for a single quadrant. Since we place limits using data from all four independent quadrants, the combined FPP is $10^{-4}$. We now need to find a “cut-off rate” $R_c$ such that the probability of this threshold being crossed by chance in $t_w$ is $\mathcal {F}$. To calculate $R_c$, we select five orbits before and after the transient (excluding the orbit containing the transient) as “witness” orbits. We create light curves for these orbits using the same time bin as used in the original analysis, then de-trend them, and create histograms of the de-trended counts. $R_c$ is defined as the point such that a fraction $\mathcal {F}$ of the data points have counts $>R_c$. A typical orbit has 4000–5000 s of usable data, so that analysis of ten orbits with parameters $\mathcal {F}=0.1$ and $t_w = 100$ s ensure that 40–50 data points are above $R_c$. This makes the method robust to the presence of another transient in the witness orbits.

There are some caveats to be noted here. Occasionally, a quadrant can be extremely noisy in some orbit. If the candidate transient is in such an orbit, that quadrant is excluded from further analysis and there is a corresponding decrease in the FPP (for instance, Mate et al. 2017; Marathe et al. 2019). Our FPP estimates are derived from the probability of getting counts $>R_c$ in each of the four quadrants anywhere in the $t_w$For all three bands, we process the window. In practice, we consider something a detection only if such spikes in counts are coincident across multiple quadrants, hence the actual FPP is even lower.

3.4 Flux calculations

Incident photons from off-axis transients are heavily re-processed (scattering, absorption, fluorescence, etc) by various satellite elements before they are incident on the detector. Hence, the mapping of incident spectra to measured spectra must be done by simulating these effects in software. We accomplish this by using a GEANT4-based mass model of the entire satellite (Mate et al. 2021). Since the effect of the satellite varies with direction, the simulations require knowledge of the source position in satellite coordinates. For transients where the position is known, Chattopadhyay et al. (2021) discuss a method of estimating the source spectrum and flux from CZTI data.

While methods for calculating the source spectrum are still under development, we have found that source flux calculations based on the mass model are quite reliable if the source spectrum is known from other instruments. We leverage this by assuming a power-law or band model spectrum for sources, and calculating the flux corresponding to the number of counts in a quadrant. The total flux from the source is the sum of fluxes in all four quadrants.

For certain transients, most notably gravitational wave events, the source location is not known precisely. Instead, discovery teams provide a sky-map with the source position probability distribution. For such sources, we evaluate the flux limit at each point on the sky map that is not occulted by the Earth at the instant of the transient. The overall flux limit is evaluated as a probability-weighted mean of these values (for instance, see Shenoy et al. 2020).

4 Blind searches for transients

The triggered searches are complemented by a broad “blind” search over all of CZTI data to identify astrophysical transients. We have two pipelines for such searches — a pipeline based on Machine Learning (ML) (Abraham et al. 2019) and the CIFT.² In this section, we discuss CIFT in detail.

The broad outline for the CIFT searches is as follows: First, data are reduced and de-trended as discussed in Section 2. Various algorithms are used to identify outliers in light curves. These outliers are used to create ‘peak maps’ to identify candidate transients in data. Flagged candidates are displayed on an interface for human vetting. They undergo similar quality checks and inspection as discussed in Section 3, and final selected transients are saved in a database.

4.1 Preparing the data

CZTI Level 2 bunch cleaned files are organised into ‘Obs-ID’s which have all the data taken during observations of any particular object requested by an observer. We undertake most of our searches Obs-ID wise, thus typically processing a few to a dozen orbits at a time. We see that noise events are more frequent in lower energies, while data are cleanest at higher energies. To leverage this factor, we divide CZTI data into three energy bands: 20–50 keV, for 50–100 keV, and 100–200 keV. For all three bands, we process the data following steps from Section 2, and create de-trended light curves with 0.1 s, 1 s, and 10 s bins. We also use a 0.01 s binning when searching for counterparts to fast radio bursts. We use the entire energy range for the Veto detector, and create light curves at 1 s and 10 s binning.

Thus, we generally create 36 light curves for CZTI data (3 time bins $\times$ 3 energy bands $\times$ 4 quadrants) and 8 light curves for Veto data (2 time bins $\times$ 4 quadrants) per Obs-ID. We run a search algorithm on each light curve to identify outliers and create ‘peak maps’: boolean masks with value 1 for time bins containing the outliers, and 0 elsewhere. The twelve CZTI peak maps are added together, and any bin with a mask value of four or higher is flagged as a candidate transient. Similarly, the four Veto masks are combined and bins with mask value $\ge 3$ are flagged as candidate transients. Next, we discuss the three outlier search algorithms currently implemented in CIFT.

4.2 Top-N

The Top-N (TN) algorithm is based on a simple heuristic: a transient is expected to have among the highest count rates seen in a given light curve. We identify the brightest N bins in a light curve and flag them as outliers for the peak map.

While testing this algorithm, we obtained better results if the searches were carried out one orbit at a time (as opposed to Obs-ID wise searches for other algorithms). By varying values of N, we obtained the best results for $N = 3$.

4.3 N-sigma

The N-Sigma (NS) algorithm is a straightforward statistics-based method to select outliers in a time series. We identify outliers by using iterative sigma clipping as implemented in the Astropy sigma_clipped_stats module. Starting with a de-trended light curve, we calculate the median and standard deviation ($\sigma$) values, and reject outliers that deviate more than 3$\sigma$ from the median. The process is repeated with the new light curve until convergence is attained, subject to a maximum cap of five iterations. The mean value $\mu$ and the standard deviation $\sigma$ of the final iteration become the key parameters of algorithm. Using these values, outliers are defined as data points with counts $> \mu + N\sigma$, where our default value is $N = 5$. The typical thresholds for flagging these outliers for various time bins, energy bands, and both detector types are given in Table 1. These values were calculated from data of entire five years of the search. We reiterate that the namesake N of this method is used only in identifying outliers for the peak map, while the iterative sigma estimation is always done at a three-sigma level.

Table 1 Combined cut-offs for cutoff rate and NSigma methods for each binning and band. These representative rates were calculated by using all the the five years of data used in this study. Note that rates are in units of counts/s, not counts per bin.

4.4 Cutoff rates based on false positive probability

The cutoff rate based search (CR) algorithm aims at attaining a given False Positive Probability (FPP) for candidate transients. Cutoff rates are determined following the procedure discussed in Section 3.3, with one important distinction. In Section 3.3, we assumed the presence of transient-free data of an order of magnitude larger duration than the timespan of interest. Since CIFT searches are meant to be conducted over all available data, this requirement clearly cannot be met. Instead, we set our FPP threshold based on the expected rates of transients, in particular, GRBs.

The rate of detectable GRBs is a function of instrument sensitivity, energy range, and field-of-view. As a baseline, we note that on average Fermi GBM detects a GRB every 1.5 days (von Kienlin et al. 2020), while the BAT on the Neil Gehrels Swift Observatory averages one GRB every four days (Lien et al. 2016). Based on these we stipulate a rough upper bound of the rate of GRBs detectable by CZTI as 0.5 GRBs per day.³ We then stipulate that only 1% of our GRBs may be false positives (FPP = 0.01), corresponding to one false positive every 200 days.

To arrive at an approximate solution for the FPP criterion, we consider the case of searching for a GRB with 1 s duration in light curves with 1 s binning. In this scenario, our false positive requirement of 1 per 200 days maps to one false positive in $1.728 \times 10^7$ bins. Since most basic acceptance criterion is coincident detection in two or more independent quadrants, each quadrant can have one false positive in $\sqrt{17280000}$ time bins, or 4156 s. This is a significant fraction of an orbit, and hence the robust estimation of $R_c$ requires data from several orbits. Decreasing the time bin size increases the number of samples in the light curve, and owing to the random underlying process, makes outliers more likely. To correct for this, we change our cutoff rate requirements based on the bin size $t_\mathrm {bin}$: $R_c$ is selected such that a fraction $0.01 \times (t_\mathrm {bin} / 4156~\mathrm {s})$ of bins have a count rate $> R_c$.

We note that this is a highly simplified argument, which ignores the 12 light curves we make for every time bin and the $>4$ peak map condition. It also ignores the small effect of presence of transients in our “witness” data sets. However, it serves as a good approximate argument for selecting our $R_c$ thresholds from data.

The typical thresholds for flagging these outliers for various time bins, energy bands, and both detector types are given in Table 1. As in the NS method, these representative rates shown in the table were calculated with all five years included in this work. Some specialised searches use the entire 20–200 keV range as a single band. For such searches with 1 s binning, the cutoff rates for the 4 quadrants are 79, 68, 68, and 69 counts/sec respectively. For searches with 10 s binning, the rates drop to 10, 10, 10, and 12 counts/s respectively, corresponding to a total of 420 counts per 10 s bin.

4.5 The CIFT interface

Once the peak maps have been created by any of the three algorithms discussed above, we apply our candidate selection criteria of requiring $\ge$4 matches out of 12 light curves for CZTI, and at least three matches out of four Veto light curves (Section 4.1). Candidate transients that meet this requirement are flagged as an “event”, and entered into an SQL database. Certain basic properties like like number of quadrants and energy bands an event was detected in, their significance, rates above background, time since last SAA, time from next SAA, etc are also calculated and stored in the database. Events having the same trigger time (for instance, if they were detected by two different algorithms) are grouped, and their corresponding event-IDs are stored under a unique trigger-ID in a separate table. Furthermore, the trigger-events which are within 100 seconds of each other are grouped into a “superevent” and assigned a super-ID. These superevents are the final transient candidates, ready for human inspection.

A separate program for plotting is run in parallel which takes input a list of Obs-IDs and fetches all the superevents in those Obs-IDs from the SQL database. For each superevent, it plots detailed time energy histograms, light curves and calculates $T_{90}$ for each temporal binning.

The CZTI Interface For Transients (CIFT) is a Flask⁴-based interface with SQL database as back-end, available to view the candidate transients (Fig. 3). The interface allows a human scanner to search all superevents by Obs-ID which are displayed in a table on a ‘scanning’ page (Fig. 3(b)). The scanning page has columns for Superevent-ID, trigger time, $T_{90}$ in CZTI and Veto, number of sub-events, a column displaying relevant statistics like background rate, peak rate, number of quadrants the candidate was detected in, etc. and a check-box option to discard multiple superevents at once if bogus. Each superevent-ID is linked to an inspect page (Fig. 3(c)) which lists all characteristics of the superevent, and of each sub-events contained within it, along with several lightcurves of different binning sizes for CZTI and Veto. After inspection, a human scanner can tag the event with custom tags, including “known”, “unknown”, “ambiguous”, “SAA Tentacle”, etc. Superevents can be searched and filtered by tags from the main page (Fig. 3(a)). The CIFT interface also has other features like undertaking triggered searches and a front-end for initiating data processing.

Figure 3

Screenshots of CIFT, showing functionality of various pages. Panel (a): The screenshot of the home page of CIFT, where the human scanner can input dates and the corresponding candidate tag which the scanner wants to see, refer Section 4.5. This main page also allows the user to navigate to other functionalities of the interface where one can add new tags, process the available unprocessed directories and access the diagnostics page, with few clicks. Panel (b): The SQL database displays all the candidates of the specified tag from all Obs-IDs contained within the date range specified on the CIFT main page. Each candidate has a dedicated row where the Superevent-ID, trigger time, $T_{90}$ in both CZTI and Veto, number of sub-events are displayed along with relevant statistics like background rate, peak rate, number of quadrants where the candidate was detected for quick reference of the scanner. Each row also has a appropriate light curve thumbnail for both CZTI and Veto for visual inspection, allowing the scanner to discard the very obvious bogus candidates from this page itself (with the help of discard multiple option). This complete list of candidates is sorted in the ascending order of the Superevent-ID. Panel (c): Each candidate is linked to their inspection page which displays the break-down of all the computed characteristics shown on the Scanning page. The inspection page also contains links to five different lightcurves for different binnings of CZTI and Veto detectors. Based on the inspection of all these parameters and lightcurves, the scanner can classify the candidate and tag the candidate with the appropriate tag.

5 Results

We used our framework to search for GRBs in data from 06 October 2015 when CZTI was first powered on, till 10 October 2020 — spanning just over five years of data. “Slew” Obs-IDs are relatively short data sets acquired when AstroSat is slewing from one source to another. These have been excluded from our search. We detected a total of 347 transients in CZTI data by using CIFT. Of these, 41 are GRBs or triggers previously reported by other missions but missed by POC triggered searches or ML pipeline (Section 5.2), while 46 are new discoveries (Section 5.3). In the same five-year span, triggered searches and the ML pipeline have detected 325 GRBs, of which our searches recovered 260. Two of these missed GRBs were in slew orbits. The reasons for missing $\sim$20% GRBs are discussed in Section 5.5.

5.1 Performance

The processing code takes less than an hour to search for transient candidates in one month of data (approximately 130 GB). Creating diagnostic plots is a slower process which is spawned in parallel, and takes 3–4 hours to complete. Users remotely connect to the http-based interface for scanning the processed data. Visual examination of candidates from a month of data takes a few hours for an experienced user.

Figure 8 shows the break-up of transient detections by the various algorithms. We see that most transients are detected by all three algorithms, followed by detections in both CS and TN. The TN method is solely responsible for the detection of 15% of Veto transients.

Table 2 summarises the performance of all algorithms. We see that there are a large number of false positives, particularly from the Veto detectors. This underscores the need for human vetting of the candidate superevents.

On an average, CIFT flags about 339 candidates per month, adding up to 19628 candidates in 58 months of data. For the months of April and May 2020, we lowered the thresholds to search for even faint bursts associated with the outburst of the galactic magnetar/FRB candidate SGR 1935+2154 (Mereghetti et al. 2020). We selected the top 5 peaks in the TN method, and required a coincidence of just 2 bands out of 12 in CR and NS methods. These reduced thresholds increased the number of candidates by a factor of 4.3, giving 2936 candidates in just 2 months.

The most common type of false positives comprised of coincident detections in two Veto quadrants in just a single second, with no discernible signal in adjacent bins. These are most likely particle events, and are rejected. A closely associated class of Veto false positives are events that have a very sharp rise and an exponential decay: again a profile common for particle events. On the other hand, Veto light curves of GRBs that are also detected in CZT detectors show a wider variety. Hence we decided to keep the coincidence threshold for Veto as 3 out of 4 quadrants at the expense of missing possible real transients, and this was the number discussed at the start of Section 4.

Other large number of bogus detections include false peaks near SAA due to bad de-trending or inadequate SAA masking which can be ruled out during human vetting. In CZTI data, many false events are caused by a single pixel, generating noise events at all energies. Visual examination of the distribution of counts in the detector helps to quickly dismiss these as false positives. If the light curves are well-behaved with no real transients or noise spikes, then the TN algorithm often generates false positives by identifying “outliers” that are completely consistent with background.

As human scanners gain more experience with the pathologies of false positives, we are working to improve automatic rejection of such candidates.

Figure 4

The normalised lightcurves of GRBs detected by CIFT, that were reported by other instruments but had not been identified in CZTI or Veto data (Section 5.2). Each GRB lightcurve is normalised and labeled with the GRB name. Panel (a) shows normalised lightcurves for the GRBs detected in CZTI. The three sub-panels are with 0.1 s, 1 s and 10 s binning respectively, and each sub-panel is ordered by peak count rate above background, increasing from top to bottom. Panel (b) shows the normalised lightcurves of GRBs that were detected in Veto. These are plotted with a 1 s binning, and are also ordered by peak count rate above background, increasing from top to bottom.

Figure 5

The normalised lightcurves of GRBs discovered by CIFT that have not been reported by any instrument before (Section 5.3). Each GRB lightcurve is normalised and labeled with the GRB name. Panel (a) shows normalised lightcurves for the GRBs detected in CZTI. The three sub-panels are with 0.1 s, 1 s and 10 s binning respectively, and each sub-panel is ordered by peak count rate above background, increasing from top to bottom. Panel (b) shows the normalised light curves of GRBs that were detected in Veto. These are plotted with a 1 s binning, and are also ordered by peak count rate above background, increasing from top to bottom.

5.2 Known transients

We detected 41 transients (referred as ‘Known’) that had previously been reported by other instruments but had not been identified in CZTI or Veto data (Fig. 4). These transients were matched to earlier reports in GCN Circulars,⁵Fermi GBM Burst Catalog⁶ and the Fermi sub-threshold trigger lists.⁷$^,$⁸ Table 3 lists the key properties of these transients: a superevent ID, standard GRB name, trigger times (UTC), algorithms that detected the transient in CZTI or Veto data, temporal binning used in analysis, and the peak time (AstroSat time, measured as seconds since UT 2010-01-01, 00:00:00). We then list the calculated parameters: the duration ($T_{90}$), peak count rates above background, background count rates and total counts across all quadrants. We prefer using CZTI data to calculate these parameters. Even when our algorithms find a transient only in Veto detectors, we manually check if CZTI data can be used for calculation for uniformity. We use Veto data to calculate transient properties only if the transient is unseen in CZTI light curves. These cases are demarcated clearly in Table 3.

5.3 CIFT discoveries

We discovered 46 new transients that have not been reported by any instrument before. As in Section 5.2, we show their light curves in Fig. 5 and list properties in Table 4. Six of these transients have been published already: GRB 180112B (Sharma et al. 2018), GRB 190628B (Marathe et al. 2019), GRB 191102A (Shenoy et al. 2019a), GRB 191105B (Shenoy et al. 2019b), GRB 191119A (Shenoy et al. 2019c), and GRB 200817B (Shenoy et al. 2020).

Figure 6

(a) Distribution of $T_{90}$ values of all CZTI GRBs. (b) Distribution of peak count rates of transients in CZT detectors. Note that high peak rates are often obtained for short duration transients analysed with 0.1 s binning. (c) Distribution of total counts in CZT detectors. Comparing the distributions of the duration ($T_{90}$), peak count rate, and total counts in three search classes. Blue lines (“POC GRBs”) denote transients detected in regular triggered searches and ML pipeline searches. Orange lines (“Common GRBs”) denote the transients CIFT detected among the “POC GRBs”. Green lines (“known GRBs”) denote transients that have been reported by other instruments (Section 5.2) but missed by POC searches or ML pipeline, while red lines (“Discovered GRBs”) denote the new transients discovered with CIFT.

5.4 Properties of new transients

The new transients detected by CIFT (Sections 5.2 and 5.3) span a wide range of properties. The shortest transient was GRB 200907A ($T_{90} = 0.13$ s), while the longest was GRB 180809C with $T_{90}$ of 290 sec. GRB 200510B had the highest count rate above background (6461.6 count/s), while GRB 200906B had the lowest (53.7 count/s). Figure 6 shows the distributions of $T_{90}$, peak count rate, and total counts for the four classes of transients:

• (a) Those reported in the past by CZTI POC,

• (b) transients reported by POC which were also found by CIFT,

• (c) CIFT-detected transients reported by other instruments, and

• (d) new CIFT discoveries.

We observe that all four classes have similar distributions of $T_{90}$. A notable difference is seen in the total counts: transients with higher number of total CZT counts tend to be easily detected in regular triggered and ML searches. Also, GRBs with low peak count rates are more likely to be found in triggered searches undertaken by the POC but missed by CIFT. Note that although the three classes “POC-GRBs”, “Known GRBs”, and “New Discovered” are mutually exclusive, the distributions overlap well at the faint end of the distribution.

We find that about 10% of all GRBs detected by CZTI are short GRBs, and the fraction remains the same for the 87 new bursts discovered with CIFT. The fraction of short GRBs is similar to the values for Swift-BAT (Lien et al. 2016), but smaller than the 26% measured in Fermi (von Kienlin et al. 2020). Here, we note an important caveat that we draw the line between short and long GRBs at the canonical value of $T_{90} = 2$ s, but it is known that this can be different for different instruments and will have to be measured separately for CZTI. As an illustration, if we adapt the Fermi boundary of 6.1 s, we find that about one-third of all CZTI GRBs are short GRBs.

5.5 Transients missed by CIFT

Sixty-five GRBs that were found in regular triggered + ML searches were missed in the blind search with CIFT. Two of the missed GRBs were in AstroSat slew orbits which were skipped while processing, as mentioned in Section 5. We analysed the remaining cases to find the reasons why these were missed. The most common reason for the missed GRBs was that the transients were too faint in terms of their peak count rates. For instance, Fig. 7 shows the multi-quadrant, multi-band light curves for GRB 190605A. Visually, it is clear that the GRB is only weakly detected in all three search bands in CZTI data. In order to quantify this further, we calculated the count rates that would have been necessary to flag a data point as an outlier in the peak maps for this orbit. These rates for the CS method are shown with dashed lines, while the 5-$\sigma$ rates for NS are shown with dotted lines. It is clearly seen that the transient is well below these rates.

Figure 7

Diagnostic light curves for GRB 190605A. Top panel: 20–50 keV light curves for all four CZTI quadrants. The shaded green region denotes the GRB. Dashed and dotted lines denote the outlier threshold for CS and NS methods respectively, for each of the four quadrants. Middle panel: Same as top panel, but for 50–100 keV. Bottom panel: Same as top panel, but for 100–200 keV. The transient light curve looks similar in all quadrants, but it is too faint to qualify as an outlier in any of the methods.

Such transients are rather easily confirmed by a human scanner inspecting the spectrogram and finding similar patterns in multiple quadrants. For quantitative analysis with say the CS method, the search window for a triggered search is usually set to 100 s, much smaller than the 4156 s window used in blind searches. This results in a lower cutoff rate, and will make more such fainter transients detectable in the current CIFT framework. Similarly, a smaller search window enables lowering the NS threshold from 5-$\sigma$ to 4-$\sigma$ or 3-$\sigma$ thanks to the fewer data points present, thereby increasing the odds of detecting fainter transients.

Table 2 Comparison of the three search algorithms running on CZTI and Veto for different classifications of the candidates identified. The ‘Candidates’ column contains all potential transient candidates identified by our pipeline. The ‘Common with triggered or ML searches’ column contains all GRBs that were originally detected by triggered or ML searches on CZTI data. The ‘Known transients’ column contains all transients that had previously been reported by other instruments but had not been identified in CZTI or Veto data. The column ‘Discoveries’ comprises of all transients that have not been reported by any instrument before. Common events in various methods are shown in Fig. 8.

Figure 8

Statistics of transients detected in CZTI (Panel (a)) and Veto (Panel (b)) detectors. The three bottom rows show the number of transients detected by each method: for instance, the N-sigma method detected 115 transients in CZTI. The bar charts at the top show overlaps between various combinations of methods. We see that of the 263 transients detected by CZTI detectors, 190 were detected by all three methods, while another 58 were detected by both the “cutoff rates” and “Top-N” methods. Among the 297 transients found in the Veto detector, 204 transients were detected by all three methods, 41 were detected only by the “Top-N” method, and 35 were detected by both the “cutoff rates” and “Top-N” methods.

6 Conclusions and future work

CZTI has proven itself to be a sensitive transient detector, but our searches had largely been limited to triggered searches. The ML pipeline (Abraham et al. 2019) was the first major step towards detection of new transients with CZTI. The development of these algorithms, software, and the CIFT interface provide us with a powerful tool to extend our work further. Here, we have demonstrated the utility of this tool with the discovery of 87 new transients that had been missed by previous searches, including 46 transients that had not been detected by any mission to date. This brings the total CZTI tally to 412 GRBs in the first five years of its operation since launch, or about $\sim$83 per year. For comparison, Swift BAT detects $\sim$92 GRBs per year from on-board triggers (Lien et al. 2016), while Fermi GBM detects $\sim$235 GRBs per year (von Kienlin et al. 2020).

The CIFT framework is constantly evolving. It has been designed to make it easy to incorporate new features including search algorithms. We are working on metrics to quantify the statistical significance of a transient, so that we can lower the FPP. We have developed and tested a new search based on Bayesian Blocks (BB; Scargle et al. 2012). We use the astropy.stats.bayesian_blocks module to obtain block representations of de-trended light curves, and search for blocks that are 3-$\sigma$ outliers. These outliers then form the peak maps discussed in Section 4.1, so the BB search can easily be integrated into CIFT as a fourth algorithm. Preliminary testing has shown promising results with significantly lower false positive rates as compared to other algorithms. We will now run the BB search on the full data set.

Searches for fast transients also stand to benefit from other developments in CZTI data processing. New methods for rejecting noise from raw data (Ratheesh et al. 2021) are improving the quality of light curves. These promise to lower the cut-off rates for CS by a factor of a few and will give a proportional boost to the count rate sensitivity of CZTI. Another notable change to be introduced is the non-removal of Veto-tagged events. The default CZTI pipeline attributes coincident events between CZT and Veto detectors to charged particles, and discards them. In case of bright GRBs, large numbers of photons are incident both on CZT and Veto detectors, greatly increasing the chance coincidence rates. Since these are real GRB photons which should not be discarded, future CIFT-based searches will disable Veto-event filtering.

Table 3 The table contains the calculated parameters for all ‘known GRBs’, which are the GRBs that had previously been reported by other instruments but had not been identified in CZTI or Veto data.

Table 4 The table contains the calculated parameters for all ‘Discovered GRBs’, which are all GRBs that have not been reported by any instrument before.

We have also added functionality to undertake specialised searches for X-ray counterparts to Fast Radio Bursts (FRBs) and Gravitational Wave (GW) sources. For instance, the magnetar source SGR 1935+2154 became active in early 2020, creating a series of bursts including one coincident with Fast Radio Burst (Li et al. 2020). We used the CIFT interface to efficiently search CZTI data for any bursts from this source. A first blind search was conducted with the default thresholds and we found three bursts, coincident with times reported by other instruments. We then lowered the search thresholds and found an additional four bursts, corresponding to those reported by other missions. We are in the process of analysing properties of these CZTI-detected bursts, and the results will be reported separately (Raman et al., in prep.). We have also incorporated the ability to process GW localisation maps to calculate direction-dependent sensitivity. These features will streamline and boost the effort to search for X-ray counterparts to GW sources from the third observing run of advanced gravitational wave detectors (Abbott et al. 2020).