Introduction

Globally, the incidence of meningitis has increased from 2.50 million (2.19–2.91) in 1990 to 2.82 million (2.46–3.31) in the year 2016 [1]. In the USA from 2000 to 2010, the incidence of encephalitis was 7.3 encephalitis hospitalizations per 100,000 population, with the highest hospitalization rates among those < 1 year and > 65 years of age [2]. The estimated mortality due to meningitis and encephalitis are 288,000 and 92,400 deaths every year, respectively, with the major burden among low- and middle-income countries (LMICs) [3]. However, the etiology in both meningitis and encephalitis syndromes is unidentified in > 50% of the hospitalized patients [4]. The early diagnosis of bacterial meningoencephalitis (BM/E) is difficult, and the delay in diagnosis can cause grave complications leading to neurological impairment or death [5]. The high disease burden and future neurological sequelae demand the development of diagnostics which can guide the treatment and lead to better outcome and prognosis.

Diagnosis by conventional microbiological methods like gram stain and cerebrospinal fluid (CSF) culture is challenging and limited by the low bacterial content in the CSF sample [6]. The clinical features of BM/E are overlapping with fungal or viral meningitis. Therefore, early identification of the pathogen can reduce mortality and disability from BM/E and improve the patient’s prognosis. Given the limited amount of CSF samples and long turnaround times, only a fraction of diagnosis is possible. The next-generation sequencing (NGS) platform has the potential to facilitate the early identification of organisms by rapid diagnosis. In cases of unexplained BM/E, the metagenomic NGS (mNGS) offers an advantage over conventional methods, especially when rare pathogens  is implicated or the patient is on antibiotics. Recently, few studies have shown better diagnostic efficacy of mNGS in comparison to conventional methods [7, 8]. However, a systematic review and meta-analysis of the diagnostic accuracy of mNGS for BM/E have not been performed. Therefore, this meta-analysis was conducted to evaluate and compare the diagnostic efficacy of mNGS for the diagnosis of BM/E using CSF specimens versus a composite reference standard (CRS).

Methods

Design and registration

This study was conducted according to Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines (Fig. 1), and the study protocol was registered in the International Prospective Register of Systematic Reviews (PROSPERO) (Registration number: CRD42021240842) [9].

Fig. 1
figure 1

Flowchart of literature retrieval

Full size image

Information sources

The electronic databases (Embase, PubMed, and Web of Science) were searched up to 15 June 2021. The studies which assessed the sensitivity and specificity of mNGS for BM/E were included.

Search strategy

RK and AA conducted the search strategies. The search strategy is as follows:

(Meningitis OR Meningitides OR Pachymeningitis OR Pachymeningitides OR bacterial meningitis OR Arachnoiditis Meningitis OR Escherichia coli Meningitis OR Haemophilus Meningitis OR Listeria Meningitis OR Meningococcal + Meningitis OR Pneumococcal Tuberculosis OR Meningeal Meningitis OR Fungal Meningitis OR Central Nervous System) AND (mNGS OR Metagenomic Next Generation Sequencing OR Next Generation Sequencing OR High Throughput Nucleotide Sequencing OR Sequencing, Next-Generation OR IIlumina Sequencing OR Ion Torrent Sequencing) AND (Cerebrospinal Fluid OR Cerebrospinal Fluids OR Fluid,Cerebrospinal OR Fluids,Cerebrospinal OR CerebroSpinal Fluid OR CerebroSpinal Fluids OR Fluid,CerebroSpinal OR Fluids,CerebroSpinal OR Spinal Fluid,Cerebro OR Spinal Fluids,Cerebro).

Eligibility criteria

Type of study

Studies such as cohort, case–control, prospective, or retrospective studies that assessed the diagnostic efficacy of mNGS were included. However, conference abstracts case reports or articles in languages other than English and studies with < 10 specimens were excluded. We excluded studies on chronic meningitis who had undergone previous neurological procedures or shunt procedures/who had previous brain tumors or cranial or spinal malformation/abnormality. Studies with confirmed diagnosis of tuberculous meningitis and fungal or viral meningitis were excluded.

Patients

Children, adults, and older age groups with suspected bacterial meningitis/encephalitis were included.

Main outcomes

Diagnostic accuracy with sensitivity and the specificity of the index test, i.e., mNGS.

Reference standards

Combined reference standard (CRS) including clinical signs/symptoms, radiographic features, smear, culture, antigen/antibody detection assays, automated platforms, and 16 s rRNA PCR.

Literature screening and selection

After the primary search, articles were imported into the Mendeley literature management software (version 1.19.8, 2008 Glyph & Cog, LLC). Two investigators (RK and AA) independently reviewed the titles and abstracts, followed by the full text for inclusion. Any discrepancies between the investigators were resolved with a third investigator (MB).

Data extraction

Data from included studies were extracted individually by both the authors. Data including author name, year of publication, country, the income of the study, continent, direction, and type of the study, with other parameters, were included. Individual studies were assessed then and data regarding true positive (TP), false negative (FN), false positive (FP), and true negative (TN) values for the assay were extracted to evaluate the diagnostic accuracy. Any discrepancies between the investigators were resolved with a third investigator (MB).

Quality evaluation

Quality Assessment of Diagnostic Accuracy Studies (QUADAS-2) was used to assess the qualities of all the studies [10]. This tool evaluates four key domains: method of the patient selection, index test, reference standard, and the flow and timing of samples/patients along the study.

Publication bias

A funnel plot was plotted to examine the presence of any publication bias. Also, a leave-one-out study plot was done to assess asymmetry in pooled sensitivity and specificity analysis, respectively. A crosshair plot was plotted to visualize the nature of the confidence intervals of sensitivity and specificity around the individual study.

Data synthesis and statistical analysis

The values of TP, FN, FP, and TN in each study were extracted; combined sensitivity and specificity with a 95% confidence interval (CI) between mNGS and CRS was calculated. The bivariate mixed-effects regression model by Reitsma et al. was done to synthesize data [11]. The forest plots for the sensitivity, specificity, positive and negative likelihood ratio,and diagnostic odd’s ratio were plotted. The area under the curve (AUC) of the summary receiver operating characteristic (SROC) was calculated. To assess the heterogeneity between the reference standard and studies, I2 statistics was used. The absence of heterogeneity was considered when I2 value was 0% and value > 50% indicated substantial heterogeneity [12]. R studio version v. 1.2.5033 with mada, meta module was used to carry out meta-analyses. We also used the statistical packages OpenMeta[Analyst] (cebm.brown.edu/openmeta) to perform the meta-regression by diagnostic random effect model with 95% CI analysis (DerSimoniane-Laird method). A p value of < 0.05 was considered statistically significant.

Results

Characteristics of the studies

We included ten studies that met the inclusion criteria, including three retrospective and seven prospective studies (Table 1). Five studies were from China, four from the USA, and only one from Bangladesh. Six studies were performed across the Asian continent and the four across American continents. Most of the studies were from upper-middle-income countries (n 5⁄10, 781 subjects), followed by high-income countries (n 4⁄10, 314 subjects), and only one study from lower-middle-income countries (n 1⁄10, 91 subjects). The total sample size ranged from 20 to 230 (n 1186). The characteristics of the included studies are described in Table 1.

Table 1 Characteristics of the included studies
Full size table

Study quality

The QUADAS-2 score of the included studies is depicted in Fig. 2. The 100% and 70% of the studies had a low risk of bias for index test and flow and timing of the studies, respectively, while high risk of bias was observed for both patient selection criteria and reference standard in 70% studies. In concern regarding applicability, 100% of the studies had standard reference of comparison. However, 80% studies had high proportion of concerns regarding applicability in regard to patient selection and index test.

Fig. 2
figure 2

QUADAS-2 quality graphs are presented as percentages across the included studies using a composite reference standard (CRS)

Full size image

Diagnostic accuracy of mNGS for BM/E

The sensitivity of mNGS for diagnosis of BM/E from CSF samples ranged from 33 (95% CI: 13–62) to 98% (95% CI: 76–99). The specificity of mNGS ranged from 67 (95% CI: 55–78) to 98% (95% CI: 95–99). The pooled sensitivity and specificity was 73.8% (95% CI: 59–84; I2 = 50%, p = 0.035) and 93.3% (95% CI: 87–96; I2 = 83%, p < 0.001), respectively (Fig. 3A, B). The estimated AUC (area under curve) by hierarchical summary receiver operating characteristic (HSROC) of the studies being analyzed was 0.912. The partial AUC (restricted to observed FPRs and normalized) was 0.806 (Fig. 4).

Fig. 3
figure 3

A, B Forest plot for the sensitivity and specificity of mNGS for the diagnosis of BM/E

Full size image
Fig. 4
figure 4

Hierarchical summary receiver operating characteristic (HSROC) estimating a summary curve from studies that have used different thresholds. The open circle represents individual study in the meta-analysis. The regression line (curve) summarizes the overall diagnostic accuracy

Full size image

The negative likelihood ratio (NLR) of mNGS ranged from 0.019 (95% CI: 0.006–0.065) to 0.677 (95% CI: 0.171–2.688). Furthermore, the combined NLR of mNGS was 0.192 (95% CI: 0.097–0.382), and the I2 value was 84% (p < 0.001) (Supplementary Fig. 1). The positive likelihood ratio (PLR) of mNGS ranged from 1.8 (95% CI: 1.1–2.8) to 30.6 (95% CI: 1.9–476). Furthermore, the combined PLR of mNGS was 11.2 (95% CI: 5.6–22.5), and the I2 value was 84% (p < 0.001) (Supplementary Fig. 2). The diagnostic odds ratio (DOR) of NGS ranged from 3.07 (95% CI: 1.33–7.09) to 1346 (95% CI: 62.4–28999.9). The combined DOR of mNGS was 52.83 (95% CI: 15.9–174.6), and the I2 value was 77% (p < 0.001) (Supplementary Fig. 3).

Publication bias

The funnel plot revealed significant publication bias and identified three outlier studies (Fig. 5). The leave-one-out sensitivity analysis revealed that the study by Wilson et al., Zhang et al., and Qian et al. had a substantial influence on the overall sensitivity estimates, and on exclusion of each study, the sensitivity was found to be 76.4% (95% CI: 0.649–0.850), 75.2% (95% CI: 0.574–0.872), and 77.4% (95% CI: 0.613–0.881), respectively [8, 13, 14]. The specificity was highly influenced by two studies, Qian et al. and Ge et al., and on exclusion of each study, the specificity was found to be 94.3% (95% CI: 0.917–0.962) and 93.9% (95% CI: 0.874–0.972) (Supplementary Fig. 4) [14, 15]. A cumulative meta-analysis also revealed overall estimate changes in sensitivity and specificity due to Wilson et al., Zhang et al., and Qian et al. as each study is added to the pool (Supplementary Figs. 5 and 6) [8, 13, 14]. The crosshair graph revealed one outlier study which mostly affected the sensitivity and specificity (Fig. 6).

Fig. 5
figure 5

Funnel plot showing the publication bias in studies reporting sensitivity and specificity in mNGS. The horizontal and vertical axes represent effective sample size (ESS) and its standard error, respectively. The red line represent the Egger's regression line and slope of the line in the funnel plot is an indication for the publication bias

Full size image
Fig. 6
figure 6

The crosshair graph showing the bivariate relationship and the degree of heterogeneity between sensitivity and the rate of false positives. It reflects the results of individual studies in the ROC space; the vertical axis and horizontal axis of the confidence intervals represents sensitivity and specificity, respectively

Full size image

The heterogeneity in all studies was explored using subgroup and meta-regression analysis on the predefined sub-groups for the different types of studies, the direction of study, income of the countries, the continent of the studies, patient diagnosis, the platform for sequencing, and mNGS data analysis pipeline used. However, a significant difference in sensitivity and specificity was observed in all the subgroups like country and continent where the study was performed, type, direction and population of study, and platform used for the metagenomics (Supplementary Table 1).

The meta-regression was performed to confirm the association of pooled sensitivity and specificity in various categories (Table 2). The meta-regression analysis demonstrated that the different types of studies (single-center vs. multi-center) had an effect on the specificity of mNGS for BM/E compared with CRS (90% vs. 96%, meta-regression p < 0.05). Meta-regression analysis demonstrated that the different types of studies (cohort vs. case–control) had effect on the specificity of mNGS (91.4% vs. 97.2%, meta-regression p < 0.01) (Table 2). Also, the patient population (adult vs pediatric vs. both adults and pediatric) had a significant effect on the specificity of mNGS (meta-regression p > 0.009). The sensitivity was affected by direction of the study (prospective vs. retrospective) (68.4% vs. 88.1%, meta-regression p = 0.03).

Table 2 Multivariate logistic meta-regression analysis (random-effects model) showing the factors associated with sensitivity and specificity in diagnosis of bacterial meningoencephalitis by mNGS
Full size table

Discussion

We analyzed ten studies evaluating the performance of mNGS for the diagnosis of bacterial CNS infections from CSF. The pooled sensitivity and specificity of mNGS for BM/E diagnosis were 73.8% (95% CI: 59–84) and 93.3% (95% CI: 87–96), respectively. The leave-one-out analysis revealed the maximum sensitivity and specificity as 77.4% and 94.3%, respectively. The study by Qian et al. contributed to maximum heterogeneity as the patient selection criteria was limited to a cohort of patients with external ventricular and lumbar drainage-associated meningitis [14]. However, the high AUC of the SROC (0.912) indicates that mNGS has excellent diagnostic performance for BM/E and demonstrated the feasibility of the testing CSF samples with mNGS. The summary diagnostic accuracy estimates derived from the current study are largely similar to observations from a recent larger meta-analysis evaluating all-cause meningitis (bacterial, viral, fungal) detection in CSF. A comparable pooled sensitivity and specificity (n = 133) of 75% (54–89%) and 96% (72–100%), respectively, were observed [7].

BM/E is associated with high morbidity and mortality [1]. Although conventional bacterial cultures are set as the gold standard for diagnosis of BM/E, they are not able to identify all infectious etiological agents [5]. These methods are limited by the volume of sample and growth in culture media resulting in false negativity. Also, the sensitivity of culture decreases, if CSF is collected after antibiotics initiation [6]. In culture-negative cases, the utility of targeted 16 s rRNA sequencing has been described previously. However, the inability to detect multiple organisms in specimens and lack of sensitivity are a few limitations [16, 17]. A more recent commercially available platform like BioFire® and FilmArray® has high sensitivity and specificity of 90% and 97%, respectively, for the detection of community-acquired meningoencephalitis [26]. However, this platform is limited by the number of organisms it can detect. Hence, a technique which offers unbiased and wider detection of pathogens in the CSF is required. The advancements in next-generation sequencing (NGS) have revolutionized biological science since 2005. NGS platform has the potential to discover microorganisms among samples with trace amounts of pathogens. Metagenomic next-generation sequencing allows us to detect multiple pathogens in a single test. In a prospective study, Wilson et al. evaluated mNGS for clinical utility in patients presenting with neurological infection [8]. Among 204 patients with suspected CNS infections, mNGS was able to identify infectious agents in 22% of patients that were not identified by conventional methods [8]. These observations demonstrate the potential usefulness of mNGS in identifying infectious agent in patients who remain etiologically undiagnosed.

The utility of mNGS has also been observed in cases with suspected BM/E where prior empirical antibiotics treatment has been given. In these cases, the DNA/RNA of the pathogen can survive in CSF for a longer period. The mNGS technology is less affected than culture in these cases [13, 18]. This widens the diagnostic spectrum of mNGS, making it more suitable than culture, especially in neonates who are treated widely by empirical antibiotics. We observed higher positivity by mNGS even though the time from CSF collection to processing was longer for mNGS in comparison to the culture. Also, many patients received antibiotics prior to mNGS. In a prospective study, where patients were empirically treated with antibiotics, mNGS significantly detected more pathogens compared to the conventional methods [8]. Similar findings were observed by Ge et al., where higher diagnostic yield was observed by mNGS than conventional methods (19.8% vs. 4.95%) [19]. The above results support the advantage of mNGS especially in undiagnosed/previously treated patients with antibiotics [20].

The meta-regression analysis revealed that multicentric studies, case series, and studies that included both adult and pediatric populations had lower sensitivity and specificity than single-center, cohort, and studies performed in either adult or pediatric population. The probable reason for this could be the difference in patient selection. Hence, a defined cohort of population with strict patient selection criteria who present with the classical signs/symptoms of meningitis/encephalitis should be included to analyze the benefits of mNGS over CRS. Furthermore, the impact of the above factors on diagnostic accuracy needs to be confirmed by future larger population studies.

The major limitation of mNGS for infectious disease diagnosis is the background interference, especially from the host DNA [21]. This is most evident in cases of purulent meningitis with a high leukocyte count in CSF. Despite this limitation, in a recent study, mNGS was still able to detect bacterial pathogens in 63% of cases with high host backgrounds [21]. Various sequencing methods with short-read techniques like Illumina HiSeq 4000/ BGI/MGISEQ and long-read technique like PromethION/MinION are available. We observed the highest analytical sensitivity and specificity in studies using Ion Torrent followed by BGISEQ and the lowest for Illumina technology for the identification of the bacterial pathogens. The assessment of reproducibility, accuracy, and utility of these sequencing platforms remains an ongoing challenge. In a previous study comparing Illumina, MiSeq, and Ion Torrent for bacterial community profiling, higher error rates with the Ion Torrent platform were observed [22]. Quail et al. also observed that the data generated using the Illumina platform had lower raw error rates than the Ion Torrent PGM platform (< 0.4% vs. ~ 1.8%) [23]. In the current meta-analysis, only one study used the Ion Torrent platform in a lower number of samples. This could have led to higher sensitivity in comparison to other platforms. Among BGISEQ and Illumina platforms, both data types were very efficient; however, BGISEQ has lower optical duplicates and unmapped read rates than Illumina technology [24]. While Illumina provided the most consistent and highest genome coverage, BGI/MGISEQ provided less rates of sequencing error. We also observed better sensitivity in studies using BGISEQ than various Illumina technique. This study demonstrates the potential for differential bias in outcome resulting from the choice of sequencing alone. However, the impact of these factors on the diagnostic performance of mNGS requires further validation by large prospective studies.

This meta-analysis had several limitations. The number of included studies was limited, and few studies were not able to distinguish meningitis/encephalitis data, which might have created some bias in the diagnostic accuracy. Additionally, studies on the diagnosis of pathogens other than bacterial were not included. The CRS varied across the studies and differences in the reference might have led to changes in patient classification. However, the availability of a large sample size was the major strength of the current analysis, allowing us to provide a vigorous comparison of the diagnostic accuracy estimates. Also, the subgroup analysis and meta-regression analyses revealed some factors such as study population (adults or children), study design, and method of data analysis that attributed to the heterogeneity.

Currently, the utility of mNGS lies as an adjunct, when the routine conventional methods are unable to identify the causative organism [25]. The turn around time and high cost are the major limiting factors. However, the local implementation of mNGS in tertiary care institutes can improve the turn around time. Haston et al, also implemented this technique prospectively and reported results in 3 days, which is comparable to time taken for culture results.

Conclusions

The current analysis revealed that the diagnostic accuracy of mNGS for BM/E was moderate. The AUC indicated a very good diagnostic efficacy. Our analysis revealed the clinical utility of mNGS, especially in cases of undiagnosed BM/E. This approach can guide for identification of pathogens and subsequently accelerate patient management.