Keywords

1 Introduction

Predictive analytics are daily used by clinical neuroscientists, mainly for nonclinical purposes. Google algorithms for example pave the way for rapid access to our personal interests and needs. Predictive analytics applied to daily clinical patient care are however less used. Ironically, clinical neuroscientists increasingly report on predictive algorithms specifically developed to support daily clinical care [1, 2]. The availability of electronic health record (EHR) systems and user-friendly statistical packages has fueled model development approaches by many clinical neuroscientists.

The overarching aim of predictive analytics in clinical practice is to improve patient outcomes in terms of quality, safety, and efficiency [3]. Nowadays, predictive analytics have become inevitable because stakeholders (policy makers, funders, and patients themselves) want to participate in decision making on which treatment to choose that provides maximal benefit together with minimal costs and patient burden.

Although it is known that predictive analytics can outperform the predictions made by clinical neuroscientists themselves [4], it has been hard to include predictive analytics in current clinical workflows. The increasing amount of available predictive algorithms induces uncertainty by potential end-users (e.g., clinical neuroscientists) which model to use, if any. Their potentiality is often not recognized by end-users.

Herein, we describe and tabulate advantages and disadvantages of predictive analytics in clinical practice (Table 30.1). We highlight the application of predictive analytics and address potential endeavors that might foster the inclusion of these tools into clinical workflows.

Table 30.1 Several advantages and disadvantages of predictive analytics in clinical practice
Full size table

2 Data Considerations: What to Put into a Predictive Tool?

2.1 Quantity Versus Quality

Large sample sizes are highly desirable when prediction models are generated, especially when highly flexible methods are used [5, 6]. However, the quantity of the available data does not guarantee high quality of the data. This is nicely demonstrated in the Google Flu Trends (GFT) analysis in 2013. Google search-term data were used to predict the seasonal flu. Some predictors identified by the Google algorithm had no (biological) relation with the flu. The GFT prediction was unreliable and a simple model from the Centers for Disease Control and Prevention outperformed the GFT model [7]. Theory-free studies and theory-free driven algorithms are prone to provide biased results since they rely too much on the data.

Hypothesis based studies use subject matter knowledge to mitigate bias, resulting in more quality and structured data sets. These data sets are therefore more tailored to the studied clinical condition. However, such data collection is often manually executed and therefore time consuming, which may result in lower sample sizes.

In 2020, Google published a predictive analytic tool that was trained on a big data set that was of high clinical quality [8]. The predictive analytic tool outperformed clinicians in diagnosing malignancies on radiological studies. Thus, to flourish and reach its potential, predictive analytics need a combination of data quantity and data quality—that may come from different data sources—to aid clinical neuroscientists in understanding and controlling complex conditions in neuroscience such as subarachnoid hemorrhage (SAH) [9].

2.2 Theoretical Construct and Empirical Construct

Predictive analytics in clinical practice are normally based on the results of empirical data. Clinical neuroscientists try to better understand theoretical constructs through empirical data. However, do the study variables (predictors and outcomes) adequately represent the condition that is aimed to be unraveled? For example, do empirical surrogate markers such as health insurance status adequately represent a tested theoretical construct such as social economic status? [10]. Other observational data from a national registry have shown that functional outcome (empirical construct) may not be a valid indicator for quality of care (theoretical construct) when comparing stroke centers [11]. Thus, measurement instruments may not fully capture the theoretical construct and may not be comparable across cases and centra.

The array of data resources that are being used for predictive analytics is increasing. Next to (national) registry data, other data sources like EHRs, open sources (such as meteorological data), and claim data have been used for analyses. Nowadays, neurosurgical procedures and diagnoses are coded in EHRs for billing purposes. These codes can be easily used for predictive analytics. For example, ICD codes have been used to predict spontaneous subarachnoid hemorrhage admissions to evaluate the gut feeling of clinical neuroscientists that SAH admissions appear in clusters [9]. Such data collection (covering a long time period, i.e., data from a decade) would not have been possible so easily with traditional manual data collection by researchers. The rationale for documentation of this kind of data is not for scientific purposes but rather for administrative purposes which may result in several data anomalies and incomplete patient information. First, confounding variables are normally not documented, and clinical outcomes are omitted. Patient frailty, for example, is not routinely assessed and documented, but is a robust predictor of poor surgical outcomes [12, 13]. Second, miscoding of variables may emerge. It has been shown that postoperative complications have been miscoded as comorbidities [14]. Using such data may create bias in effect estimation of predictor-outcome associations and ultimately in prediction. Third, coding behavior varies between hospitals and among health care professionals. If no one codes a SAH, the patient does not have a SAH and will be wrongly excluded from analysis. Fourth, different EHR software is currently being used between hospitals, such as HiX and Epic. Thus, currently, patient data is spread across multiple inter-institutional and intra-institutional data sources. The lack of an integral EHR system makes it hard to include all the relevant data from a patient into a predictive analytic tool.

Clinical empirical data can also be noisy and threaten the theoretical construct studied. For example, cardiopulmonary variables such as pulse oximetry, capnography, and heart rate are prone for artifacts. Blood samples taken from a patient may be hemolytic and hence subjected to artifacts such as falsely elevated potassium levels. A predictive tool is only able to provide sensible predictions if the input is adequate. In general: garbage in, garbage out.

2.3 Analyzing Available Data or Analyzing Clinical Equipoise

Intraarterial nimodipine therapy and norepinephrine infusions for symptomatic vasospasm in patients suffering from aneurysmal subarachnoid hemorrhage are highly predictive for poor functional outcome and patient mortality. Although such a predictive model may be highly accurate, it does not provide an option to reduce the risk for the patient. Such predictive models do not influence clinical decision making by clinical neuroscientists and will not improve clinical outcomes. The model is not able to provide interventions to prevent patients from becoming poor grade patients. In other words, the actionability of such models is low. A much more interesting question is to predict rebleeding prior to cerebral aneurysm treatment (microsurgical clipping or neuroendovascular treatment) when an aneurysmal SAH patient arrives at the hospital at 22:00 hours, because there is equipoise regarding the optimal management. Can we wait for another 12 h to treat the aneurysm at daytime leaving the patient at risk for a rebleeding? Or should we intervene immediately and expose the patient to a probably fatigued and less experienced team? A predictive tool that accurately classifies those patients into high and low rebleeding risks will help the clinical neuroscientist to make informed decisions and will influence patient outcomes accordingly.

Another example: predicting readmissions after brain surgery with data automatically drawn from EHR has become of increasing interest [15]. These noble predictive analyses often overfit the small number of patients however. Furthermore, the local EHR will not notice a readmission in another hospital. The actionability is again supposed to be low, because providing the risk of a readmission in 30 days is unlikely to change the behavior of the clinical neuroscientist. The model might be useful for just informing patients, however. The performance measures of such models is generally low, likely due to the fact that clinical data alone is insufficient and other factors such as social determinants of health are not considered, yet more appropriate for predicting hospital readmission [16].

3 Interpreting the Model’s Output: An Essential Role for the Clinical Neuroscientist

3.1 Clinical and Scientific Competencies

Clinical decision making on new patients currently still involves clinical judgment and personal preferences extrapolated from our previous experiences. In contrast to the number of patients predictive models are exposed to (models are commonly trained on hundreds, thousands or even bigger numbers of patients), the number of patients a clinical neuroscientist is exposed to is relatively small. Therefore, clinical decision making based on our own clinical experience can be moot.

To interpret a model adequately, basic knowledge on quantitative predictive analysis is needed for clinical neuroscientists to understand and integrate probabilistic data in their patient work-up. Predictive analytics using logistic regression for example, will provide a probability of an event to occur. The probability provided will likely be incorrect, because either the patient will undergo the event or not. In clinical practice, a patient cannot be 75% shunt-dependent after aneurysmal SAH after 30 days. The patient will be judged as shunt-dependent and will have a permanent shunt inserted or will be judged as not shunt-dependent. Another important aspect to be aware of is statistical overfitting. Overfitting is a common problem due to complex modeling relative to the effective sample size. Using an overfitted model on new patients may be harmful. Overfitted models likely provide overestimated risks for high-risk patients and underestimated risks for a low-risk patient, which can be observed in a calibration plot. Therefore, clinical neuroscientist should be aware of the model’s performance. Discrimination and calibration are well-known model performance measures. Discrimination refers to the ability of a prediction model to discriminate between patients with and without the event of interest and is quantified using the c-statistic. The c-statistic ranges from 0.5 to 1, where 0.5 means that the prediction model is equivalent to a coin toss and 1 refers to perfect discrimination. Calibration refers to the agreement between predicted and observed outcome and is highly consequential to medical decision making—it has been labeled as the Achilles heel of predictive analytics [17].

Methodological aspects such as study bias should be considered as well. Confounding is a critical aspect in translating results from predictive analytics into clinical decision making. Predictive analytics are often hampered by confounding by indication. Causal inferences can therefore not be drawn. An example in which confounding by indication matters is the use of predictive analytics based on retrospective glioblastoma patient data. Predictive analytics for patient survival often include treatment effects such as extent of surgical resection and type of post-surgical therapy. Drawing conclusions on the effectiveness of therapies should be done cautiously. Exemplifying this, it is likely that glioblastoma patients with a good general condition as reflected in the Karnofsky performance score (KPS) with a relatively good prognosis for survival will get standard post-surgical therapy (radiotherapy plus concomitant and maintenance temozolomide) and that glioblastoma patients with a poor general condition with a worse survival prognosis have a greater probability to receive subparts of standard therapy and/or experimental designs. However, if bias is adequately taken into account, such models can be well used for shared decision making with relatives or patients themselves.

Thus, interpreting results from predictive analytics urge for an adequate risk communication to patients and their relatives across all educational levels, especially in shared decision making situations. This will be a vital new skill that clinical neuroscientists should master in the future, because—at least for now—a computer cannot take over this skill.

Clinical neuroscientists are also at risk of de-skilling of their clinical competencies. Overreliance on predictive analytics may negatively affect the ability of making firm interpretations of signs and symptoms [18]. In addition, it may induce stereotyping of patients and decrease clinical knowledge and self-confidence [18,19,20]. In unforeseen situations, such as the local shutdown of the academic hospital in Düsseldorf in 2020, neurosurgeons and other clinical neuroscientists should be able to provide adequate patient care without the use of modern predictive analytics, which may be difficult for younger professionals as a result of de-skilling [21]. We should be aware of de-skilling due to overreliance which can be controlled by regular reflections of end-users.

3.2 Clinical Neuroscientist’s Vigilance

The imperfect nature from predictive analytics should be considered and is highly consequential. Predictive analytics are dynamic processes. The lifetime of a prediction tool may be limited. It is known that the performance of predictive tools wane over time if the tool is exposed to more data or to new promising prognostic variables.

Another vital aspect to consider is the condition of the investigated patients. For example, the course of the aneurysmal SAH disease may be complicated by meningitis which in a worst-case scenario progresses into a meningitis-sepsis. Clinical neuroscientists may consult predictive analytic tools that are trained to identify (meningitis)-sepsis. Likely, those tools have learned from patients diagnosed with a sepsis [22]. Utilizing such a tool for clinical decision making should be done cautiously in patients without a diagnosed sepsis, since those tools are commonly not trained on patients that are—although they were at risk for a sepsis—prevented from a sepsis due to adequate medical care at the neurointensive care unit.

Sometimes the inclusion or exclusion of interesting variables into a predictive tool can make the model impractical. Recently, a predictive analytic approach for predicting shunt-dependency after aneurysmal SAH showed an impressive performance [23]. The use of prognostic variables that may emerge in the course of the disease, such as delayed cerebral ischemia, make application of the tool by clinical neuroscientists, however, complex. Another example: surgical resection of a recurrent glioblastoma during the course of the disease in glioblastoma patients is difficult to include in a predictive analytic tool because this data is not available at baseline or at the moment the model is intended to be used. However, this may alter survival time. Thus, if a predictive tool is used for prognostication, the clinical neuroscientist should critically evaluate if his/her patient resembles the patients used for model generation.

4 Integrating the Model into the Clinical Workflow: Reporting Is Imperative

4.1 User Trust

Why do we trust our patient interview? Why do we trust our clinical patient examination? Why do we trust the additional investigations—such as laboratory results from our lumbar puncture and radiological results of the MRI scan—of our patient? One of the reasons is that we know they are reliable at most of the time. We trust them, because we observe the glioblastoma in the left temporal lobe as we perform the surgical procedure. We see that our liquor tap is purulent, and that it becomes clearer during therapy with antibiotics. Furthermore, we are able to (re)weight the strength of our observations in the light of the clinical course of the patient. Although the literature provides many reports of predictive analytics that should have promising effects for our daily clinical routine, why don’t we use them regularly in our daily clinical practice? Do we not trust these tools? One of the reasons might be that we are not familiar we these techniques, and probably the lack of technical know-how. Clinicians are commonly not trained in statistics and scientific methodology like epidemiologists and statisticians—understanding the structure of algorithms from machine learning methods can be challenging even for experts, however. End-users remain wary, especially when machine learning algorithms are used, as they cannot directly and exactly see, control, and understand how the patient data is weighted and modeled by the developers in opaque predictive analytic tools. End-users want to know how a predictive tool got the results provided [3].

4.2 Transparency

To make predictive analytics convincing for the clinical neuroscientist, model transparency is imperative. Transparency is key to trust and application of the predictive tool. Transparent reporting according to the Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD) guidelines is needed for transparency in model development and for external validation and impact study attempts [24]. Caveats for clinical use of the model should be clearly explained and readily available. For example, is the model for shared decision making and confounding by indication should be taken into account, then the clinical neuroscientist should be aware of that. Trust in predictive analytics by clinical neuroscientists will further enhance if models are regularly updated as more data becomes available, since patient populations may evolve over time and the half-life of clinical data can be short [25]. Recently, an innovative calibration drift detection system identifying the need for model updating has been proposed [26]. Such reporting systems are important because miscalibration may lead to severely flawed predictions. For example, patients identified by a miscalibrated model as having low risk of postoperative complication may be falsely withdrawn from preventive treatment.

4.3 Safe Use and Regulatory Approval

It is known that clinicians may incorrectly interpret the results of predictive analytics, and many biases may have to be taken into account [6, 27]. Applying a predictive tool to patients not taking into account methodological shortcomings can harm many patients and is unethical. Developers should ideally provide online calculators, apps or desktop applications that possibly can be embedded within EHRs to aid uptake in the clinical workflow together with sufficient detailed background information of the model development including its caveats.

Merely presenting a clinical predictive tool without a clear recommended action will likely not survive in clinical practice. However, a clinical predictive tool with a clear recommendation that disrupts the workflow of a clinical neuroscientist will also not survive. The variables needed for the predictive tool should be easily accessible and being measurable without minimal measurement error [1]. The predictive tools provided should offload the clinical neuroscientists and not load them with additional work. Ideally, clinical neuroscientists should not have to open additional packages next to their EHR to use a predictive tool. Re-entering patient data into a model to obtain individual prognosis estimates should be avoided if these data can be derived directly from the EHR, such as age, patient gender, and KPS.

Clinical neuroscientists need impact studies that show the benefits, harms, and sustainability of the clinical prediction models used. Unfortunately, these studies are clearly underrepresented in the literature. There is an over-emphasis on model development studies and a focus on increasing model performances measures. Model performance measures are likely not convincing enough for end-users; yet the impact of predictive tools on the outcomes—i.e., effectiveness of the model—tracked over time will increase model trust and usability [3]. In addition, a label that certifies a prediction model to be deployed in clinical practice might be a next step to enhance clinical uptake. Attempts to estimate the value of predictive analytics in clinical practice, such as the “number needed to benefit” have been suggested [28]. Regulatory approval endeavors have been underway [22]. Food and Drug Administration (FDA) approval or Conformité Européenne (CE) approval may ultimately help to convince clinical neuroscientists that a particular predictive tool meets clinical quality standards and can be applied safely.

5 Concluding Remarks

In this relatively new era of predictive analytics, clinical neuroscientists play a critical role in outlining the clinical problems the predictive analytics have to solve. In addition, clinical neuroscientists play a critical role in interpreting the output of predictive analytics in light of the clinical scenario of the individual patient. Only clinicians can discuss the results with the patients and activate treatment regimes. Clinical neuroscientists should be therefore ideally trained and skilled on how to integrate a model in their patient work-up. To fully use the potential of predictive analytics, clinical neuroscientists need to understand at the one hand the difference between his/her patient and the ones included in the predictive algorithm, and the available resources that might be considered to intervene in the course of the patients’ disease.

Combining predictive analytics with the knowledge clinical neuroscientists have of the pathophysiology and patient’s preferences will have a positive synergistic effect on individual patient care what neither can do alone. Ultimately, if used sensibly, predictive analytics have the potential to be an additional component in the history taking—clinical examination—additional investigations—(predictive analytics)—diagnosis/treatment plan patient work-up of clinical neuroscientists. It can enhance this clinical process by making better informed decision together with their patients.

To foster the progress of predictive analytics into the clinical workflow of the clinical neuroscientist, (1) the used data sets should be more refined to the clinical scenario studied, (2) predictive analytics should ideally be used to study patients in equipoise regarding optimal management, not to study the available data, and (3) clinical neuroscientists should have knowledge on effective implementation of the designed predictive tools for the right patients.