Introduction

Shared decision-making has become a cornerstone of practice in neurocritical care [1, 2] and involves using information about a patient’s estimated chance of recovery to make decisions about goals and the trajectory of care. Given that patients with serious neurologic illness are often unable to communicate, family members are frequently tasked with participating in high-stakes treatment decisions on behalf of the patient. To this end, families integrate the prognostic information provided by the physician or medical team with the patient’s presumed goals of care and their own belief system [3, 4]. Although formal disclosure of prognostic information typically falls to the treating physician, for hospitalized patients the bedside nurse is often also influential [5, 6]. Nurses interface with the family more frequently at the bedside [7] and often function as a key conduit of information between clinical team and family.

For patients with severe acute brain injury (SABI), a group of neurologically devastating conditions that includes stroke, traumatic brain injury, and hypoxic ischemic encephalopathy after cardiac arrest, prognostication can be particularly challenging. In addition, after SABI, functional outcomes may be as pertinent as survival because of the impact on quality of life, and functional outcomes are challenging to predict [8]. The general critical care literature suggests that disagreement in the predictions of critically ill patients’ survival is frequent between physicians and families [9] and between physicians and nurses [10, 11]. Physicians may be more accurate than families in predicting long-term survival [11], whereas nurses may be more accurate in predicting survival to hospital discharge [10]. In patients with intraparenchymal hemorrhage, physicians were more accurate than formal prognostic scales and similarly accurate to nurses when predicting 3-month functional recovery [12]. We recently showed that prognostic disagreement of 20% or more between physicians and families regarding a patient’s recovery to independence after SABI is also common and that disagreement is associated with certain patient and family characteristics [13]. The accuracy or degree of agreement between family and physician predictions and between nurse and physician predictions for patients with SABI have not been evaluated. Prognostic agreement may be particularly important for the SABI population, as many surviving patients face long-term disability, and the predicted extent of these disabilities may inform treatment decisions.

The objective of this study was to evaluate the agreement between 6-month prognostic estimates by families, physicians, and nurses of patients with SABI and to measure the prognostic accuracy of each group.

Methods

Design

This prospective cohort study was conducted from January 2018 to July 2020 in the neurosciences and medical/cardiac intensive care units (ICUs) of a comprehensive stroke and level I trauma center in the Pacific Northwest, United States, and has been previously described [13]. Data collection involved in-person surveys of families, physicians, and nurses during the acute hospitalization, review of the electronic health record (EHR), and 6-month follow-up surveys of patients and families. Data were managed using REDCap electronic data capture tools [14] hosted by the Institute of Translational Health Sciences.

Participants and Enrollment

We identified consecutive patients admitted to the ICU with a diagnosis of SABI, defined as stroke, traumatic brain injury, or hypoxic ischemic encephalopathy after cardiac arrest, and a Glasgow Coma Scale (GCS) of ≤ 12 after hospital day 2. Eligibility also required having an English-speaking family member available at the bedside or by phone. With permission from the attending physician of the primary medical team, the patient’s family was invited to participate. If multiple family members were interested, the one that took chief responsibility for decision-making was asked to complete the survey. Given that patients were unable to consent to enrollment themselves, the durable power of attorney or legal next of kin was asked to consent to the review of their medical records on patients’ behalf. The University of Washington Institutional Review Board approved this study (STUDY 00,003,393).

Data Collection

On the day of enrollment, we asked the patient’s family member, the attending physician of the patient’s primary medical team, and the patient’s bedside nurse, separately, to predict the patient’s likelihood of recovering to independence within 6 months. The response option was a visual analog scale from 0 to 100% and scored from 0 to 100 on a continuous scale. Participants were approached early in the hospitalization (within the first 2 weeks), regardless of whether a formal family meeting had occurred. Discordance in this early time frame is important to capture, as numerous consequential treatment decisions for patients with SABI are made within the first week of presentation [15]. Additionally, although it is the standard of care at our institution for physicians to provide every family with a medical update at least once within the first 3 days, these meetings are inconsistently documented in the EHR. Patient characteristics, including age, sex, race, ethnicity, specific diagnosis, and GCS, were collected from the EHR. Family member characteristics were self-reported on the day of enrollment and included age, sex, race, ethnicity, relationship to patient, and level of education.

Six months following enrollment, family members were sent a follow-up survey that included a question about the patient’s current function using the modified Rankin Scale (mRS). We defined independence as an mRS ≤ 3 (moderate disability; requiring some help, but able to walk without assistance), consistent with prior studies evaluating recovery following SABI [16, 17]. If the survey was not returned, we assessed patient mRS at follow-up clinical visits via medical record review. To ensure consistency between mRS determined by family members and mRS determined by record review, 15% of family-reported mRS values were corroborated via record review.

Prognostic Agreement

Agreement regarding prognosis for each pair of participant groups (family-physician; family-nurse; physician-nurse) was evaluated in three different ways: First, we calculated the intraclass correlation coefficient (ICC) for each pair of groups by using a two-way random-effects model with absolute agreement [18]. Second, we measured agreement by plotting the difference between paired predictions for each patient against their means by using Bland–Altman plots [19]. We used paired Student’s t-tests to compare mean predictions by group [20]. Third, we measured the prevalence of concordance for each of the three pairs of groups. We defined concordance as an absolute difference of less than 20 percentage points between the prognostic predictions of participants of two different groups for the same patient. This difference is consistent with previous studies evaluating prognostic concordance in the ICU [9, 13] and aligns with research demonstrating that patients are less likely to pursue life-sustaining treatment when prognosis decreases by 20% [21].

Prognostic Accuracy

Accuracy regarding prognosis was measured for each participant group by generating receiver operating characteristic (ROC) curves and calculating the area under the curve (C statistic) [22]. To create the ROC curves, we first calculated the sensitivity of each prediction (0 to 100) in detecting recovery to independence. Sensitivity was then graphed as a function of false positivity rate (1 − specificity) for that same probability in detecting lack of recovery to independence. DeLong’s test was used for comparing C statistics for each pair of participant groups [23], with a p value of 0.05 used to indicate statistical significance.

The prediction accuracy for each of the three groups was then stratified by GCS categories and concordance with the other groups. GCS strata were defined as a GCS of 3–5, 6–8, and 9–12 to reflect “high,” “medium,” and “low” injury severity, respectively. Concordance strata were defined for each participant group on the basis of the presence or absence of concordance (absolute difference between predictions < 20%) with each of the other two groups. ROC curves and C statistics were then calculated for each of the strata. Statistical comparisons were not made across strata because of the exploratory nature of the measurements.

Accuracy analyses were repeated by using a subset of the data that excluded patients who were transitioned to comfort measures only (CMO) during their hospitalization. This was done to explore the possibility that participants may advocate for CMO in light of a perceived poor prognosis and thereby increase their own prediction accuracy.

Results

During the enrollment period, we identified 250 eligible patients. For 28 patients (11.2%) the family declined participation, resulting in 222 patients enrolled in the study. The median day of enrollment was hospital day 4 (interquartile range [IQR] 3–6). Forty-five physicians and 103 nurses participated in the study. Physicians participating in the study completed surveys for a median of 3 patients (IQR 1–5) and nurses for a median of 1 (IQR 1–2). Prognostic predictions were unavailable from family members for 24 patients, from physicians for 2, and from nurses for 13, with some overlap, resulting in 187 patients for whom all questions regarding prognostic predictions were answered. These 187 patients were, on average, 58 years old (standard deviation 19), 49% were women, and 76% were White. Family members for these patients were, on average, 50 years old (standard deviation 16), 65% were women, and 72% were White, with no notable differences between the enrolled cohort and the cohort with complete prediction data (see Table 1). Among these patients, 177 (95%) had mRS data at 6 months, allowing for calculations of prognostic accuracy. Data for mRS were gathered by patient or family survey for most patients (n = 144, 81%). For those who did not complete a follow-up survey, we were able to collect follow-up mRS from return clinic visits using the medical record for another 33 patients (19% of those with 6-month mRS). To confirm consistency between family survey and chart abstraction, we compared mRS by chart review with that from the family survey in 22 patients (15%), with no discrepancies noted.

Table 1 Characteristics for 187 patients and families with complete prediction data (included in analyses) and 35 with missing data (excluded from analyses)
Full size table

Prognostic agreement

The ICC for physician-nurse predictions was the highest at 0.66 (95% confidence interval [CI] 0.55–0.75), in which 1 indicates perfect correlation and 0 indicates no correlation. For family-physician predictions, the ICC was 0.49 (95% CI 0.05–0.70) and for family-nurse predictions 0.40 (95% CI − 0.06 to 0.64). The Bland–Altman plots (Fig. 1) demonstrate a bias toward more optimistic family predictions, compared with physician predictions (Fig. 1a), with an absolute difference in means of 23.5 (p < 0.001). A similar bias was evident toward more optimistic family predictions relative to nurse predictions (Fig. 1b), with an absolute mean difference of 25.4 (p < 0.001). No significant bias was demonstrated between physician and nurse predictions (difference 1.9, p = 0.38). The physician-nurse plot indicated particularly high agreement when predictions are pessimistic, demonstrated by smaller differences in predictions when the mean of the two predictions is lower (Fig. 1c). The prevalence of concordance in prognosis predictions was highest for physician-nurse pairs, with 105 of 187 pairs (56.2%) meeting criteria for concordance (absolute difference between predictions less than 20%). For family-physician predictions, 74 pairs (39.6%) were concordant. Family-nurse pairs had the lowest prevalence of concordance with 56 pairs (30.0%) meeting criteria for concordance.

Fig. 1
figure 1

Bland–Altman plots for prediction agreement. Agreement between physicians and nurses appears higher when predictions are pessimistic (c, far left)

Full size image

Prognostic accuracy

Of the 177 patients for whom we obtained mRS at 6 months, 44 (25%) had a mRS of ≤ 3, which met our prespecified criteria for independence. ROC curves by group demonstrate highest prognostic accuracy among physicians (C statistic = 0.82; 95% CI 0.75–0.88; Fig. 2). The C statistic for nurses was 0.76 (95% CI 0.67–0.84), which was not significantly different from that of physicians (p = 0.18). For families, the C statistic was 0.65 (95% CI 0.56–0.73), which was significantly more accurate than chance alone (C statistic = 0.5), and significantly less accurate than both physicians (p < 0.01) and nurses (p = 0.05).

Fig. 2
figure 2

Receiver operating characteristic curves for accuracy in predicting recovery to independence at 6 months. CI, confidence interval

Full size image

Family, physician, and nurse prognostic accuracy did not demonstrate any clear pattern across GCS strata, with overlapping confidence intervals between the groups (Table 2; Supplemental Fig. 1). Prognostic accuracy was higher in all groups when concordance was present (Table 3). Specifically, family predictions were more accurate when concordance with either physicians or nurses was present compared to when discordance was present.

Table 2 C statistics for each participant group by GCS strata
Full size table
Table 3 C statistics for each participant group when concordance with other groups is present
Full size table

Sixty-one patients (34%) in the study were transitioned to CMO during hospitalization. Analyses of the subset of data excluding these patients found a decrease in accuracy for all groups. Comparisons between C statistics for participant groups were unchanged (see Supplemental Fig. 2).

Discussion

A shared understanding between clinicians and families about a patient’s prognosis is fundamental to shared decision-making [24, 25]. Our findings of poor agreement between families, physicians, and nurses may therefore indicate an important target for improving the shared decision-making process. SABI is often characterized by substantial prognostic uncertainty [8, 26], which may play a role in poor agreement between families and physicians. However, a study in a mixed medical-surgical ICU population found similar poor prediction agreement regarding a patient’s chance of survival [9]. The focus in our study on recovery to independence rather than survival is important in the setting of SABI, in which survivors face a range of functional and cognitive deficits and a threat to their personhood, quality of life, and previous personal and professional abilities [8, 27, 28].

Possible adverse effects of poor agreement on shared decision-making are evident from a clinician and a family perspective. For example, clinicians have described a family’s inability to accept their loved one’s poor prognosis as a barrier to communication about goals of care and decision-making [29]. In another study, prognostic disagreement between families and physicians was associated with disagreement about the appropriateness of certain treatment options [30]. Disagreement about treatment appropriateness was also associated with lower family trust in the ICU team [30], which has the potential to further compromise shared decision-making.

Reasons for poor agreement between families and clinicians include family misunderstanding of clinician predictions, potentially due to poor or ineffective clinician communication, or family interest in believing a different, typically more optimistic, prognosis [10, 13]. These possible causes have implications for how poor agreement could be remedied, for although misunderstanding may be addressed by improved communication, optimistic belief differences may represent a coping mechanism and require different approaches [31, 32].

Despite the relatively high prognostic agreement between physicians and nurses relative to the other paired groups, concordance was present less than 60% of the time. Differences in prognostic perceptions between physicians and nurses may, in turn, limit agreement between these clinicians and family members. Given that nurses and physicians provide prognostic information to patients, discrepancies between the messages conveyed by these parties could result in additional uncertainty in prognosis for the families. Clarification between physicians and nurses of their own perceptions of prognosis may inform how they communicate prognostic information to families and may improve agreement overall.

Physicians and nurses were significantly more accurate in their predictions than families. Accuracy appeared to be highest for physicians and families when GCS was very low, although statistical comparisons were not performed due to low sample sizes. One possible explanation for increased accuracy when prognosis was perceived to be worse is the phenomenon of the self-fulfilling prophecy [33]. In this case, someone who perceives a poor prognosis may be more likely to advocate for CMO, which typically leads to death and consequently fulfills that prediction [34, 35]. In a sensitivity analysis, we excluded those patients who were transitioned to CMO during the hospitalization and found no significant changes in our results, suggesting that the self-fulfilling prophecy is unlikely to be a key driver of accuracy. This finding is consistent with another study that found increased physician prediction accuracy for patients with low GCS, even after excluding those with early CMO [12]. Outcomes may simply be more evident when illness severity is higher, leading to more accurate prognostication. The finding that physician and nurse accuracy was consistently higher than family accuracy across GCS strata may suggest that the relative accuracy of participant groups was not mediated by illness severity.

Finally, accuracy appeared to be higher for the concordant subgroups, a finding that has also been noted in a mixed medical-surgical ICU regarding survival predictions [10]. Although the interpretation of this finding is limited because of the complex relationship between illness severity, agreement, and accuracy, this finding is notable for its potential implications regarding decision-making. Agreement is an important step in the decisional process, and therefore clinicians and families may benefit from discussing their perceptions of prognosis and reconsidering their assessment if it differs substantially from another’s assessment.

Strengths of this study include the participation of three different groups of stakeholders (families, physicians, and nurses), which provides a multidimensional view of prognostic perceptions, and the study’s longitudinal nature, in which we followed patient outcomes over time. This study also has several important limitations. First, this study was conducted at a single center and may not generalize to other institutions or regions. This limitation may be mitigated by the fact that this hospital is the sole tertiary referral center and level I trauma center in a five-state region. Second, the presence or timing of formal family meetings, which could impact agreement between and accuracy of groups, was not included in this analysis. Measuring agreement within the first week of hospitalization may be biased toward poor agreement due to high prognostic uncertainty and high levels of emotion, missing improved agreement later in the hospital stay; however, we specifically chose to evaluate agreement early on in the hospitalization when many consequential decisions are made. Because the occurrence and content of formal family meetings and other family updates are not consistently documented in the medical record, we did not attempt to assess the influence of a family meeting on agreement and accuracy. Third, it is possible that more than one family member was involved in conversations or decisions about prognosis, and our survey did not explore discordance between family members. Finally, we could not completely account for the fact that physician prognostic accuracy may appear high, in part because they are more likely to recommend withdrawal of life-sustaining treatment if they believe prognosis to be poor. However, sensitivity analysis in which patients who were transitioned early to CMO revealed no important changes in our results.

Conclusions

For patients with SABI, agreement in prognostic predictions between families, physicians, and nurses regarding the likelihood of recovery to independence is poor. Additional research may be helpful in evaluating (1) what impedes or facilitates prognostic agreement between families and clinicians, (2) what are the patient and family outcomes associated with low prognostic agreement, and (3) what specific interventions may increase prognostic agreement.