Introduction

Physicians routinely under-diagnose depression and suicidal ideation [1, 2], substance use [3, 4], poor medication adherence [5–7] and HIV transmission risk behaviors [8, 9], all of which impact long-term clinical outcomes among people living with HIV (PLWH) [6, 10, 11]. Assessment of these conditions and behaviors relies on patient self-report. In routine practice, providers elicit this information from patients at the point-of-care. Improving elicitation and delivery of patient self-reported data may be critical for improving clinical care and outcomes.

Growing recognition of the value of systematic routine collection of patient-reported data in care has led to development and use of self-report assessments, referred to as patient-reported data, measures, or outcomes (PROs) [12]. PROs assess conditions and behaviors that might otherwise be overlooked or are difficult and time-consuming for providers to assess in brief clinical encounters.

Past efforts integrating PRO reports into clinical care have often shown disappointing results [13, 14]. Potential explanations for this include time gaps between PRO completion and provider appointments rather than same-day assessments [15, 16] and measurement of domains providers do not necessarily deem clinically relevant [13]. Recent technical advances can address several concerns, in particular, difficulty collecting data in real time in high-volume clinic settings without causing delays in patient throughput. Touch-screen computer technology now facilitates high completion rates, even by patients with low levels of computer literacy. Computerized PRO platforms permit time-saving skip patterns that minimize patient burden and facilitate real-time delivery of results to providers at the point-of-care [14, 17, 18]. These advances have allowed successful integration of PROs into routine care with minimal disruption to clinic flow [14, 17].

We compared chart documentation from all patients who self-reported depression, inadequate HIV medication adherence, alcohol and substance use, and sexual risk behavior on the PRO assessment in the 8 months before and after we began to routinely deliver PRO reports to providers. We also compared provider documentation and actions taken from the subset of patients with depression or risk behaviors both before and after feedback initiation at any time during the study (not limited to 8-month windows). The purpose of this study was to determine the extent to which a well-integrated, clinically relevant, web-based, touch-screen PRO collection and feedback delivery system influenced provider documentation and actions in the care of PLWH.

Methods

Study Setting

This study was conducted among patients in the University of Washington (UW) HIV Cohort, a longitudinal observational cohort of PLWH who receive primary care in the UW Harborview Medical Center HIV Clinic.

Study Participants

PLWH ≥18 years of age who attended the clinic for a routinely scheduled appointment and completed the PRO assessment were eligible. Patients unable to complete the assessment, such as those with dementia or who did not speak English or Spanish, were excluded, as were those who declined to give informed consent. This study was approved by the UW’s Institutional Review Board. All patients signed written informed consent.

PROs

Patients awaiting routine visits with their providers began completing PRO assessments as part of a research protocol in 2006. We used a web-based, open-source survey software application, developed specifically for PROs (http://cprohealth.org) [18–23]. Patients used tablet PCs with touch-screens. Questions were displayed in large, easy-to-read type with clearly labeled radio buttons to indicate responses. Patients completed the assessment approximately every ~4–6 months at the time of routine appointments.

PRO data were initially intended solely for research and were collected 2 days a week. As implementation of PRO collection was smooth and non-disruptive [18] this was expanded to daily collection. Based on discrepancies between PRO results and clinician documentation from the same visit date showing lower rates of risk behaviors in clinical documentation, clinic leadership supported delivery of PRO reports to clinicians as part of routine care processes beginning January, 2009 [18]. Before initiating delivery of PRO reports to providers, we conducted a brief introduction and training session at the monthly provider meeting and sent out a summarizing email to clinic staff. Since then, assessment completion has automatically generated a brief printed report that is delivered to the provider immediately before the clinic visit with other visit paperwork. More recently, results for domains such as depression are also integrated into the electronic health record meeting a Meaningful Use objective [24]. This approach to PROs builds on the theoretical underpinnings of the Chronic Care Model including a clinical information system, delivery system design, and decision support to improve clinical care [25–28].

Instruments

We selected domains for the PRO assessment based on importance for clinical care and research. Using the Medical Outcomes Trust criteria [29], we considered instrument validity, reliability, responsiveness, efficiency in terms of patient burden, and interpretability when choosing instruments for each selected domain. When possible, we chose instruments with evidence of validity established among PLWH. The assessment contained 50–101 items depending on skip patterns generated by patient responses.

Instrument Scoring

For these analyses, we focused on five clinically important domains. We measured depression symptoms using the nine-item Patient-Health Questionnaire (PHQ-9) from the Primary Care Evaluation of Mental Disorders [30, 31], with scores ≥10 indicating moderate-to-severe depression, considered at-risk. We also conducted sensitivity analyses requiring scores ≥10 and this score had to include elevated results for the items measuring depressed mood or anhedonia [31]. While feedback began January 2009, even when PRO assessments were collected solely for research purposes (prior to January 2009), providers and/or case managers always received an automated notification of suicidal ideation (PHQ-9 item 9).

We measured antiretroviral medication adherence using the four-item Adult AIDS Clinical Trial Group instrument, a visual analogue scale, and a self-rating scale item [32–34]. Inadequate adherence for these analyses was defined as missing ≥1 dose in the prior four days. We measured substance use using the Alcohol, Smoking, and Substance Involvement Screening Test [35, 36]. At-risk substance use was defined as any non-prescribed use of opiates/heroin, crack/cocaine, or methamphetamine/crystal/speed within the prior 3 months. Alcohol use was measured using the Alcohol Use Disorders Identification Test consumption questions [37, 38]. At-risk alcohol use was defined as scores ≥5 for men and ≥4 for women [39]. Sexual risk behavior was measured using a modified version of the HIV Risk Assessment for Positives (HRAP) [40, 41]. We defined sexual risk behavior as being sexual activity with one or more partners in the prior 6 months and reporting using condoms some of the time or never (in contrast to using condoms most of the time or always). The HRAP was not initially included in the assessment, it was added June 2007.

Medical Record Review

We reviewed medical record documentation from provider clinic notes from the same day as the assessment among PLWH self-reporting clinically relevant levels of depression or risk behaviors. We determined whether depression or risk behaviors (i.e., inadequate adherence, substance use, at-risk alcohol use, sexual risk behavior) were identified by the provider and if any related actions were initiated by the provider. Documentation for a domain included in problem lists, discussion in the assessment and plan, or any other provider documentation for that day. Actions included referrals, prescriptions, or documentation of discussions. For example, if a patient had depression, potential provider actions could include a new prescription or dose change of anti-depressant medication, referral to psychiatry, referral to the case manager or health educator, or discussion with the patient regarding depression and available resources.

For patients with inadequate adherence, in addition to noting any acknowledgement of the issue in provider notes, we also identified notes in which providers inaccurately documented very good medication adherence (examples included “missed no doses,” “>95% adherence,” and “perfect adherence”). All medical record reviews were performed by chart reviewers blinded to study goals and the PRO results for each patient.

Analyses

Our primary analyses for each domain compared same-day problem identification and actions documented by providers in the 8 months before versus after the start of feedback among patients with depression or risk behaviors using chi squared tests. We limited the time-windows for this analysis to minimize potential impact of changes in care guidelines or other period effects. This analysis focused on assessment results by visits before and after feedback initiation and did not require an individual patient to have an at-risk assessment in both time periods.

We conducted additional analyses for each domain that included patients with depression or risk behaviors both prior to and after feedback initiation. These secondary analyses differed from the primary analyses as they were within-person analyses not limited to 8-month windows. However, patients had to endorse depression or risk behaviors both before and after feedback initiation. To account for repeated measures and the matched quality of the data, we used generalized estimating equation logistic models to verify that the statistical tests were robust to the issues of matching and repeated measures. We present the measures on the percentage scale, due to the easy clinical interpretability and consistency with other results. We compared provider-documented problem identification or actions before and after feedback initiation.

We repeated both analyses among the subset of patients with moderate-to-severe depression excluding those who indicated suicidal ideation, as this always resulted in automated notification even before feedback was routinely integrated. We conducted additional sensitivity analyses examining 6 and 10-month periods prior to and after initiation of provider feedback.

Finally, to address the possibility that changes in documentation were due to temporal trends or other period effects unrelated to the clinical assessment, we assessed documentation for four outcomes not included as part of the clinical assessment (diabetes, hypertension, pneumococcal vaccination, and hepatitis C virus screening) and compared rates of documentation in the 8 months before and 8 months after initiating provider feedback among a random selection of 300 individuals who had completed the assessment.

Results

During the study period, 1 January 2006–15 October 2010, the assessment was completed 2289 times by 1083 PLWH, of these, 722 were completed by 381 PLWH in the 8 months before and after feedback initiation. Because of the initial ramp-up period during implementation, more assessments were completed in later than in earlier years. Refusal rates were ~1%. The mean number of assessments per patient was 1.9 (median 2, range 1–10). Providers received feedback from 99% of PROs completed after the initiation of provider feedback. Completion rates were high with minimal missing data. All 9 depression items were completed in 2150 assessments (94%) and adherence data were available for 1839 assessments (99% of patients receiving HIV medications). Patients not receiving HIV medications were not asked to complete adherence items. Information regarding alcohol use was available from 2207 (96%) assessments, for substance use from 2227 (97%) assessments and for sexual risk behavior from 1778 (78%) assessments (note sexual risk behavior was added in 2007).

The mean age at initial assessment was 43 (SD 9), 85% were men, 60% were non-Hispanic White, 21% were African-American, and 12% were Hispanic. Demographic and clinical characteristics of study patients were similar to those of all patients receiving care at the clinic during the study period (data not shown).

Provider Identification and Action in the 8 Months Before and After PRO Feedback

These analyses were based on at-risk assessments in the 8 months before or after feedback initiation from n = 722 assessments.

Depression

Patients reported moderate-to-severe depression symptoms in 317 assessments in the 8-month windows before and after provider feedback initiation. Prior to feedback, providers acknowledged depression in 74% (95% CI 62–85) of same-day clinic documentation versus 87% (95% CI 82–91) after feedback (p = 0.02; Fig. 1a) among patients who reported depression symptoms on the assessment. Providers took action (e.g., prescription for antidepressant medication, referral to mental health treatment) in response to moderate-to-severe depressive symptoms in 58% (95% CI 45–71) of patients prior to feedback, versus 66% (95% CI 60–72) after feedback (p = 0.3; Fig. 1b). Findings were similar in sensitivity analyses of patients with moderate-to-severe depression excluding those with suicidal ideation (documentation 70 vs. 85%, p = 0.012, provider action 51 vs. 64%, p = 0.09). Similar findings were also found in those with moderate-to-severe depression defined as a PHQ-9 ≥10 including depressed mood and/or anhedonia.

Fig. 1
figure 1

a Provider documentation in the 8 months before and after initiation of provider feedback for patients with at-risk symptoms and behaviors. b Provider action in the 8 months before and after initiation of provider feedback for patients with at-risk symptoms and behaviors

Full size image

Adherence

Patients reported inadequate adherence in 205 assessments in the 8-month windows before/after provider feedback initiation. Among these patients, providers acknowledged adherence in some way in 81% (95% CI 70–93) of same-day documentation in the 8 months prior to feedback versus 85% (95% CI 80–91) after feedback (p = 0.5; Fig. 1a). Providers documented action in response to inadequate adherence (i.e., adherence counseling, case management referral) in 23% (95% CI 11–36) prior to feedback and 38% (95% CI 31–46) after feedback (p = 0.07; Fig. 1b).

Despite substantial provider documentation of adherence of some kind (good or bad) prior to feedback, we noted that adherence documentation in provider notes among patients with inadequate adherence was often inaccurate. Therefore, for provider notes that acknowledged adherence, we also tracked the percent of discrepant documentation when compared to PRO reports of inadequate adherence. Providers documented discrepant adherence (“perfect adherence,” “missed no doses”, >95% adherence) for 42% (95% CI 27–57) of patients with PRO documented inadequate adherence in the 8 months prior to feedback versus 24% (95% CI 17–31) in the 8 months after feedback was implemented (p = 0.02; Fig. 1a).

At-risk Alcohol Use

Patients reported at-risk alcohol use in 155 assessments in the 8-month windows before/after provider feedback initiation. Providers documented alcohol use for 41% (95% CI 20–61) in the 8 months prior to feedback versus 64% (95% CI 56–72) after feedback (p = 0.04; Fig. 1a). Providers documented action in response to alcohol use (e.g., health educator referral) in 14% (95% CI 0–28) of notes before versus 27% (95% CI 20–35) after feedback (p = 0.2; Fig. 1b).

Substance Use

There were 212 assessments completed by patients reporting current substance use in the 8-month windows before/after provider feedback initiation. Among these patients, provider documentation acknowledged substance use 60% (95% CI 47–73) of the time in the 8 months prior to feedback, and 80% (95% CI 73–86) after feedback (p = 0.004; Fig. 1a). Providers documented action (e.g., treatment referral) in response to substance use in 33% (95% CI 20–45) prior to feedback, and in 41% (95% CI 33–48) after feedback (p = 0.3; Fig. 1b).

Sexual Risk Behavior

Patients reported sexual risk behavior in 344 assessments in the 8-month windows before/after provider feedback initiation. Among these patients reporting sexual risk behavior, providers documented risky sexual behavior in 44% (95% CI 31–56) of visits before versus 53% (95% CI 47–59) after initiation of feedback (p = 0.2; Fig. 1a). Providers documented action taken (i.e., safer sex counseling, referral to health educator) in 11% (95% CI 3–19) of notes before versus 18% (95% CI 14–23) after feedback (p = 0.2; Fig. 1b).

The pattern of findings across domains were similar using 6 and 10-month windows instead of 8 (data not shown).

Provider Identification and Action Among Patients with Depression or At-risk Behaviors Both Before and After PRO Feedback Initiation

These secondary analyses were based on at-risk assessments throughout the study period, not limited to the 8 month windows; however, patients had to report at-risk depression or behaviors both before and after PRO feedback initiation.

Depression

There were 156 PRO assessments completed by patients reporting moderate-to-severe depression symptoms both before and after initiation of provider feedback. Providers acknowledged depression in 74% (95% CI 63–84) of these patients prior to feedback versus 90% (95% CI 84–97) after (p = 0.008; Fig. 2a). Providers documented action in response to depression symptoms in 54% (95% CI 43–66) prior versus 74% (95% CI 64–83) after feedback (p = 0.01). Findings were similar in sensitivity analyses excluding those with suicidal ideation (acknowledged 71 vs. 89%, p = 0.01, provider action 49 vs. 72%, p = 0.009; Fig. 2b).

Fig. 2
figure 2

a Provider documentation among patients with depression or at-risk behaviors both before and after PRO feedback initiation. b Provider actions among patients with depression or at-risk behaviors both before and after PRO feedback initiation

Full size image

Adherence

There were 78 assessments reporting inadequate adherence both before and after provider feedback. Providers acknowledged adherence in 67% (95% CI 51–83) of patients before versus 86% (95% CI 76–97) after feedback (p = 0.05; Fig. 2a). Providers documented action in response to inadequate adherence in 18% (95% CI 5–37) before versus 52% (95% CI 38–67) after feedback (p = 0.003; Fig. 2b). Providers inaccurately documented good adherence in those with inadequate adherence in 36% (95% CI 20–53) before versus 23% (95% CI 10–35) after feedback (p = 0.2).

Alcohol Use

There were 72 assessments completed by patients who reported at-risk alcohol use both before and after initiation of provider feedback. Providers documented alcohol use in 44% (95% CI 27–61) of notes prior to feedback, and in 62% (95% CI 47–78) after feedback (p = 0.2; Fig. 2a). Providers documented action in response to at-risk alcohol use in 9% (95% CI 0–18) before versus 24% (95% CI 10–38) of the same-day notes after feedback (p = 0.1; Fig. 2b).

Substance Use

There were 100 assessments completed by patients who reported substance use both before and after initiation of provider feedback. Substance use was acknowledged in 63% (95% CI 49–78) before versus 88% (95% CI 80–96) of same-day notes after feedback (p = 0.002; Fig. 2a). Providers documented action in response to substance use in 29% (95% CI 15–43) before versus 46% (95% CI 33–58) after (p = 0.03; Fig. 2b).

Sexual Risk Behavior

There were 100 assessments completed by patients reporting risky sexual behavior both before and after initiation of provider feedback. Providers documented sexual risk behavior in 38% (95% CI 22–55) of notes before versus 41% (95% CI 29–53) after feedback (p = 0.5). Providers documented action in response to sexual risk behavior in 12% (95% CI 1–23) before feedback and 23% (95% CI 13–33) after (p = 0.1).

Control Analyses

These analyses were for outcomes unrelated to the clinical assessment to ensure that changes seen in documentation were not due to temporal trends or other factors besides the clinical assessment.

We examined rates of documentation related to four outcomes unrelated to the clinical assessment: diabetes, hypertension, pneumococcal vaccination, and hepatitis C screening. Among 300 randomly selected patients who completed assessments in the 8 months prior to or after feedback initiation, there were no differences in documentation between these two time periods (p values 0.4–0.5).

Discussion

The provision of PRO feedback to clinicians caring for PLWH was associated with improved provider awareness of depression, poor medication adherence, alcohol, and substance use as measured by documentation. A small, but not significant, increase in documentation of sexual risk behavior also occurred after implementing provider feedback. There was a decrease in documentation of ‘excellent adherence’ for patients who self-reported inadequate adherence. Across all domains, impact on provider action was smaller than impact on provider awareness, suggesting that PRO assessment and feedback is only the first step in addressing these complex issues.

Providers documented depression in a high proportion reporting depression even prior to PRO assessment feedback. Nevertheless, documentation was greater after feedback was provided. We were concerned that the automated follow-up to self-reported suicidal ideation would impact our results, but sensitivity analyses excluding those reporting suicidal ideation indicated similar results. These findings demonstrate that the PRO assessment and feedback not only identified a substantial number of patients with suicidal ideation (44 assessments in the 8 month windows before/after feedback), but also increased awareness of moderate-to-severe depressive symptoms among those without suicidal ideation.

PRO feedback did not impact overall adherence documentation for those who self-reported inadequate adherence in the 8 months after feedback initiation, although there was a significant difference in secondary analyses among those reporting inadequate adherence both before and after PRO feedback integration (67 vs. 86%, p = 0.05). Furthermore, providers were more likely to take action addressing adherence after feedback was provided (18 vs. 52%, p = 0.003). After PRO feedback started, providers documented excellent adherence less often for those who self-reported inadequate adherence (42 vs. 24%, p = 0.02 in the primary analysis). While providers often acknowledged adherence in their documentation, the lack of accurate provider documentation of adherence is consistent with studies demonstrating provider-based assessments of adherence are poor [5, 7]. Systematic implementation of PROs that include adherence may help address this issue. While increasing provider awareness of inadequate adherence is important, Wilson and colleagues reported that provider discussions of adherence tend to be directive, rather than problem-solving, and not particularly effective [42]. This suggests PRO feedback may be more effective when targeting the entire healthcare team (including case managers/health educators) rather than relying solely on providers and that additional interventions may be needed to improve the approach providers take to adherence counseling.

Providers acknowledgement of at-risk alcohol use (41 vs. 64%, p = 0.04), and substance use (60 vs. 80%, p = 0.004) increased significantly after feedback, however, there was not a significant increase in provider actions except in the secondary analyses for substance use. These findings underscore the urgent need for provider education and increased use of proven interventions to address all forms of substance use.

Providers did a poor job acknowledging risky sexual behaviors both before and after feedback (44 vs. 53%, p = 0.2) and rarely documented action plans (11 vs. 18%, p = 0.2). This is consistent with prior studies that have suggested providers are not necessarily comfortable discussing sexual risk behavior with patients and that there are many missed opportunities to do so [43–46]. This is concerning for prevention efforts, particularly considering the importance of sexual risk behavior as a means of HIV transmission.

There were substantial differences in the impact of PRO feedback on provider awareness and actions across domains. These cross-domain differences may be because providers prioritize certain clinical problems above others or perceive certain problems as more modifiable than others. Availability of interventions such as mental health services may impact provider actions [47]. Lower rates of identifying and addressing sexual risk behaviors, substance use, and at-risk alcohol use may be because providers believe these issues are more effectively addressed by other clinic staff, such as health educators or social workers.

Feedback regarding risky sexual behaviors had little impact on documented provider identification and actions. This may be due to the reasons mentioned or because providers are not comfortable discussing sexual risk behavior with patients, or may presume existing sexual partnerships are stable or exclusive.

Regardless of variations in feedback impact across domains, our results demonstrate PROs are a promising tool to supplement and enhance patient-provider communication. PRO integration was easily accomplished and well received by providers and staff, in part because of excellent feedback delivery rates (99%), on-site access to referral resources, and provider involvement and satisfaction with the PRO design, collection and feedback delivery process [17].

PROs may be especially useful when caring for patients with multiple morbidities and behavioral issues in a time-limited clinic visit. PROs may also be useful in reducing social desirability bias with potentially greater reporting of risky behaviors than would be reported directly to providers. Research is needed to understand the factors influencing differences in provider use across domains, changes in provider utilization of PROs over time, and the effect of PROs on outcomes.

Strengths

This study evaluated the impact of systematic integration of PROs into a clinical care setting and therefore may be more generalizable than findings from a clinical trial. Providers were involved in selecting domains clinically relevant to PLWH potentially increasing the usefulness and clinical impact. PRO assessments were completed the same day as clinical appointments using touch-screen tablets. Technological advances allowed for automatic scoring and generation of feedback reports, reducing staff burden compared with earlier studies and thereby improving feasibility for implementation in large busy clinics. In fact, the platform and assessment described here have now been integrated into multiple clinics across the US as part of Centers for AIDS Research Network of Integrated Clinical Systems (CNICS) cohort with >70,000 PRO assessments completed to date as part of routine clinical appointments [48]. Additional information on CNICS and its data elements can be found at https://www.uab.edu/cnics/.

Limitations

First, our data relied on medical record documentation raising the possibility that providers identified or addressed issues, but forgot or chose not to document them. However, we have no reason to suspect that documentation completeness would have been systematically higher or lower during the two time periods.

Second, this was an observational study in a clinical care setting comparing documentation across time periods, rather than a randomized controlled trial. However, this is also a strength of this study in that the goal was to evaluate the impact of implementation in clinical care. Also, in secondary analyses where we addressed temporal trends by considering people with at-risk PRO results in both time periods, results were just as strong if not stronger. The within-patient secondary analyses confirmed that the PRO assessment feedback intervention’s success was not primarily driven by temporal differences in the patient population served. Furthermore, when we examined outcomes not included in the clinical assessment such as pneumococcal vaccination, there were no differences over time. This suggests that the differences we noted in provider documentation were in fact due to the feedback and not due to temporal trends.

Third, this study focused on the impact of PROs and feedback specifically on physician awareness and actions. Other health care team members, such as case managers and health educators, may also benefit from the use of PROs and the importance of non-medical providers will increase in the era of patient-centered medical homes. Furthermore, PRO feedback may have other system implications such as billing benefits with systematic documentation of review of system elements.

Fourth, patients filling out the assessment may have increased their awareness of depression or risk behaviors and communicated that to providers directly.

Finally, this study was conducted with only PLWH, limiting generalizability to other chronic disease settings, and was conducted at only one clinic so results may have been different elsewhere. PLWH may have higher rates of risk behaviors than other patient populations providing greater opportunities to intervene and potentially larger impact.

Future Steps and Ongoing Studies

In addition to expanding PROs and their feedback to multiple clinics across CNICS, we are conducting studies on various approaches to optimize providing feedback including having providers sign paper-based feedback forms and implementing results directly into the electronic medical record in real time to facilitate clinical care visits. We are now testing the use of this approach as a building block combining PRO results with additional interventions integrated into clinical care targeting such domains as adherence and alcohol use. We are particularly interested in additional studies looking at the impact of PROs not just on provider behavior but on patient outcomes such as adherence and depression over time.

Conclusions

Implementing same-day PRO collection and feedback into HIV care improves care. Specifically, it improves provider awareness of depression, inadequate adherence, alcohol, and substance use as measured by documentation. PROs decrease how often providers inaccurately documented good adherence. However, PRO impact varies across domains and there is a much greater impact on provider awareness than on actions suggesting the need for additional interventions. Additional multi-disciplinary domain-specific interventions and trainings to effectively address risk behaviors may enhance the effectiveness of PROs and further improve clinical outcomes.