Introduction

Retention in HIV care is important for both the individual health of people with HIV (PWH) [1, 2], as well as for public health to prevent onward HIV transmission [3]. While there is no true gold standard for measuring retention in care, various different measures of retention in care are used, each with different advantages and drawbacks [4, 5]. Two broad categories of the commonly used measures include measures of kept visits (e.g., no more than 6 months gap between sequential kept visits) and measures of missed visits (e.g., number of missed visits within a year) [4]. Within each category, different retention measures have previously been highly correlated with one another [4].

With the onset of the COVID-19 pandemic in early 2020, HIV care delivery across the United States drastically changed. In-person HIV care appointments were cancelled, and HIV clinics rapidly scaled up telehealth appointments in an attempt to provide continuity of HIV care while protecting PWH from exposure to COVID-19 [6,7,8,9]. It is unclear how these changes in HIV care delivery during the COVID-19 pandemic have impacted traditional measures of retention in care, although some have already proposed including telehealth in retention measures [10, 11]. We aimed to evaluate the correlation among different measures of retention in care in the year before and the first year of the COVID-19 pandemic, as well as to determine how HIV retention measures are impacted by the inclusion of telehealth appointments.

Methods

We collected data for adults (18 years of age or older) with HIV who received care at a Ryan White-funded HIV care clinic at an academic medical center in Chicago, Illinois between March 15, 2018 and March 14, 2021. We defined two time periods: pre-pandemic which included the year prior to the pandemic, March 15, 2019 through March 14, 2020, and pandemic, which included the first year of the COVID-19 pandemic, from March 15, 2020 through March 14, 2021. March 15, 2020 was the date that the HIV care clinic began cancelling in-person appointments for patients and rapidly transitioning to telehealth in response to the COVID-19 pandemic. We included patients that had at least one scheduled HIV care appointment during the first 189 days of either the pre-pandemic or the pandemic observation periods and had attended at least one HIV care appointment in the year preceding the time period of interest (3/15/2018–3/14/2019 for the pre-pandemic year and 3/15/2019–3/14/2020 for the pandemic period). These inclusion criteria were chosen to ensure that only active patients were included and that all included patients had the opportunity to fulfill all retention measures in the period of interest. Patients were eligible to be included in both the pre-pandemic and pandemic time periods if they met the inclusion criteria for both. For all patients included in the study, we collected demographic information, comorbidities, HIV care appointments including in-person and telehealth, and the outcome of each appointment (kept vs. missed). Of note, prior to the COVID-19 pandemic, our clinic did not offer telehealth appointments. During the pandemic, both telephone and video visits were offered as options for telehealth appointments. Patients could complete telephone visits using any type of telephone. Video visits could be completed via smartphone, tablet, or computer using either Zoom or MyChart, the electronic patient portal associated with the Epic electronic medical record.

Six retention measures were calculated to measure retention in HIV care. These retention measures included three measures of missed visits and three measures of kept visits. The three missed visit measures included a count of the number of “no show” visits, a dichotomous categorization of any versus no “no show” visits, and visit adherence, defined as the proportion of kept visits (a “completed” appointment) to scheduled visits (“completed” and “no show visits”). Kept visit measures included 4-month constancy (the number of 4-month periods with at least one kept visit), 6-month gap measure (a dichotomous measurement of no more than 189 days between sequential kept visits during the 12-month observation period), and the Health and Resources Services Administration HIV/AIDS Bureau (HRSA HAB) measure (2 kept visits separated by more than 90 days during the 12-month observation period). These six measures were chosen because they have been previously utilized in prior literature and each have distinct advantages and disadvantages in measuring retention in care [4, 5, 12]. For example, missed visits are easy to measure and have been shown to be independently associated with increased risk of mortality among people with HIV. However, missed visit measures are limited in that they do not necessarily account for patients who are lost to follow-up and have no scheduled visits. Measures of kept visits (e.g., 6-month gap) can identify patients who are lost to follow-up, but do not necessarily account for patient complexity as judged by a provider (i.e., certain patients may need to be seen more frequently than every several months). For all measures other than the count of missed visits measure, we coded values so that a higher value indicated better retention (e.g., for the 6-month gap measure, PWH with a 6-month gap in care were coded as “0” since that indicated lack of retention whereas those without a 6-month gap were coded as “1” since that is associated with higher retention). Same-day cancellations by patients were counted as missed visits. Visits cancelled in advance by patients and visits cancelled by the clinic were excluded from the analysis.

We calculated these six retention measures separately for both the pre-pandemic time period and the pandemic time period. For the pandemic time period, we calculated the measures two different ways: (1) including all in-person and telehealth appointments as HIV care appointments and (2) including only in-person appointments as HIV care appointments, excluding telehealth appointments.

We used student T-test (test statistic T) and Z-test (test statistic Z) to compare the six retention measures for the pre-pandemic time period to both the pandemic time period including telehealth and the pandemic time period in-person only, excluding telehealth. We used Chi-squared test (test statistic χ2) to compare demographics for people who completed all of their scheduled telehealth appointments to people who missed at least one of their scheduled telehealth appointments in the pandemic year. We used Spearman rank correlation to assess the correlation between different measures of retention in care for (1) pre-pandemic period, (2) pandemic period, including both in-person and telehealth appointment, and (3) pandemic period for only in-person appointments, excluding telehealth visits.

This study was approved by our institution’s Institutional Review Board. Because this study was a retrospective analysis with minimal risk to participants, a waiver of consent was obtained.

Results

Five hundred and forty-four people with HIV had at least one scheduled HIV care appointment during the total study time period. The median age was 40 [interquartile range (IQR) 30–56] years with the majority of participants being male (64.2%, 349/544) and Black (86.6%, 471/544). The most common insurance utilized was Medicaid (42.1%, 229/544) followed by Medicare (26.3%, 143/544) and private insurance (30.7%, 167/544). The most common comorbidities among PWH were hypertension (43.6%, 237/544), depression (24.8%, 135/544), anxiety (9.0%, 49/544), and Hepatitis C (7.4%, 40/544).

Table 1 displays the outcomes for each of the six measures of retention in care in the 1 year before the COVID-19 pandemic and during the first year of the COVID-19 pandemic. There was higher retention based on HRSA HAB and 6-month gap measures in the year prior to the pandemic than during the pandemic excluding telehealth appointments (HRSA HAB, 72.8% vs. 51.2%, Z = − 5.20, p < 0.001; 6-month gap, 60.5% vs. 51.2%, Z = 2.48, p < 0.05). Including telehealth appointments as HIV care appointments when calculating measures of retention in care resulted in higher measured retention rates as compared to those excluding telehealth for the missed visit count measure of retention [median 0, IQR (0–1) vs. median 1, IQR (0–1), T = 2.67, p < 0.01] and the HRSA HAB measure (58.2% vs. 51.2%, Z = 3.48, p < 0.001). When comparing patients retained according to the 6-month gap measure of retention in the year prior to the pandemic and during the pandemic both with and without telehealth, there were not statistically significant differences in sex (pandemic with telehealth Z = 0.14, p = 0.89; pandemic without telehealth Z = 0.35, p = 0.73), race (pandemic with telehealth Z = − 0.16, p = 0.88; pandemic without telehealth Z = − 0.27, p = 0.79), or insurance (pandemic with telehealth Z = 0.13, p = 0.90; pandemic without telehealth Z = 0.40, p = 0.69).

Table 1 Frequency and percentages of the 6 retention measures for 1 year pre-pandemic (3/15/2019–3/14/2020) and the first year of the COVID-19 pandemic (3/15/2020–3/14/2021), with and without inclusion of telehealth appointments
Full size table

Two hundred and eighty-nine people had at least one telehealth appointment scheduled during the pandemic time period. Of these, 249 (86.2%) people completed all of their scheduled telehealth appointments and 40 (13.8%) people missed at least one of their scheduled telehealth appointments. There was not a significant difference in sex for those who completed all telehealth appointments compared to those who missed a telehealth appointment [65.9% (164/249) vs. 70.0% (28/40) male, χ2 = 0.11, p = 0.74]. There was also not a significant difference in race for those who completed all scheduled telehealth appointments versus those who missed a telehealth appointment [81.9% (204/249) vs. 90% (36/40) Black, χ2 = 1.07, p = 0.30]. There was a significant difference in insurance for those who completed telehealth appointments compared to those who missed a telehealth appointment [28.1% (70/249) vs. 37.5% (15/40) Medicaid, χ2 = 6.14, p < 0.05].

Table 2 displays the Spearman rank correlation of the six retention measures pre-pandemic as well as during the pandemic. When telehealth appointments were included as HIV care appointments in the analysis, there was a statistically significant higher correlation between visit adherence and missed visit count (− 0.93, confidence interval (CI) − 0.95 to − 0.91 vs. − 0.86, CI − 0.90 to -0.80) and visit adherence and the dichotomous missed visit measure (0.94, CI 0.92 to 0.96 vs. 0.89, CI 0.85 to 0.91). However, there was a statistically significant lower correlation between the 6-month gap and visit adherence measures (0.35, CI 0.26 to 0.44 vs. 0.56, CI 0.46 to 0.65).

Table 2 HIV retention measures compared using Spearman rank correlations pre-pandemic (3/15/2019–3/14/2020) and during the first year of the COVID-19 pandemic (3/15/2020–3/14/2021); including or excluding telehealth appointments
Full size table

Discussion

We compared different measures of retention in HIV care both during and prior to the COVID-19 pandemic and with and without inclusion of telehealth appointments. To our knowledge, this is the first study comparing multiple different measures of retention in HIV care during the COVID-19 pandemic. Although fewer patients were seen during the pandemic period (both with and without telehealth), demographic characteristics of patients who were retained in care remained similar.

For retention measures associated with kept visits, patients were more likely to be retained pre-pandemic than during the COVID-19 pandemic. For example, 72% of patients were retained according to the HRSA HAB measure pre-pandemic, compared to 51.2% for in-person visits during the pandemic. This finding is expected given the national trend in delay and cancelation of elective healthcare visits, especially during the early phase of the COVID-19 pandemic [13]. Norwood et al. similarly found a 33% decrease in medical encounters among PWH in the second quarter of 2020 compared to the second quarter of 2019 [14]. While our study did not examine reasons for poor retention in HIV care during the pandemic, other studies have found that unstable housing, food insecurity, substance use, and mental health care disruptions were associated with missed HIV care visits [15].

Of note, when telehealth appointments were included in the analysis for visits during the pandemic time period, more patients were considered retained than when telehealth appointments were excluded for all three kept visit measures and the dichotomous missed visit measure. Whereas 59.2% of patients missed at least one appointment when in-person visits alone were considered, only 45.9% of patients missed visits when considering both in-person and telehealth appointments. These findings add to the growing literature suggesting that telehealth improved the ability of patients to maintain access to HIV care during the COVID-19 pandemic. Boshara et al. similarly found higher appointment attendance with telehealth than in-person appointments during the COVID-19 pandemic in a safety-net clinic in Chicago [16]. Telehealth appointments have some limitations, including not providing the opportunity for patients to have laboratory tests obtained, vital signs measured, or immunizations given. However, telehealth appointments do offer the opportunity for providers and patients to discuss treatment adherence and prescribe needed refills. PWH have reported that telemedicine improved their ability to communicate with their providers during the COVID-19 pandemic [17]. Even prior to the pandemic, others have called for telemedicine to be included in HIV retention measures [11]. In light of the rapid increase in telehealth during the COVID-19 pandemic, inclusion of telehealth in retention measures seems especially prudent.

When considering the correlations between different retention measures, we observed strong correlations among measures of missed visits (pre-pandemic Spearman coefficient = 0.920 − 0.935, pandemic in-person only Spearman coefficient = 0.857 − 0.915, pandemic in-person and telehealth Spearman coefficient = 0.932 − 0.957). There were moderate to high correlations among measures based on kept visits (pre-pandemic Spearman coefficient = 0.471 − 0.790, pandemic in-person only Spearman coefficient = 0.760 − 0.857, pandemic in-person and telehealth Spearman coefficient = 0.696 − 0.859). Correlation was lower across the two different groups of measures. This finding is similar to what was reported by Mugavero et al. when considering correlations among different retention measures prior to the pandemic [4]. Interestingly, we found that the correlation among kept visit measures was higher during the pandemic than pre-pandemic. This finding provides reassurance that traditional measurements of retention in care may continue to be reliable during the COVID-19 pandemic. Inclusion of telehealth appointments had variable effects on the Spearman correlation coefficients with some slightly increasing and some slightly decreasing. However, there were no large changes in correlation when telehealth appointments were included, suggesting that at least the correlation between different measures is not greatly impacted by inclusion or exclusion of telehealth appointments.

Our study has several limitations. Although we evaluated the correlation of six different measures of retention in care, we did not evaluate how these measures correlate with viral suppression. We chose not to include viral suppression because laboratory values were missing for many patients during the COVID-19 pandemic, and so results may not be reliable. Studies prior to the pandemic have shown strong association between all six retention measures and viral suppression [4], but these studies have not been replicated during the COVID-19 pandemic. Reports have been conflicting regarding the impact of the COVID-19 pandemic and telehealth on viral suppression rates among PWH [10, 18]. Our study was performed at an urban academic medical center among patients who were largely insured, and results might not be generalizable to other settings or clinics that serve a large population of patients without health insurance. In addition, some patients may have transferred care or died during the study and could have been inappropriately classified as not retained in care. However, we only included people with at least one visit during the first 6 months of each study period in an attempt to limit the study to active patients.

Conclusion

In conclusion, we assessed correlations among six different measures of retention in care during the COVID-19 pandemic with and without inclusion of telehealth. We found that more patients were considered retained in care during the COVID-19 pandemic when telehealth appointments were included in the analysis. Our findings suggest that correlation among retention measures were not greatly impacted by the COVID-19 pandemic and that telehealth appointments should be included in the measurement of retention in HIV care. More research is needed to better understand the impact of the COVID-19 pandemic on the correlation between different retention measures and viral suppression and other long-term outcomes among people with HIV.