Introduction

Adherence to antiretroviral therapy contributes to optimal clinical outcomes among HIV-infected individuals. High adherence levels are associated with suppressed HIV-1 viral load and less drug resistance, disease progression, and death [1–8]. Although several methods exist to monitor adherence [9], there is no widely-accepted objective measure for use in routine clinical care and clinical-care based research settings [10]. Body fluid sampling and directly observed therapy are not feasible except in limited clinical trial settings, due to their invasive nature, impracticalities, or high cost. Provider reports are unreliable since they often overestimate adherence [11]. Patient self-report can be a useful measure in clinical care settings [9, 12, 13], with the advantages of being less invasive, more informative, less burdensome to administer, and relatively low-cost [10, 12–17]. However, factors such as ceiling effects, cognitive impairment, poor recall, or social desirability bias may sometimes render self-report sub-optimal [16–18].

Pill counts are a commonly used method of measuring adherence [9]. They avoid one of the limitations of electronic medication monitoring because they do not interfere with the use of pillbox organizers, which are an important aid to adherence for many patients [19]. They also avoid the inaccurate adherence estimates from electronic drug monitoring that occurs when patients remove more than one dose at a time or open the device without taking a dose [20]. Clinic-based pill counts can be problematic because they require that patients bring their medications to clinic, and are potentially inaccurate, as patients may not bring all of their remaining medications, or may discard pills before clinic visits (“pill dumping”) [9, 21–23].

Unannounced home-based pill counts address some of the limitations of clinic-based pill counts and have been used in a number of studies [3, 5, 11, 19, 24–29]. While this is a widely used objective measure of adherence that addresses the key limitation of clinic-based pill counts, specifically pill dumping, it poses substantial logistical and cost barriers. It can take many visits to find someone at home, patients may not want home visits, and it is often not feasible in rural or other large geographic areas [30].

To address the logistical issues of home-based unannounced pill counts, Kalichman and colleagues developed a rigorous, alternative approach using unannounced telephone-based pill-counts [30, 31] based on the home-based unannounced pill count protocols of Bangsberg and colleagues [3, 24, 25]. Kalichman et al. [31] found 93 % agreement between telephone and home-based pill counts among a primarily African–American, unemployed patient population in the metropolitan Atlanta area (n = 68). Further, higher adherence levels as measured by phone-based pill counts were found among those with undetectable viral load levels [31].

Phone-based unannounced pill counts are much more practical and less expensive than home-based unannounced pill counts, but their validity has not been widely assessed in patient populations outside of Atlanta. The purpose of this study was to replicate and expand on previous phone-based pill count validation studies in a new patient population with a different geographic layout and demographic and clinical characteristics. Furthermore, while more practical than home-based unannounced pill counts, existing phone-based protocols are still too burdensome for use in many clinical care settings as they include providing patients with cell phones and extensive in-person training for patients. To address these issues, we examined the validity of phone versus home-based pill counts using a simplified protocol streamlined for studies embedded in clinical care settings.

Methods

Subjects and Setting

The study setting was University of Washington’s Harborview Madison HIV Clinic, an outpatient HIV clinic located within Harborview Medical Center, a busy public hospital in Seattle, WA. Study subjects were 18 years of age or older, receiving antiretroviral medication (ARVs) at the time of recruitment, had been Madison Clinic patients for a period of 6 months or more, and had signed the Madison Clinic research registry indicating a willingness to consider participation in research projects (>90 % of clinic patients have signed the registry). Non-English speakers and patients unable or willing to provide informed consent were excluded from the study.

Parent Study

Patients completed a touch-screen-based baseline self-reported clinical assessment that included medication adherence as part of routine clinical care at Madison clinic. Approximately 80 % of clinic patients complete the assessment: there is a 1 % refusal rate. Non-English and non-Spanish speakers are not asked to take the assessment at this time, although we are currently adding a 3rd language: Amharic. Patients who arrive late for their appointments or are being seen for urgent care or other acute issues are not asked to complete the assessment at that visit. Among patients who completed the assessment, 5 % were excluded from being eligible for this study who completed it in Spanish and were not English-speaking and 12 % were excluded due to not currently receiving ART. Eligible patients were approached in the Madison Clinic waiting room prior to their scheduled appointments. A total of 541 patients were approached to enroll 240 patients in the parent study (48 % refusal rate). Patients enrolled in the parent study did not have different demographic characteristics than those who refused. Patients were informed that the study measured medication adherence, and involved a combination of unannounced home-based and telephone-based pill counts, with up to seven possible contacts over a six-month period. Participants were not told that the home-based visits would occur on the same day as a telephone-based count. Patients were not told whether they would receive phone, home, or both types of pill counts in a month. Patients could receive an in-person home-based pill count in association with any of their phone counts. In-person home-based pill counts were conducted after completion of the phone visits on the same day. In contrast to earlier studies, patients were not told that someone was on their way to their home for an unannounced home-based count at the completion of the phone-based count. Patients had the option of refusing calls or visits when contacted.

Current Substudy

The parent study goals were to examine the validity of a number of different adherence measures and items using objective unannounced pill count data as the gold standard. No specific criteria were used to determine which patients in the parent study received home visits. Instead specific calendar weeks and days were identified and patients in the parent study who were due for their pill counts during those timeframes received both phone and home-based pill counts. These analyses included data from 100 paired visits from 93 patients where patients received both an unannounced home and phone-based pill count on the same day.

Pill Count Training

Research Coordinators were trained to conduct unannounced pill counts using a well-established training protocol developed by the Research on Access to Care in the Homeless (REACH) study at UCSF [3, 32]. Coordinators were trained to conduct home-based unannounced pill counts by staff from the REACH cohort.

Pill Count Protocol

Procedures for unannounced pill counts were adapted from protocols for home and phone-based counts kindly provided by Drs. Bangsberg [3, 32] and Kalichman [30, 31, 33]. However, to increase feasibility for wide-scale use and use in clinical care, several modifications were made to existing protocols. In addition to less patient training (they were not asked to bring in medications for an in-person training exercise) [30], and minimizing supplies provided to patients, the most important difference from prior protocols for phone-based unannounced pill counts was that patients were not provided with a cell phone.

A Research Coordinator contacted each patient within 7–12 days of recruitment, with subsequent contacts typically occurring in ~28-day intervals. Approximately 21–35 days from their last phone call/visit was considered ideal, allowing for some contacts between 14 and 42 days from the date of last call/visit if necessary, i.e., if a patient was unreachable after several attempts. Contacts were considered “missed” if they could not be contacted by the end of 42 days. During each home visit, patients were asked to retrieve all of the ARVs currently in their possession. The Research Coordinator conducted two pill counts. If the results did not match, a 3rd pill count was performed. During telephone contacts, the patient was asked to count out their medications aloud using the same procedures.

During every contact, patients were also asked to complete a brief self-reported adherence assessment. This assessment was self-administered using touch-screen tablets by the patient during home visits; during telephone contacts, the Research Coordinator read the questions aloud to the patient, recording the results. The assessment included the following adherence measures: 4-item Adult AIDS Clinical Trial Group (AACTG) instrument [34], a rating scale item [35], and a visual analogue scale item [25, 36, 37]. Adherence items were preceded by a statement regarding the normalcy of missing doses from time to time to help patients understand that they were not being judged and to indicate that honest answers were being sought [38]. All data were collected on touch screen wireless tablet PCs, using a password-protected secure web-based server accessible only by research staff. Data were downloaded daily to the server and removed from the tablets.

University of Washington HIV Information System (UWHIS)

The University of Washington HIV Information System (UWHIS) provided the following data: age, race/ethnicity, sex, HIV risk factor, CD4 cell counts, HIV-1 viral load values, ARV regimen, clinical diagnoses, medication refill dates, medication dosages, and dispense dates. We also obtained depression and substance use data available from the touch-screen based self-reported clinical assessments completed by patients as part of routine clinical care at Madison clinic [39, 40]. Patients were considered depressed if their current Patient Health Questionnaire (PHQ-9) score was 10 or higher [41, 42]. Current substance use was assessed using the Alcohol, Smoking, and Substance Involvement Screening Test (ASSIST) [43, 44] and defined two ways, (1) as any illicit drug use in the prior 3 months, or (2) as any illicit drug use excluding marijuana in the prior 3 months. At-risk alcohol use was assessed using the abbreviated version of the Alcohol Use Disorders Identification Test consumption questions (AUDIT-C) to identify patients with at-risk alcohol use [45, 46]. We used a score of 4 or higher for men and 5 or higher for women to define at-risk alcohol consumption [47].

Compensation

Individuals were compensated $15 for each home- or phone-based pill count. After telephone contacts, patients could retrieve their compensation upon their next clinic visit, or at their convenience.

Analyses

We described continuous variables using mean and standard deviations while categorical variables were described using frequencies and percentages.

Reliability

We examined reliability by comparing the concordance between telephone-and home-based pill counts. Based on the methods of Kalichman et al. [30] we tested agreement between same-day unannounced home and phone-based pill counts using interclass correlations (ICC). We calculated adherence as has been done previously [30] based on the difference between two consecutive pill counts taking into account the number of pills dispensed or taken that day. Based on the methods of Kalichman et al. [30], we repeated ICCs after censoring participants with 90 % adherence or higher, and then 80 % adherence or higher, to remove the potential bias of ceiling effects. In addition to total counts, we repeated ICC comparisons by individual medication for any medication used by seven or more patients. We also repeated ICC comparisons by key demographic and clinical characteristics. We repeated these analyses using percent adherence calculated from telephone- and home-based pill counts rather than total counts.

We further tested concordance using Kappa coefficients of agreement for patients divided into adherence categories defined as <100, <95, <90 %, etc., dropping by 5 % increments down to <75 % adherence.

Discrepancy Analyses

We used χ2 tests to compare demographic, and clinical characteristics, as well as predictors of adherence such as depression and substance use, across patients who were perfectly concordant between total phone and home-based pill counts and patients who were not concordant. For the seven patients with two sets of paired counts, the first pair was used for all discrepancy analyses.

This study was approved by the Human Subjects Division of the University of Washington, and all participants provided written informed consent. Analyses were performed using STATA. All statistical tests used p < 0.05 to define statistical significance.

Results

A total of 100 paired unannounced phone and home-based pill counts were available for analysis. The mean age of study patients was 48 (SD 9) years, 84 % were men, and mean current CD4+ cell count was 539 (SD 280) cells/mm3. Mean CD4+ nadir was 176 (SD 120) cells/mm3. With the exception of race and HIV transmission risk factor, no statistically significant differences were found for any of the key demographic or clinical characteristics between the subset of patients in the current analysis and patients in the parent study who did not have paired visits. This study was slightly more likely to include black patients (26 vs. 18 %), and patients who reported injection drug use as the HIV transmission risk factor (41 vs. 23 %).

On average, approximately 4 attempts were made to reach participants by phone before successfully reaching them for phone-based pill counts. Patients lived a mean distance of ~4 miles from clinic.

At the time of the 100 paired counts, 25 patients were taking a single combination ARV (25 %), 20 were taking two separate ARVs (20 %), 43 were taking three ARVs (43 %), 11 were taking four ARVs (11 %), and 1 person was taking five (1 %).

Concordance of Pills Counted

Table 1 shows the description of pill counts both at home and by phone. Concordance between telephone and home-based pill counts was high. The ICC between phone and home-based pill counts was 0.99 (95 % CI 0.99–1.0, p < 0.01) with an average of 54.1 pills counted by phone and 55.3 by home. When we censored for levels of adherence above 90 and 80 %, as had been done before, the ICCs between home and phone-based pill counts were all 0.99. We examined the ICC comparing phone and home-based pill counts for 10 individual medications, all of which were being taken by 7 or more patients, ICCs ranged from 0.63 to 1.0, with 5 at or above 0.99. Similarly, we examined ICC for phone- versus home-based counts examining demographic and clinical characteristics such as age (<45, 45 or older), race (white, black, other), sex (male, female), current CD4 count (<200, 200–349, and ≥350 cells/mm3), and CD4 nadir (<200, 200–349, and ≥350 cells/mm3), and ICC ranged from 0.96 to 1.0, all p values < 0.05.

Table 1 Descriptive statistics for home-based and phone-based pill counts (N = 100 paired counts)
Full size table

We repeated these examinations using percent adherence rather than total counts and found slightly lower ICC but the association between phone and home adherence rates was significant with an ICC of 0.96 (95 % CI 0.94–0.97, p = 0.01). We examined demographic and clinical characteristics as listed above (age, race, sex, current CD4 cell count and CD4 nadir) using percent adherence rather than total counts and found similar although slightly lower ICC (0.83–0.99, all p values < 0.05 except for current CD4 >350, p = 0.13 and female sex p = 0.06).

We examined the concordance using different levels of adherence from <100 to <75 % using 5 % intervals. The Kappa coefficient for agreement at 90 % adherence was 0.97, p value < 0.01. All Kappa coefficients for agreement for adherence levels from <100 to <75 % were between 0.95 and 0.98, p’s < 0.01.

Discrepancies of Counts

Among 100 paired total counts (by patient, not individual medication), 77 were perfectly concordant, while 23 were discordant. Table 2 describes the demographic and clinical characteristics by whether paired counts were concordant or discordant, focusing on the initial set of paired counts (N = 93). Patients who were not perfectly concordant differed from those who were by race (χ2 6.0, p = 0.05). Specifically concordant patients were more likely to be white, and discordant patients were more likely to be African–American. Patients with discrepant counts also differed by HIV transmission risk factor, and specifically were less likely to be MSM and more likely to be heterosexual than those with concordant counts (χ2 11.4, p < 0.01) (see Table 2). Patients who were not perfectly concordant did not differ from those who were by current depression levels or current substance use (including or excluding marijuana) or at-risk alcohol use. In secondary analyses defining concordance as pill counts that differed by no more than 2 pills, 88 counts concordant, while 5 were discordant. Using these less stringent criteria, the differences by race and HIV transmission risk factor were no longer found however these analyses were based on only 5 discordant events potentially limiting power.

Table 2 Selected demographic and clinical characteristics of individuals with discordant and concordant pill counts (N = 93)
Full size table

Similar to Kalichman et al. [30], we conducted debriefing interviews with pill counters. The most common problem they reported that explained the slight discrepancies between home- and pill-based counts is that patients were more likely to remember or include medications in other places (example the two doses kept in the car) for the home-based counts but not the phone-based counts.

Discussion

We found high concordance levels between phone-based and home-based unannounced pill counts. This study is unique because it was situated in routine clinical care and patients were not given cell phones or extensively trained. Concordance was high for total pill counts, as well as individual medication counts and calculated adherence. While unannounced phone- and home-based pill counts were not perfectly concordant, these discrepancies were often small, and most differed by no more than two pills.

This study supports the high concordance rates found between phone and home-based unannounced pill counts in prior studies [30, 31, 48]. Prior studies that found high rates of concordance between phone-based and in-person home-based pill counts include a small study of 46 patients taking a month of varenicline [48]. In contrast to the current study, the varenicline study had a ~4 day gap between telephone and in-person pill counts [48] and was conducted in a trial setting rather than routine clinical care [48]. Furthermore, prior studies have often used a much more elaborate training protocol with in-office training sessions for patients and even provided cell phones decreasing the feasibility of using them in many clinical care settings [30, 31, 49].

While this study found small numbers of patients with discordant phone and home-based pill counts, there were differences in race and HIV transmission risk factor among those who were perfectly concordant and those who were not. Reasons for these differences are not clear and could be due to socioeconomic and other factors. Of note, all home-based counts were conducted by white male research coordinators.

Home- Versus Phone-Based Pill Counts

Home-based pill counts have been widely used in a number of studies. For example, home-based unannounced pill counts have been used to examine adherence among homeless and marginally housed patients [24], the validity of visual analogue scales [25], and provider assessments of adherence [11]. They have also been used to compare adherence among patients on different treatment regimens [26]; to evaluate the impact of pill boxes on adherence [19]; to examine the impact of food insecurity on viral load [27]; and to evaluate the impact of adherence on the development of drug resistance [5] and viral load [3, 50]. Home-based unannounced pill counts predict changes in viral load [25, 26, 28], and correlate highly with electronic medication monitor measures of adherence [3, 28, 29].

In contrast, phone-based pill counts have been used less widely to date, but they have been used to evaluate the association between adherence and food insufficiency [51, 52], to validate a visual analogue scale self-reported adherence item [53], to examine the association between health literacy and adherence [54], and even as part of medication adherence improvement interventions [55, 56]. Adherence measured using phone-based counts has been found to be associated with viral load [31, 33]. One recent study examined whether phone-based pill counts may have reactive effects and inadvertently improve adherence over time, which could impact findings in intervention study settings, and found that it did not [33].

Both home- and phone-based pill counts are intrusive, raising the question of a potential Hawthorne effect, and have high staff burden [53], however phone-based counts are much more feasible due to elimination of the travel time and reduced implementation costs [48]. Advantages of both are that they are accurate, objective, measure adherence over weeks, limit pill dumping, and include pocketed doses. They do not rely on patient memory, and, unlike some electronic monitoring and other methods of measuring adherence, they do not prohibit pillboxes and other memory devices. Although an impact of the counts themselves on adherence is possible and could interfere with phone-based counts for research purposes, one small study found no difference in adherence among those who received the pill counts alone without also receiving feedback on their count results as well as additional counseling [55].

Challenges in Using Pill Counts

Several considerations should be kept in mind when using pill count approaches to measuring medication adherence whether home- or phone-based. We found that dispense dates were not 100 % accurate, and often were the date a pharmacy filled the bottle, but not necessarily the date the patient picked up the medication. While both home- and phone-based pill counts may provide accurate estimates of adherence, they do not accurately pick up adherence patterns such as treatment interruptions which may have an important impact on outcomes such as viral rebound even within specific levels of adherence [27, 57].

Study Limitations

This study has several limitations that warrant consideration. While one of the larger comparisons of home- versus phone-based pill count studies to date, this is still a relatively small study from a single clinical site. Based on the design and study enrollment, the sample was likely to potentially over-represent individuals with somewhat more stable housing and telephone service, because all participants had access to an in-home landline, and/or cell phone. We did not provide cell phones because a key goal of this study was to assess scalability for use with larger samples. While this could impact inclusion into the study, cell phone availability in the US has rapidly increased from an estimated 38 % of the US population in 2000 to 91 % in 2009 [58]. The small sample size limits the ability to identify subtle differences between those with concordant and discordant counts. Patients may have been motivated by the prospect of compensation, and patients open to unannounced visits may be more adherent. It may be that the phone-based pill counts are impacted by participant awareness that there may be a home visit. However, this was minimized by the fact that they did not know if they were one of the patients who might receive the visit, or that it would occur on the same day as a phone-based pill count. They were given a 6-month window for a possible home-based pill count without additional information. An average of 4 attempts were required for phone-based pill counts. It is possible patients were selective in terms of when they answered the call although the call did come from a blocked number. While both home- and phone-based unannounced pill counts may be a good measure of adherence, they have key limitations including not identifying key patterns of non-adherence, they require a high staff burden, and there is the potential for a Hawthorne effect where the pill counts themselves may impact adherence [53]. With these limitations in mind, we believe this study adds further support to the use of phone-based pill counts for assessing adherence.

Study Strengths

Key strengths of this study include that it was one of the larger comparison studies of phone-based vs. home-based unannounced pill counts, and it was conducted in a routine clinical care setting. Comprehensive clinical data including measures of CD4 and viral load were available, decreasing the reliance on self-reported values. This study included all ARV medications taken by enrolled patients and in contrast to trial settings did not include patients all on the same regimens or limit examination to only one medication such as varenicline [48]. Patients were not aware that the home-based pill counts were going to occur on the same day as the phone-based pill count, and in contrast to prior studies [30], were not warned when completing the phone-based pill count calls of the pending home visit making these visits completely unannounced. This study used a more diverse population than prior studies evaluating phone-based pill counts, which were predominantly African-Americans [30, 31, 33, 48] and therefore both support and enhance the generalizability of prior study findings.

Conclusions

Phone-based counts are comparable to home-based counts for measuring medication adherence but are less expensive and logistically easier to implement. This study demonstrates that a simplified phone-based pill count protocol can be implemented among patients from a routine clinical care setting and is a feasible means of monitoring medication adherence.