Background

Many behavioral health studies require measurement of health services utilization. Depending on study objectives, some studies require measurement of comprehensive service use and cost, whereas others require only brief measures of utilization. Brief measures might include any use of behavioral health services in the previous year. The validity of self-reported service use may be limited by inaccurate recall, especially for longer time frames. Some studies find that individuals underestimate their health service use,1,2 while other studies have found that self-report methods tend to overestimate the actual use of health services.3–5 Most reports have concentrated on the validity of counts of visits or hospital days, while only a few have focused on the validity of self-reports of whether behavioral health services have been utilized. Briefer dichotomous measures are particularly important when studying access to behavioral health care or when entry into behavioral health treatment is the outcome of interest. Such questions are frequently asked in community population studies in which use of behavioral health services is a relatively rare event, even when the sample is restricted to potential users such as at-risk drinkers.6 It is important to understand the validity of self-report of any behavioral service use for several reasons: (1) because use of treatment services is very low,6 errors of self-report negatively affect the kappa statistic (low base-rate problem); (2) because the stigma associated with alcohol problems and alcohol disorders may preclude self-report of treatment; and (3) because there are many barriers to obtaining treatment, understanding initiating treatment is of policy concern. Therefore, self-report of any service use, particularly for alcohol and mental health treatment, needs to be as accurate as possible.

The authors developed the Brief Health Services Questionnaire (BHSQ) to obtain self-reported information regarding health provider contacts over a 2-year interval for a community sample of at-risk drinkers.6,7 The BHSQ was also used to initiate the search for complete medical records for manual review and abstraction, including information on insurance coverage and health care providers utilized in the previous 2 years. It is important to note that collection of medical records and eventual abstraction would not be possible without earlier information on service providers and insurance companies. However, the question arose as to whether the BHSQ might have validity in its own right as a brief tool for measuring use of behavioral health services, general medical services, and other types of health services. The current report examines the feasibility, validity, and average cost of using the BHSQ alone relative to a hybrid approach8 that combined administrative and insurance data with medical records abstraction. The report also provides discussion on the feasibility and obstacles to assembling complete medical records for a multi-state community sample.

Methods

The larger study from which participants were drawn, the Rural Alcohol Study (RAS), has been previously described in greater detail.6,7 The RAS was a population-based, 18-month longitudinal survey of individuals residing in six southern states, designed to evaluate service use and course of drinking for at-risk drinkers.6 Follow-up rates of above 90% were maintained at 6- and 12-month follow-up interviews. A total of 78% (n = 573) completed the final 18-month follow-up interview. Participants who completed the final interview were contacted by telephone and invited to participate in an ancillary study to document their heath care utilization and costs. A total of 441 of the eligible 573 contacted (77%) agreed to participate, signed a written consent to release all of their medical, insurance, and pharmacy records, and completed the interviewer-administered BHSQ via telephone for which they were reimbursed $15. The study participants' medical records were collected for the 2-year period spanning 6 months prior to enrolling in the study through the 18-month follow-up (see below). The 441 participants were significantly less likely (P < .05) than nonparticipants included in the 18-month interview from the RAS to have a high school education (86% vs. 93%), but did not differ on any other demographic or clinical variables. Table 1 presents the characteristics of the 441 study participants at the baseline assessment.

Table 1 Self-reported demographic and clinical measures at baseline (N=441)
Full size table

BHSQ self-report of provider contacts

The seven-item BHSQ, shown in Table 2, is easily administered in person or by telephone in about 5–10 min. Trained interviewers administered the BHSQ by telephone to record all provider contact information (by type of service) for the 2-year study period. The current analysis is primarily concerned with responses to items 3 through 6 of the BHSQ.

Table 2 Brief health services questionnaire (BHSQ)
Full size table

Obtaining access to records

The RAS research team developed a multi-informant protocol to collect complete medical, billing, insurance, and pharmacy records from health care providers, agencies, and insurers. The protocol was modeled after previous data collection approaches, and although somewhat labor-intensive, it has been used to collect complete utilization and cost data in several successful projects.9–12 After permission was obtained from participants to release each source of health services records, all health care providers noted on the BHSQ by participants were then contacted by RAS team members, often multiple times by letter and/or telephone, to obtain photocopies of all insurance, medical, billing, and pharmacy records. Health care providers were predefined as physicians, nurse practitioners, osteopaths, nurse anesthetists, nurses, physician assistants, optometrists, dentists, ophthalmologists, podiatrists, speech pathologists, audiologists, physical therapists, occupational therapists, chiropractors, psychologists, counselors, certified substance abuse counselors, social workers, and psychological examiners. Provider contact information for routine dental and optometry services were not requested, but provider contact information was requested for any surgical or other nonroutine procedure. Records were also reviewed by the RAS team to identify additional sources of care that may not have been directly reported by participants. New sources of care were then contacted and records obtained, if appropriate. This iterative process continued until records were deemed complete. Any provider or insurer who refused to send records was contacted by the project coordinator and again by the project director, who then made final decisions whether to stop pursuing records. Each participant's file was considered complete when all essential utilization and charge data were obtained from the participant's provider(s) and/or insurer(s).

Abstracting medical records

A chart abstraction spreadsheet was developed by the RAS research team to structure the process of manually abstracting and coding information from health records. The abstraction spreadsheet was largely based upon and modeled after previous work.13,14 A research assistant was trained by a psychiatrist to abstract the medical records with a sample of 10 reabstracted by a second psychiatrist to ensure accurate review. For each service from a health care provider, the medical record abstractor reviewed and abstracted information from records for provider contact types, including hospital, general medical, psychiatrist, psychologist, or counselor.

Costs of data collection and abstraction

Costs for administering the BHSQ were calculated to include research assistant time and training time, and the average cost/BHSQ obtained was calculated. The total cost to collect and abstract data included training time (research assistant and psychiatrist trainer and reabstractor for reliability evaluation), research assistant time to collect and abstract data, project coordinator time to oversee the data collection process, project director time to direct the study, and other payments for records, postage, and long distance telephone calls. Total cost was divided by the number of participants to calculate an average cost per subject. Research costs associated with designing the chart abstraction spreadsheet form and overhead costs such as office space, office supplies, and miscellaneous items were excluded.

Data analysis

All chart abstraction data were entered into a computer database and linked by subject identifier to previous RAS datasets. The types of provider contacts from BHSQ responses and from abstracted records (e.g., hospital, general medical, psychiatrist, etc.) were collapsed into three groups of providers: (1) general practitioners, which included general medical professionals, obstetrics–gynecology specialists, medical specialists, surgical specialists, and hospital professionals; (2) mental health and substance abuse providers, which included mental health specialists, substance abuse specialists, psychiatrists, psychologists, social workers, and counselors; and (3) other providers, which included chiropractors, physical and occupational therapists, speech pathologists and audiologists, home healthcare professionals, radiologists, pathologists, anesthesiologists, and lab work only contacts. For each broad category of provider contacts, raw percent agreement between the BHSQ and medical records was calculated, and kappa coefficients were used to assess agreement above chance. McNemar's test for matched pairs was applied to each paired type of provider contact to estimate whether there was a significant direction in disagreement between the BHSQ and the medical record data.

Recall that administration of the BHSQ was necessary to obtain the self-report information necessary to identify and abstract health care provider records. Thus, the average cost of abstraction is the estimated average cost of self-report plus abstracted data collection. If a standard measure of “outcome” or “effectiveness” were available, then a simple cost-effectiveness analysis could be performed to determine the incremental cost of abstraction (relative to self-report alone) per unit of additional outcome. Unfortunately, such a standard outcome does not exist in this context because health care services have numerous domains and “unresolved errors” were found to occur with both self-report and abstracted data. Furthermore, it is not clear how such cost-effectiveness ratios would be evaluated relative to some benchmark values. Rather than run a formal cost-effectiveness analysis, we examined the extent of service provider agreement between self-report and abstracted sources, and then compared the average cost of each approach.

Results

Validation of the BHSQ self-reported provider contacts

Table 3 summarizes comparisons between self-reported and abstracted provider contacts for the three types of providers and an aggregated variable that combines all provider contacts. Raw percentile agreement ranged from 61% to 93% for the provider contacts. Kappa statistics were moderately high for general practitioner and mental health specialist contacts (% agreement = 79% and 93%, kappas = .50 and .62, respectively) and aggregated provider contacts (% agreement = 82%, kappa = .54). Kappa was low for other health contacts (kappa = .04). McNemar's test statistic showed no significant differences in the direction of disagreement for general medical and mental health specialty provider contacts or aggregated contacts (general medical = 3.04, P = .0811; mental health = 3.33, P = .0679; aggregated = 1.85, P = .1742). However, there was a significant McNemar's test for other provider contacts (P < .0001), with participants self-reporting far fewer contacts (14%, n = 63) than was found in medical records (37%, n = 163).

Table 3 Comparison between self-reported and abstracted medical and other health services provider contacts (N=441)
Full size table

Table 3 also presents the reasons for nonmatched contacts. For cases that had nonmatched provider contacts, the majority of the nonmatched contacts were attributable to situations in which relevant providers, identified by self-report on the BHSQ, were not identified by chart abstraction (e.g., subject reported general practitioner contact, but medical record showed a different type of provider). The second most common reason for nonmatched contacts was attributable to situations in which self-report and/or provider records were outside the 2-year study window. Additional discrepancies, although not as frequent, were found for instances where self-report on the BHSQ could not be verified by provider records, and when nonmedical provider contacts were reported on the BHSQ but were not obtained from medical records. In some cases, a provider sent medical records that did not include all dates of service found in the insurance record. Of the 441 study participants, 17.7% (n = 78) had incomplete provider medical records because of missing progress notes or difficulty obtaining records.

Cost of administering the BHSQ

One research assistant devoted 50% effort over the course of a year to invite study subjects to participate, administer the BHSQ, and obtain signed authorizations to release records. We calculated the cost as $31 per obtained BHSQ ($13,794/441, with the total representing research assistant time plus training costs) with the research assistant spending substantial time trying to locate and contact the entire 18-month study sample. There were minimal training costs for the BHSQ (an approximately 30-min training included in the general training and orientation of the RAs), with an overall cost of $44, which could be added to the BHSQ total cost without changing the $31 estimate.

Cost of obtaining and abstracting medical records

Collecting and abstracting health care records for the study participants took approximately 12 months and required a staff of 3.5 FTE research assistants, one full-time project coordinator, and 10% effort by a project director (see Table 4). Included in the costs of obtaining records is the cost described above for the BHSQ, the instrument used to initiate the process. In terms of research assistant time, it took an average of 10 hours to collect and 4 hours to abstract the records, per participant. Copying and mailing charges were approximately $9.50 per person, postage charges were $7.85 per person, and long distance phone charges were $13.50 per person. Training costs were calculated as 2 weeks' RA training time and 2 weeks' work of psychiatrist time. Overall, the total cost to collect and abstract the medical, billing, and insurance records was $186,953, for an average of $424 per participant.

Table 4 Estimated costs to collect and abstract medical records (N = 441)
Full size table

Discussion

Validity

Choosing the best measurement method to assess use of behavioral health services, general medical services, and other types of health services is a process that is highly dependent on the context in which information is collected, the quality and availability of data, and the relative costs and benefits of particular approaches. The study on the validity of self-reported health services provider contacts by using the interviewer-administered BHSQ alone compared to the BHSQ linked with more detailed and more costly medical record abstraction found acceptable levels of agreement (i.e., >75%) for general medical and mental health provider contacts, and for a summary variable that aggregated the provider contact types. Furthermore, the kappa statistics indicated that there was no apparently “direction” of disagreement—in other words, sometimes the abstraction identified providers not indicated in the BHSQ, and sometimes the reverse. Unacceptable agreement was found for “other” provider contacts, most likely a result of the heterogeneity of this category. Results for general medical, mental health, and aggregated services suggest that the corresponding items should be retained in the BHSQ, but that use of the “other” category might benefit by adding prompts for provider types most likely to be reported. Another possible improvement is to divide this miscellaneous category into two or more separate categories.

This study examined a relatively lengthy recall period, so it is not surprising that most of the reasons for disagreement between BHSQ reports and abstracted medical records were related to participants' inaccurate recollection of the relevant provider type or health utilization episodes within the 2-year study window. Shorter recall periods would likely produce even better agreement. Other factors, accounting for about 10% of the discrepancies in provider contacts reported and utilized, were related to situations where insurance companies and/or service providers did not have or were unable or unwilling to provide requested copies of participants' records.

Admittedly, BHSQ did not elicit information regarding the number of contacts for each provider identified, mainly because the original purpose was to measure whether any services had occurred rather than quantity of use. The validity of this type of information is unknown, especially for a 2-year window, but could be generated with a slight modification to the BHSQ and collection of data from external sources.

Costs

It is important to note that the cost estimates to administer the BHSQ may be overestimated in our study because calculations included both time to administer the BHSQ and procedures related to obtaining informed consent from participants for collecting medical records. Naturally, the cost of administering the BHSQ would be minimal if it were embedded in already scheduled interviews. Several health-related studies have presented cost data for subject recruitment and screening methods,15,16 or detailed costs associated with health-related survey administration methods,17–20 and data collection.21,22 But only a few have specifically estimated the costs associated with the collection and abstraction of complex23 health services utilization data. Given the potential utility of this type of information (e.g., in planning budgets for grant applications, conducting health services research, allocating resources), it is somewhat surprising that a dearth of published research exists on the topic.

With these caveats in mind, we found that the average cost of administering the BHSQ alone ($31) was about 14% of the average cost of the BHSQ plus record abstraction ($424). If budget constraints are lacking, then a combination of self-report and record abstraction to verify contacts and increase precision is indicated. However, for studies with limited budgets, administering the BHSQ alone might be a viable and financially prudent option for obtaining data on health services contacts, especially for behavioral health and general medical providers.

Feasibility and utility of the BHSQ vs. medical record review and lessons learned

The BHSQ was a very useful instrument in the current study and was critical in the process of obtaining essential medical and billing records for this community sample. Population-based studies such as the current one, national or regional, on the other hand, typically have to rely on at least some measures of self-reported utilization and many participants in these types of studies may not have health insurance or regular providers where they obtain mental health and medical care. Approximately one-fourth of the participants in the current study did not have any type of health insurance, thus removing this source of information for utilization and cost. The current study involved a complex, detailed protocol involving multiple states, providers, and agencies. Previous experience with a small single state, using similar data collection methods, revealed fewer barriers in obtaining records, primarily because of a single major insurer that was well known to the research group and a more limited number of providers.13 Thus, the BHSQ administered alone or used in combination with more intensive medical records verification may be very practical for less complex projects as well as less clinically detailed research studies.

It is also important to mention that while the BHSQ was very useful in the present study of community-dwelling at-risk drinkers, there are a number of practical issues to consider if there is a need to measure complete utilization of services over extensive time intervals. Insurance plans, especially out-of-state insurance companies, may not be receptive to external requests for medical records, and often require time-consuming release procedures that may involve their legal departments. Some providers, especially hospitals, may require patient authorizations to be less than 90 days old. Furthermore, some providers may require signatures on their own institutions' release of information forms. Reobtaining patients' consent was common and necessary in this study because of the inability to obtain records quickly enough after the participants' initial approvals, our limited staff size, and the iterative nature of the medical record collection approach employed. Some insurance companies stated their databases were not designed to readily provide the requested data, while others seemed to err on the side of caution in terms of protecting the confidentiality of their patient records (e.g., some agencies wanted additional agency-specific disclosure statements or were only willing to mail records to patients). Other delays arose in cases where the insurance contact information reported by respondents was on file with the insurer under a spouse's social security/membership number.

The authors suspect that some of these practical issues could be even more problematic in today's health care climate. This study was conducted before the implementation of the Health Insurance Portability and Accountability Act (HIPAA) health information privacy rule, which went into effect in April 2003.24 The high rate of provider compliance with supplying medical records to the research team found in this study (90%) may be more difficult to achieve today given the heightened focus on protection and confidentiality of health records, as a result of HIPAA regulations. In addition, if the medical record and abstraction component of this study were replicated, an HIPAA authorization form would need to be obtained from participants, potentially increasing the cost estimates presented in this paper.

Implications for Behavioral Health

The process of obtaining and validating self-reported health provider contact and utilization data can be difficult, time consuming, and expensive. Results of this study indicate that a short and relatively inexpensive instrument, the BHSQ, demonstrates adequate validity as an approach for collecting self-reported health provider contacts for general medical and mental health specialty providers among community-dwelling at-risk drinkers. The BHSQ would likely be applicable to other studies and can be used in several ways to collect health contact and utilization data. For example, self-reported health care utilization on the BHSQ could aid in determining whether a patient is in contact with the general medical system (e.g., documenting follow-up with a doctor after an ER visit). The BHSQ also may be useful as a brief recall of basic use of health services and the range of services that patients receive. The process of asking for contact information including the names and addresses of all health service providers might be a particularly useful exercise to aid in respondent recall. For behavioral health services projects that need to obtain accurate information on healthcare costs, the most reliable methods would likely involve some melding of both insurance records and abstraction of medical records, recognizing the barriers and complexities for multi-state community studies described above. For projects that were only able to obtain self-reported provider contacts, the BHSQ has definite cost advantages over more elaborate methods.

In conclusion, a strength of the BHSQ instrument is that it is simple and easy to administer and shows promising validity for measuring a range of health provider contacts. Furthermore, this approach could be useful for some study designs such as those focusing on treatment entry and barriers to treatment. Finally, this instrument is a very useful way to initiate the search for medical records should a study require a more intensive review of utilization, quality of care, or healthcare costs.