Research on children’s mental health service utilization is imperative because of the estimated 5% of children in the United States who have serious mental illness, only 51% have had contact with a specialty mental health professional (Federal Interagency Forum on Child & Family Statistics 2010). Service utilization is studied as a means to the end of improving health outcomes, and access and engagement in high quality mental health services may be critical to positive outcomes (Weisz et al. 1995). The importance of children’s mental health service utilization ultimately rests in the outcomes it achieves. As such, an increased interest in children’s usual care, otherwise known as “treatment as usual” (TAU) has emerged (National Institute of Mental Health 2002).

Treatment as Usual

TAU has been applied in children’s mental health research in varying ways, making it difficult to consistently define or measure it. One author points out, “We know little to nothing on how to even describe TAU and even less about its effectiveness” (Bickman 2008). In many studies, TAU is the typical treatment that is provided to a child with a specific diagnosis (e.g.,talk therapy, case management). Researchers often investigate how their newly developed mental health treatments compare to the TAU in impacting outcomes (e.g. Corcoran 2006; Ruffolo et al. 2005). Another way TAU is conceptualized in the research is as the set of all services that the client is utilizing. Researchers may investigate how specific mental health treatments impact TAU as an outcome in clinical research (e.g. Mendenhall et al. 2009). In other words, did the specified treatment affect the pattern or quality of the other services the client was receiving? This interpretation of TAU closely relates to investigation of quality of care and overall child outcomes, and so for this article, we will be focusing on this latter interpretation of TAU.

For TAU to be used as a justifiable variable of interest in children’s mental health research, accurate methods of recording and tracking patient usage of and experience with medications and services are needed. An instrument measuring TAU should fully capture the pathways through treatment (Horwitz et al. 2001) and the details of the mental health services and medications a child is utilizing (e.g. episodes of care, types, dosage, intensity; Burns 1999; Kessler et al. 1980; Weisz 2004). When TAU is studied as a potential clinical outcome, measures of TAU should include data about the client’s patterns of use and experience with services and medications in order to understand what affect the identified treatment may have had. Several instruments for measuring children’s mental health service utilization exist but whether they are appropriate for measuring TAU must be investigated. This article summarizes the currently available methods for measuring service utilization, and introduces a new instrument for assessing TAU, the Service Provider and Medication Usage Grids.

Existing Instruments

Various instruments are available for measuring children’s mental health service utilization but these have primarily been developed for use in large-scale epidemiological studies. The Child and Adolescent Service Assessment (CASA) was developed for the Great Smokey Mountains Study as a self-report instrument to collect information on mental health service utilization of informal, personal and community resources by youth ages 8–18 years old in the public and private sectors (Ascher et al. 1996). Additionally, the CASA assesses attitudes toward treatment costs and perceived barriers to care. Test–retest reliability for the CASA ranges from good to excellent, with restrictive services being reported with highest reliability (Ascher et al. 1996; Farmer et al. 1994). The CASA has demonstrated 85–90% agreement with provider records on whether services were received (Ascher et al. 1996).

The Services Assessment of Children and Adolescents (SACA) was developed for possible use in a national survey of children’s mental health service use and need, and it records children’s use of mental health services in three broad domains: inpatient, outpatient, and school (Hoagwood et al. 2000). The measure captures the following areas of children’s mental health service utilization: parent and youth perspectives, lifetime and 1 year use, disaggregation of types of services from provider and setting, and assessment of the duration, intensity and content of services. Test–retest statistics for ever using children’s mental health services based on caregiver report ranged from K = 0.82 for school services to K = 0.94 for specialty inpatient services, and for use within the past year ranged from K = 0.79 for school services to K = 0.86 for any use (Horwitz et al. 2001). The validity of caregiver report when compared to treatment records was strong with K = 1.0 for inpatient use, K = 0.67 for outpatient use, and K = 0.31 for school use (Hoagwood et al. 2000).

The Services for Children and Adolescents-Parent Interview (SCAPI) was developed to assess parent report of services across multiple systems, settings, types of providers, and types of services in the National Institute of Mental Health Multimodal Treatment Study of Children with Attention Deficit Hyperactivity Disorder (MTA Study; Jensen et al. 2004). Additionally, the SCAPI collects information such as amount of services, start and stop dates, and type of system and organization within which the service is situated. Evidence of descriptive and criterion validity was found when comparing a large clinically treated ADHD sample and a local normative sample (Jensen et al. 2004). Test–retest for all services was high (K = 0.97) with 7 of the 10 service types having K values of 0.75 or higher (Hoagwood et al. 2004).

As described, these instruments (CASA, SACA, and SCAPI) have proven psychometric properties for measuring service utilization, and have been successfully used in children’s mental health research studies. However, complex instruments such as these with multiple screeners and modules can be unwieldy in clinical treatment studies, where large amounts of data from a variety of areas (e.g., developmental history, symptom assessment, family psychiatric history) need to be collected in a finite period of time to minimize participant burden. These instruments are specifically designed for large scale epidemiological studies whereas an instrument measuring TAU would need to collect more detail but in an easy format appropriate for clinical settings. Thus, it is important that instruments designed for use in clinical treatment studies provide a way to assess and record service utilization efficiently, accurately, and succinctly.

Measuring Medication Usage

Though several instruments for measuring service utilization have been developed, fewer options are available for collecting information on medication usage in children’s mental health. A review of the literature found no consistently used instruments for measuring medication usage even though many studies collected medication information. Additionally, the majority of the research focused on medication safety and efficacy but other areas related to medication usage by children have been neglected, such as the social and psychological experience of taking medications (Kilian and Angermeyer 1999). These are areas that may be significantly impacted by treatment or may impact clinical outcomes. Studies focusing on children’s mental health often develop their own measures for collecting data on medication usage, and collect basic medication information such as medication type and dosage and start and stop dates or use pre-existing sources such as billing data. Little or no information is reported about the format or psychometric properties of these measures. Both the SACA (Hoagwood et al. 2000) and the SCAPI (Jensen et al. 2004) include a section with basic medication questions. As a part of these larger service instruments, these two medication measures have proven psychometric property, but were not designed to be used separately and do not include questions about the child’s experience with the medication.

The Service Provider & Medication Usage Grids

Given the scarcity of published instruments to collect data regarding TAU, we developed two semi-structured interviews, the Service Provider Grid and the Medication Usage Grid for use in a clinical treatment study examining the efficacy of family psychoeducational psychotherapy compared to TAU in a sample of children with mood disorders. The framework for development of the Grids was that by assessing the basic details of treatment in addition to the families’ qualitative experience with the services and medication, the Grids would collect process and outcome information that could be used to understand the overall quality of services received by families (Noser and Bickman 2000). It was also hoped that the collection of both process and outcome treatment information would make this instrument applicable in both research and clinical settings. The review of available service utilization measures found several weaknesses of the instruments for use in clinical settings or in treatment research. As a result while developing the new Service Provider and Medication Usage Grids, the following issues were taken into consideration: (1) the unwieldiness of available service utilization instruments for both the researchers and families; (2) applicability for both research and clinical settings; (3) ease of reviewing and collecting information at follow-up visits (Kessler et al. 1980); and (4) importance of collecting information about families’ experiences with each service or medication (Kilian and Angermeyer 1999).

The Service Provider Grids were created as a concise way to assess, document, and track child experiences with treatment providers and educational services (see Appendix). During a semi-structured interview with a clinician, the parent or guardian is asked to recall institutions and individuals involved with their child’s care. For each service reported, information is gathered about the child’s age at the time the service was initiated, length of the intervention, perceived service provider effectiveness, how the treatment was helpful, and the reason for addition and/or termination of the intervention. Perceived effectiveness of services is rated on a 1–5 scale, with 1 being “not at all helpful” and 5 being “very helpful.” At follow-up assessments, the service provider grids are reviewed to assess for any changes in service provision.

The Medication Usage Grid was created to assess, document, and track a child’s past and current psychotropic medication trials (see Appendix). In a semi-structured interview with a clinician, the parent or guardian is asked to provide a list of the child’s medication trials. For each medication reported, information is gathered about the dosage, child’s age when the trial was initiated, length of trial, perceived effectiveness, and reason for discontinuation, if relevant. Perceived medication effectiveness is rated on a 1–5 scale, with 1 being “not at all helpful” and 5 being “very helpful”. After rating medication efficacy, parents are asked to provide a rationale for their rating. Parents also are asked to report any side effects their child experienced with each medication, as well as how side effects were managed. The Medication Grids are reviewed at follow-up assessments to update information.

Purpose of this Study

The pilot study presented in this article provides an examination of the utility of the Medication Usage Grid and Service Provider Grid for measuring TAU. Parents were interviewed during their child’s first visit to an outpatient treatment center using the Service Provider and Medication Usage Grids regarding their child’s past and current psychiatric and school service utilization. It was hypothesized that the Service Provider and Medication Usage Grids would contain the same amount or more information about services and medications than standard patient charts.

Methods

Participants & Procedures

Participants were recruited from a Midwestern medical center’s Child and Adolescent Psychiatry Outpatient Clinic. The outpatient clinic possesses a substantial percentage of the market share in the area for providing children’s mental health services. Each week approximately 25 intake evaluations are completed in the clinic; 10 of these are with children aged 6–12. A random sampling of outpatient evaluations indicated that a majority of the children are White (90%).

For inclusion in this study, parents had to have a child under the age of 18 attending the outpatient clinic for the first time. Study participants were 20 parents of 20 consecutively evaluated and eligible children at the outpatient clinic. Nineteen were mothers, one father participated. Children’s ages ranged from 7 to 16 (M = 10.85, SD = 4), 10 were female.

Parents and children were approached regarding study participation when they arrived for their first clinic visit. Prior to participation, parents gave written consent and children provided verbal assent. Interviews were conducted by a graduate research assistant after the parent and child met initially with their intake clinician to discuss the presenting problem. When the clinician met alone with the child to conduct a clinical interview, the research assistant met with the parent to complete the Service Provider and Medication Usage Grids interviews; completing the Grids took, on average, ten minutes (approximately 5 min for each Grid).

Following the visit, the graduate research assistant reviewed the child’s outpatient chart to summarize information documented regarding lifetime medication and service utilization. This information came from two sources: (1) parents had completed a standard clinic intake form mailed to their home prior to the first clinic visit. This form includes queries about the child’s living environment, developmental history, and family medical and mental health history as well as the child’s past or current mental health services; and (2) notes taken by clinician during the intake interview.

Data Analysis

Data analyses investigated the convergent validity of the Grids by comparing the Grids to standard clinic charts. The number of medication and service “matches” between the Grids and the chart was recorded. That is, any service listed on a Grid or in the chart was reviewed to determine if the other source also listed it. Any service noted via both methods was considered a match. Any service noted via only one method was considered a mismatch. For example, if the parent reported in the structured interview that the child had been on Prozac, the chart was reviewed to determine if Prozac was listed as a past medication trial. If Prozac was reported in both sources, this was considered a match. As some medication and service codes were rarely endorsed, it was not feasible to compute kappas. Instead, Spearman’s correlations were calculated to test the overall level of agreement between observations recorded via the two methods. Because the Grids collected additional information not available in the charts (e.g., ratings of perceived effectiveness), the reliability of that data could not be determined.

Results

Service Provider Grids

One hundred and eighty service observations for the 20 participants were collected from the Service Provider Grids and chart reviews. The overall correlation between services listed on the Service Provider Grid and in chart records was 0.92 (p < 0.01; see Table 1), indicating high convergent validity. In four cases, the Grid provided more information than the chart (i.e., services were reported during the structured interview that did not appear in chart data). The services in these four cases that were reported only on the Grids were psychologist, pediatrician, school psychologist, and cognitive testing. In five cases, the chart provided more information than the Grid (i.e., services appeared in chart data that were not reported during the structured interview). The services in these five cases that were reported only in the charts were pediatrician, social worker (2 cases), school psychologist, and other service. In 171 cases, information obtained on the Grid and chart was equal (i.e. services were similarly reported as occurring or not occurring on both).

Table 1 Comparison of service provider and medication usage grids to chart data
Full size table

Medication Grids

One hundred and eighteen medication observations for the 20 participants were collected from the Medication Grids and chart reviews. The overall correlation between medications listed on the Medication Usage Grid and in patients’ charts was 0.99 (p < 0.01; see Table 1), indicating high convergent validity. In one case, the Grid provided more information than the corresponding chart (i.e., medication was reported in the structured interview that did not appear in chart data). The medication in the one case that was reported only on the Grids was a stimulant. In no cases did the chart provide more information than the Grid. In 117 cases, data from the Grid exactly matched those found in the chart.

Discussion

The goal of this pilot study was to provide an initial evaluation of the utility of the Service Provider and Medication Usage Grids, two instruments created to measure TAU in a psychotherapy outcome research study. Results suggest the Grids are a valid means of gathering and tracking data about children’s past and current mental health treatment histories. Correlations between the service and medication data collected in the Grids interviews and chart reviews were high (Service Provider Grid: r s  = 0.92; Medication Usage Grid: r s  = 0.99) indicating agreement of information from the two data sources or convergent validity. These correlations of agreement are comparable or higher than those found in psychometric tests of the CASA and SACA (Ascher et al. 1996; Hoagwood et al. 2000). These results indicate the Grids provided basic information regarding medication and service history comparable to standard chart records; in addition, to more detailed information (i.e. reason for initiation/termination, side effects, helpfulness) that is not consistently recorded in chart records but is necessary when studying service quality.

The Grids provide additional information about medications and services not routinely collected in the charts (e.g., reasons for addition or termination, perceived effectiveness) that can be useful in evaluation of a client’s TAU and overall clinical outcomes. Research has identified the importance of these additional service variables, such as client satisfaction and client outcomes, and has pointed out the need for further research on these variables as important outcome variables (Cheon 2009). For example, the Grids were recently used by clinicians in a treatment study of children with mood disorders to determine how a psychoeducational intervention impacted the quality of the children’s TAU consequently affecting mood symptom severity (Mendenhall et al. 2009). The extensive information collected with the Grids was used by the researchers to rate the quality of TAU utilization on a five point scale where 1 = no services and 5 = optimal services.

Even though agreement was high, the Service Provider and Medication Usage Grids have several strengths that are absent in standard clinical charts and the other service utilization measures described previously in this article. The Grids provide a means for documenting mental health care services in a succinct manner by recording services or medications used historically or currently in a simple format. Both Grids are set up in a similar format making it necessary for interviewers to become familiar with only one basic format. The two Grids can be completed together or may stand alone as separate instruments. In the present study, completion of the Grids took ten minutes on average, which makes them feasible for use in both research and clinical settings, which may not be possible for the other currently available instruments that are longer and more complex. The one page format also requires less storage space. Even though the Grids are brief, they include questions about the child’s experience with each service or medication which is important information that is missing in many other instruments. Gathering this additional data is useful for assessing change in treatment utilization over time and analyzing reasons for these changes during the course of research studies and the overall quality of treatment. Another advantage of the Grids is the resulting concise summary of treatment that can be referenced at follow-up assessments to document changes in services since the previous assessment. Though this instrument was designed specifically as an instrument for a study on family psychoeducation, it’s research- and clinic-friendly format make it a viable option for assessment in a variety of treatment settings and with research on various intervention methods.

Grid Revisions

Based on limitations discovered during this initial pilot study, slight improvements have been made to the original version of the Grids. The newer, revised version of the Grids is included in the Appendix. In the original version of the Service Provider Grid, services were coded into nine categories: psychiatrist; psychologist; pediatrician; social worker; school psychologist; school services; cognitive testing; other; and none. Piloting of the Grid revealed additional categories of services, and in the current version of the Service Provider Grid, services are coded into the following categories: psychiatrist; psychologist; pediatrician/family physician; other therapist (social worker, family therapist, counselor); school services; school psychologist/counselor; psychological testing; inpatient hospitalization; residential treatment; respite care; bibliotherapy; online support groups; crisis management; other; and none. In the original version of the Medication Grids, six categories were created to code medication class: antidepressant; stimulant; anti-obsessional; mood stabilizer; other; and none. Piloting of the Grid revealed additional categories of medications, and the current version of the Medication Grid includes eleven categories for medication class: anti-depressants/anti-anxiety; stimulants; anti-obsessional; mood stabilizer; antipsychotics; sleep medication (prescribed); herbals; non-stimulant ADHD medications; other; and none.

Limitations

There are several notable limitations of this study. First, sample size was small and therefore may have not been representative of the population, which weakens the power and generalizability of the study. Second, one informant (parents) was used to provide two sets of information (Grids, chart information). As such, high levels of agreement would be anticipated. Additionally, relying solely on parent report does not guarantee accuracy of the information. Many parents had difficulty recalling exact dosage of past medications and may have forgotten to report past medications or services they had received to both the clinician and the research assistant. Thus, correlations may be inflated due to information forgotten in the interview and on the background information form. However, parents are still the best source of this information regarding their child’s treatment history. While it would have been preferable to have first hand information regarding medication and services from past treatment providers, this was not feasible, nor would it reflect standard clinical practice. Even though both sources were parent report, the data was collected at two different time points and in two different settings, the home where parents may have had access to medical records or paperwork and the clinical setting where parents probably relied on memory. Also limited recall on the part of parents reflects the clinical reality of service provision as past research has revealed poor parent precision and reliability for recall of longitudinal data (Angold et al. 1996). Finally, the same researcher conducted both the interview and the chart review which could be a limitation because the researcher was already familiar with the case and knew what to look for when reviewing the chart.

Future Research & Conclusions

Further psychometric validation of the Service Provider and Medication Usage Grids is recommended including use with a larger, more diverse sample and in other contexts (i.e. with more time between data collection, in inpatient settings). Additionally, studies are needed that assess test–retest stability and how Grids compare to the other service utilization instruments. While this pilot study was done in a children’s mental health outpatient clinic setting, we expect the Grids could also be useful in other mental health and health settings. Additionally, the simplicity of the Grids format may be suited to a computerized version that could easily be utilized in clinical or research settings. The increasing use of TAU as a variable or outcome of interest in treatment studies has led to a need for clear, concise, and accurate ways to collect, document, and track this information. The Service Provider and Medication Usage Grids provide an initial step towards achieving psychometrically sound measures to do so. With an accurate understanding of the usual services utilized by children with mental illness, we have a greater chance of enhancing the quality and fit of these services to consequently improve the overall health of the child and family.