Introduction

“Social validity” is a term introduced in the behavior therapy literature to refer to the value placed by consumers on therapeutic procedures received and the success of those procedures (Baer et al. 1987; Kazdin 1977; Wolf 1978). In 1995, Poling and LeSage (1995) surveyed 68 drug studies in the mental retardation literature that were published between 1987 and 1993. They found no trials providing social validity data in the form of parent, other caregiver, or patient satisfaction. Poling and LeSage observed that obtaining such data should add little cost or difficulty to most pharmacological trials and that inclusion of such findings would enable us to place the findings in a “broader, socially significant, context”.

In a double-blind, placebo-controlled trial of risperidone for the treatment of behavioral difficulties in individuals with developmental disabilities, McAdam et al. (2002) found that 100% of the caregivers felt the participation was a positive experience for themselves and the participants and that 88% were satisfied with the conclusions reached for their child and, in general, responded positively to all the questions regarding the medication trial. In a recent review of treatment satisfaction in trials involving children with attention deficit hyperactivity disorder (ADHD), eight reports were identified (Bukstein 2004). In general, parents were highly satisfied with their research experiences. In the first-ever pharmacological–social validity study involving pediatric subjects, Aman and Wolford (1995) found a high level of satisfaction among parents whose children participated in two trials involving methylphenidate and fenfluramine in children with mental retardation and ADHD. The remaining studies, which occurred between 1993 and 2003 (Johnston and Fine 1993; Pelham et al. 2001; Wolraich et al. 2001; Biederman et al. 2002; Dirkson et al. 2002; Wan and Bukstein 2003; Wilens at al. 2003), evaluated parent satisfaction with types of psychostimulants and various delivery systems. Thus, these studies examined satisfaction with the product rather than the research experience per se. The National Institute of Mental Health Multimodal Treatment Study of Children with ADHD (MTA) found that parents and teachers rated satisfaction with intensive multi-component behavioral treatment alone significantly higher than medication management alone (Arnold, 2005, personal communication, October). By contrast, these same raters scored medication alone significantly better on scales of ADHD symptom severity (MTA Cooperative Group 1999), suggesting a disparity between outcome and satisfaction. We were not able to locate any other published reports of parent satisfaction with respect to child psychopharmacological research.

In the context of this paucity of information about parent satisfaction with psychopharmacological studies, there is increased scrutiny on the use of placebo in randomized clinical trials (Frank et al. 2003). Recent years have witnessed an increase of oversight by institutional review boards (IRBs) on all forms of research. In particular, the ethics of using placebo in psychopharmacology has been the subject of continuous debate. The need to rely on data rather than theoretical assumptions, on one hand, and the dearth of empirical studies addressing the acceptability of placebo, on the other hand, have been pointed out. Vitiello (2003) posed the following as key questions especially relevant to research involving children and adolescents: (a) How do children and their parents perceive the research experience?, (b) Do subject characteristics predict response to the research experience?, and (c) Does placebo administration lead to negative health outcomes?

The satisfaction data reported in this paper were obtained during a randomized clinical trial of risperidone in children with autistic disorder and high levels of irritable behavior. At the end of the acute 8-week double-blind phase of the risperidone trial, we asked parents to rate satisfaction on a questionnaire to evaluate subject and family burden, the impact of ethnicity on the clinical trial experience, parental view on the use of placebo, and the association of patient outcome and parent satisfaction.

We hypothesized that, in general, parents/guardians would have moderately high satisfaction with the study experience and that parents of children showing improvement during the trial would be more satisfied than those whose children did not improve.

Subjects and methods

Subjects

This study was approved by the ethics committees of Johns Hopkins University, Ohio State University, NIMH, Yale University, Indiana University, and the University of California at Los Angeles. This study was performed in accordance with the ethical standards laid down in the 1964 Declaration of Helsinki. The parents or guardians of all subjects gave their informed consent before their inclusion in the study. In general, subjects were not cognitively capable of giving assent.

One hundred one outpatients with autistic disorder aged 5 to 17 years (mean + SD, 8.8 + 0.7) who had serious disruptive behavior such as aggression, self-injury, and severe tantrums were randomized to receive either placebo or risperidone in an 8-week double-blind, parallel-groups clinical trial. Of these 101 subjects, 80 completed the full 8-week clinical treatment period, and the parents of 96 (95.0%) completed the parent satisfaction form, 47 of which were assigned to placebo and 49 to risperidone.

Sixty-six percent of the respondents were White, 11% were African American, 7% were Latino, 8% were Asian, and 8% were of other racial/ethnic groups. Parents were required to be fluent in English to complete the clinical trial assessments. Of the parents, 42 (45%) reported an annual income above $40,000. Twenty (21%) had no more than a high school education, 28 (29%) had attended some college or trade school without achieving a degree, and 48 (50%) had a university or advanced degree. Active medication (risperidone) was markedly superior to placebo, as evidenced by a positive response rate of 69% on risperidone vs 12% on placebo (p < 0.001; RUPP Autism Network 2002), determined by improvements on the severity domain of the Clinical Global Impression (CGI) scale (Guy 1976) and on the irritability subscale of the Aberrant Behavior Checklist (ABC; Aman et al. 1985). Payment was made to the families for their travel expenses, and some sites provided payments to parents to offset costs related to the study, such as sitter expenses and time lost at work.

Measures used in the clinical trial

The measures used to determine inclusion eligibility were the diagnostic and statistical manual (DSM)-IV (APA 1994), Autism Diagnostic Interview—Revised (ADI-R; Lord et al. 1994), CGI, and the ABC. All subjects met the criteria for autistic disorder by both the DSM-IV and the ADI-R. Individuals with other types of pervasive developmental disorders were not included. The primary outcome measures were improvement in the severity scale of the CGI and the irritability subscale of the ABC. In addition to those measures, the parents completed the modified Children’s Yale–Brown Obsessive Compulsive scale (CY-BOCS; Scahill et al. 1997; Scahill et al. 2006) and 13 additional measures. Learning tests were performed to assess cognitive development at baseline and at the end point.

The ABC has 58 items that were completed by parents on a 4-point Likert scale (0 [not a problem] through 3 [the problem is severe]). Its five subscales are designated as: I, irritability; II, lethargy/social withdrawal; III, stereotypic behavior; IV, hyperactivity; and V, inappropriate speech (Aman et al. 1985). The modified CY-BOCS for pervasive developmental disorders was used to evaluate the severity of compulsive behavior (Scahill et al. 2006). These and the remaining elements of the battery are described elsewhere (Arnold et al. 2000; McDougle et al. 2000).

The complete protocol called for the screen, a baseline, and eight consecutive weeks of assessment. The screen visit usually required between 5 and 7 h to complete, the baseline assessment typically required 90 to 120 min to finish, visits for weeks 2 to 7 required between 35 and 65 min to complete, and the endpoint visit at week 8 (or the last visit if the subject had to discontinue the study medication) lasted 3 to 5 hours. The study medication was adjusted upward according to a weight-based schedule during weeks 1–4, and dosage was held constant from weeks 5–8 (described in detail in Scahill et al. 2001). Blood was drawn at the beginning and end of the study, and the parents were invited to assist in the process.

The instrument used in the satisfaction study

At week 8 (end of the acute double-blind trial) or upon early termination, parents were given a 10-item questionnaire, which was adapted for this trial from Aman and Wolford (1995), with a 3-point Likert scale (Table 1). The identity of the respondent was recorded on the survey form.

Table 1 The Parent Satisfaction Questionnaire and number of responses (%) to each question
Full size table

Statistical analyses

The satisfaction data were examined by frequency counts and percentages. Relationships between expressed satisfaction and demographic characteristics of the participating families were assessed by Fisher’s exact tests. To create subgroups large enough for statistical analyses, the satisfaction categories were collapsed from three to two categories. The most positive category (e.g., “just right”) defined positive response; the remaining two categories (e.g., “too many” and “not enough”) were combined into another subgroup.

Exploratory analyses considered the possible relationship between magnitude of improvement in the child and expressed satisfaction by the parent. Such an association might suggest that the behavioral changes that parents perceived influenced their appraisal of the aspects of the study. To address this issue, we calculated two new indexes for the CGI severity subscale, ABC subscales, and CY-BOCS total score. The change score was computed by subtracting the week-8-treatment score from the baseline score. The ratio score was computed by subtracting the week-8-treatment score from the baseline score and dividing this by the baseline score [(BL-W8)/BL]. In both cases, larger numbers indicated greater improvement with treatment. To conduct these analyses, we first calculated the ABC subscales and the CY-BOCS total score. Because the risperidone and placebo treatments constituted very different treatment experiences, we analyzed each set of variables within the risperidone and placebo groups. In each case, the comparison group of interest was composed of the parents who did not endorse the “satisfaction” measure versus those who did.

Results

Satisfaction ratings

The results for the satisfaction questionnaire appear in Table 1. Parents appeared to be highly satisfied with the clinical trial experience, with a range of 80.0 to 96.8%, expressing a high degree of satisfaction in the answers to the questions. Questions 7 and 8, which asked if the parents would recommend the clinical trial to other families and if parents would participate in the clinical trial again, seemed most relevant to the social validity issue. On question 7, 90.7% said that they “definitely” would recommend the study to other parents, 8.2% said that they would but with some reservations, and 1.1% said that they would not. On question 8, 92.7% said that they would (definitely) participate again, 7.3% said that they might, and no parents said that they would not participate again.

Participants also had an opportunity to make comments in two open fields on the questionnaire (items 9 and 10 in Table 1). Nearly equal numbers of positive and negative comments were made (24 vs 25). Of participants who commented on aspects of the clinical trial that they disliked, (a) six subjects did not like some component related to the travel involved, (b) six had some reservations about the service received, (c) three said the clinical trial was too time-consuming, (d) three had problems related to manipulation of the clinical trial medicine, (e) three were uncomfortable with the blood draws, (f) two experienced difficulty with the questionnaires, and (g) two stated that the structured diagnostic interview (ADI-R) was too long.

Among the aspects of the clinical trial that participants considered valuable were the following: (a) two commented favorably on the effects of the medicine, (b) five liked the weekly evaluations, (c) three appreciated the thoroughness of the assessments, (d) three appreciated the information they acquired on autism, (e) three stated that they liked “all aspects,” (f) eight liked some aspect of the following: the physical exam, staff interactions, amount of time devoted to assessments, staff support, quality of communication, and/or overall effort toward subjects’ well-being.

Association between subject and family characteristics and satisfaction ratings

We also used Fisher’s exact tests to compare family demographics with satisfaction ratings, and the significant comparisons are presented in Table 2. Families with higher incomes expressed more satisfaction on the item concerning the number of visits than low-income families. Regarding satisfaction with the learning tests: (1) Parents of children with more severe behavioral difficulties (baseline CGI severity score of 6 or 7) were less satisfied with the learning tests than parents of children with lower CGI severity scores; (2) Parents of children with higher IQs were less satisfied with the learning tests than parents of children with low IQs; (3) Parents of white subjects were less satisfied with the learning testing than the parents of minority children; and (4) Parents with university degrees were less satisfied with the learning tests than the parents who had no college degree.

Table 2 Summary of significant demographic information in relation to satisfaction ratings on individual items of the Parent Satisfaction Questionnaire
Full size table

Next, we reported two findings that, although not statistically significant, were included because of their salience in discussions of the ethics of conducting drug studies in children. Across several comparisons, there were no significant differences in the level of satisfaction by drug assignment (placebo vs risperidone). This finding of no difference by treatment group included items on whether or not parents would recommend the clinical trial to other families; indeed, the trend was for parents whose children were assigned to risperidone to be less likely to recommend the clinical trial. Similarly, there was no significant association between drug assignment and willingness to participate again. Families whose children were assigned to risperidone were slightly less likely to indicate that they would join such a clinical trial again.

Perceived change and satisfaction

Tables 3 and 4 summarizes these results. For the placebo group, parents who did not feel that the use of behavior rating instruments was optimal (n = 9) tended to see substantial improvement (4.33 points) on the stereotypic behavior subscale of the ABC, whereas the 38 parents who indicated high satisfaction with the rating instruments perceived relatively small reductions in stereotypic behavior (1.29 points) on the ABC. For the placebo group, greater change (6.50 points) was recorded for ABC stereotypic behavior change among the four families who did not regard the number of visits as optimal, whereas less change (1.44 points) was recorded for the 43 families who endorsed the number of visits. Whereas low endorsement was associated with more improvement in the variables that were repeated across analyses (immediately above), high satisfaction was always associated with greater improvement within the risperidone group (see Table 4).

Table 3 Significant associations between parent satisfaction scores for subjects assigned to placebo and perceived change during the study as measured with the Aberrant Behavior Checklist, CY-BOCS, and the Clinical Global Impression scale severity score (t scores, degrees of freedom, and probability values are detailed below)
Full size table
Table 4 Significant associations between parent satisfaction scores for subjects assigned to risperidone and perceived change during the study as measured with the Aberrant Behavior Checklist, CY-BOCS, and the Clinical Global Impression scale severity score (t scores, degrees of freedom, and probability values are detailed below)
Full size table

Discussion

Levels of satisfaction reported

The levels of satisfaction reported by parents in this study were high, ranging from a low of 80.0% (endorsement of learning assessments as “very important”) to 96.8%, reporting satisfaction with the assessments for adverse effects. The two key social validity questions: (a) whether parents would recommend the clinical trial to other parents and (b) whether families would choose to do the clinical trial again, showed that 99 and 100% of parents either fully endorsed or partially endorsed those choices. Only one family indicated that it would not recommend the clinical trial to other parents having children with similar problems. Our findings are slightly higher than that of two previous studies that posed the question about whether parents would participate again. Aman and Wolford (1995) reported that 88% of parents said that they would choose to join the clinical trial again, whereas McAdam et al. (2002) reported that 82.4% of families whose children participated in a risperidone trial indicated they would choose to join the clinical trial again. It is not clear whether the higher expressed social validity in our study was due to the population studied (i.e., children with autism and highly disruptive behavior), the medication, the intensive monitoring, or to some other factor.

Bukstein (2004) reported a range of 62 to 87%, across trials, of parents who were moderately or very satisfied with the clinical trial. Hence, the percentages in this study of those who stated that they would definitely recommend the trial (90.5%) and would definitely choose to participate again (92.7%) appeared to be high in relation to ADHD clinical trials with psychostimulants. In the MTA study, 69% of parents whose children were in the medication management group and 91% of those in the behavioral treatment group stated that they would recommend or strongly recommend the treatment (MTA Cooperative Group 1999). In that same study, 70% of those in the medication management group and 95% of those in the behavioral treatment group were satisfied or strongly satisfied with the treatment (Arnold, 2005, personal communication, October).

In terms of “open-ended” responses, parents indicated a total of 25 features that they disliked and a total of 24 features that they liked. Some write-ins were aspects over which researchers have little control, such as hassles related to traffic and parking (although parking and travel were reimbursed by some sites). There were several open-ended comments about the difficulties of the drug study, such as consumption of too much time (n = 3), phase-out of prior medicines or manipulation of clinical trial medicine (n = 3), and drawing of blood samples (n = 2). However, it is interesting that many or more parents commented favorably on some of these same categories such as favorable reaction to the minimum of nine scheduled weekly visits (n = 5), positive reaction to the clinical trial drug (n = 2), liking the “thoroughness” of clinical trial assessments (n = 3), and endorsing “all aspects” (n = 3). Thus, although there were some complaints regarding subject burden, at least many families seemed to like the fact that numerous aspects of pharmacological effects were being assessed. The concern over the blood draws is highly understandable in light of the fact that it was often difficult or impossible to explain the need for the blood tests and/or to reassure the children verbally, given that many of the subjects had mental retardation and/or were nonverbal. One unanticipated benefit reported by three families was education about medication and autism received during the trial.

It is interesting to consider these favorable responses in combination with the level of knowledge demonstrated by the parents of these children. In a separate article from this same sample, Vitiello et al. (2005) reported a high level of parental knowledge about the benefits and risks associated with the clinical trial. When questioned at the end of the trial, parents clearly understood the possibility of placebo assignment, their right to withdraw at any time, the medication’s main side effects, and possible alternatives to research participation. These and other results (e.g., Aman and Wolford 1995; McAdam et al. 2002) may reassure IRBs and the public that many research participants feel positively about their experiences.

Family demographics and satisfaction ratings

Relatively few demographic features were associated with stated satisfaction level of the parents. Parents of low-income families expressed less satisfaction with the number of visits, with six families indicating that there were too many visits and one family indicating that they were not enough. It may be that lower-income families experienced more difficulty and/or expense in missing work than high-income families. The relationship of the child’s IQ (above or below 45) to endorsement of the learning tests indicated that the parents of higher-functioning children were less likely to endorse the importance of the learning tests. We can hypothesize various reasons. It is possible that the higher-functioning subjects had more disruptive behavior when demands were placed upon them. Higher-functioning children may have had more prior testing so that the additional testing in the clinical trial was not seen as informative. In this clinical trial, the higher-functioning subjects were more likely to be tested to a greater degree (because they were able to do the tests) and may have protested when such demands were made. The only significant finding in relation to ethnicity concerned parents’ responses to the learning tests in which more nonwhite than white parents gave high endorsement to the use of learning tests. It is possible that a greater percentage of the minority families had children enrolled in poorer school districts, which did not have testing and other services readily available. In particular, IQ or educational testing might have been helpful to the families to obtain additional school services and/or information about their children. However, the analysis of income by endorsement of the learning tests was not significant.

Parental education was associated with the level of support for behavior ratings, with a greater percentage of parents without college degrees expressing lesser importance for behavior ratings. This may indicate a need, in future studies, for investigators to provide an explanation to parents about the value of standardized instruments for evaluating behavior throughout the trial. This finding may also reflect reading difficulty for some parents. Greater awareness of this possibility on the part of investigators may be useful for such parents.

Influence of placebo assignment and treatment response

The use of placebo control in pharmacological studies in children with serious behavioral problems has been a much-debated topic. Most observers conclude that placebo-controlled trials are justified (Vitiello 2003; Derivan et al. 2004; Roberts and Krystal 2003). However, some investigators contend that we should move toward comparison of the two active treatments in part because of the concern about parental acceptance of placebo assignment (Gilbert and Buncher 2005). Families in this clinical trial did not reflect this view. For example, drug assignment (placebo vs risperidone) did not influence parental willingness to recommend the clinical trial to others or their stated willingness to join the trial again. These findings suggest that parents of children assigned to placebo found this control condition quite acceptable.

In addition, the status of the child as a clinical responder or not did not sway parents’ ratings of their satisfaction with clinical trial procedures. Some 5% of parents of the responders said that they “might” have some hesitation in joining the clinical trial again as compared with 8.9% of parents of the nonresponders (Table 2). A possible design feature that could have enhanced parents’ acceptance of placebo assignment is that all participating families were offered an open trial of risperidone if the child was initially assigned to placebo and did not respond. Failed treatment on placebo could legitimately be seen as a benefit, as parents would be able to compare the placebo condition with the open medication phase. This observation suggests that clinical trial designs that are responsive to consumer concerns will encounter less difficulty in gaining acceptance of placebo assignment.

Correspondence between behavior change and reported satisfaction

One reasonable hypothesis could be that marked positive behavioral change would be associated with strong expressions of parent satisfaction. The only variable that showed consistent relationship between behavior change on one hand and parent satisfaction on the other went in the opposite direction to this hypothesis. In the case of ABC stereotypic behavior change, greater behavioral improvement was related to weaker endorsement of both the value of behavior ratings and the value of the number of visits for subjects assigned to placebo condition and for all subjects pooled (Tables 3 and 4). Inconsistently found changes also appear in Tables 3 and 4 as “exploratory” results, which may be informative for future investigators. It is interesting, however, that all associations with active medication (Table 4) indicated that less endorsement was associated with less improvement. Thus, preliminarily, these associations appear to be critically linked to whether treatment was active or placebo.

Limitations of satisfaction study

We obtained satisfaction data during the last visit in the acute clinical trial. Other investigators have proposed that allowing some time to elapse between the end of the trial and the satisfaction survey could reduce pressure on parents to give favorable rating (Aman and Wolford 1995). Thus, our satisfaction survey may have been influenced by subtle pressure felt by parents to “say the right thing” in our presence. Another possible limitation is that this satisfaction survey did not address a multitude of potentially important issues such as acceptability of randomization procedures and use of alternative designs. Another limitation is that multiple comparisons were run to derive the “significant” results presented in Tables 3 and 4. If the alpha levels were adjusted by a Bonferroni correction, none of the findings presented in Tables 3 and 4 would be significant at a p value less than 0.0005. We present these associations for subsequent confirmation or disconfirmation in future research.

Implications for IRBs

The high level of acceptance expressed by parents in this satisfaction study may have implications for IRBs and others concerned about the pros and cons of placebo-controlled trials in children with serious psychiatric and developmental disorders.

Conclusion

Parents who view experimental procedures as worthwhile are more likely to participate in clinical trials and comply with study procedures. Designing studies that consider the consumer’s perspective will help to forge a better alliance between participants and researchers. Participants who accept the legitimacy of study procedures are more likely to adhere carefully to often-demanding protocol guidelines. Satisfaction surveys such as this must be capable of identifying points of disagreement or dissatisfaction when they occur and offer the possibility for parents to provide feedback, leading to better trial designs in the future. After a body of such data is compiled, we can study the protocols themselves for ingredients that would likely lead to high satisfaction. In this way, researchers can begin to develop a “culture” where participants and investigators interact to produce a better product, namely, rigorous but acceptable studies with valid outcomes. To achieve this, researchers should consider adding social validity questions to their investigations whenever possible.