In their letter, Coyne and colleagues [1] raise a number of questions and concerns about our recently published meta-analysis [2] regarding the efficacy of psychosocial interventions for breast cancer (BC) patients. The authors use the phrase “lack of transparency” frequently, but they do not define what they mean. Exactly what type of information were they looking for that was not provided? And should this information (whatever it might be) always be reported in meta-analyses? We would like to comment on the critiques Coyne and colleagues raised.

Critique 1

Coyne and colleagues criticize “a number of missing studies” which influenced our research questions of whether the efficacy varied as a function of the interventionist who delivered the intervention. Indeed the two studies mentioned by Coyne and colleagues were missing; however, one of these studies investigated a sample of breast cancer patients who had undergone autologous bone marrow transplantation [3] which we did not consider in our meta-analysis. We did a very careful research of the literature including screening for relevant studies in citations in identified articles and reviews. In every meta-analysis attempting to compile a variety of studies over decades, collecting all ever published studies remains difficult.

Critique 2

Lack of definition shall have led to the inclusion of a mélange of studies which are presumably not really psychosocial.

Coyne and colleagues criticized the missing definition of “psychosocial intervention.” Psychosocial intervention is a well-known term in the behavioral medicine literature [4] defined as an approach aimed at improving people’s well-being that uses cognitive, cognitive-behavioral, behavioral, and supportive interventions. These include patient education, interventions aimed at aiding relaxation, psychotherapy, and structured or peer support. The two studies included in our meta-analysis mentioned by Coyne and colleagues evaluating a tour of the cancer center and a cosmetic class were coded as follows: The study of McQuellon et al. [5] included besides the clinic tour also general information about clinic operations and a question and answer session with an oncology counselor which justified the classification as “educational” from our point of view. As described in our coding procedure, we defined “psychoeducational” according to Fawzy et al. [6] as “treatments primarily providing information about the nature of the cancer and its medical treatment....” [p. 227, 2]. The study of Manne et al. [7] investigating the efficacy of a cosmetic class was only considered for calculating the overall ES because it was indeed categorized as a psychosocial intervention—as it was also viewed by the authors themselves because “cosmetic instructions alone reduced depression and improved self-concept” (p. 85), but it was not included in the moderator analyses as can be seen from Table 1. Table 1 demonstrates that this study has been classified as “other” (which means it was not categorized as one of the primary intervention types) and was therefore not further considered. Thus, we believe that Coyne et al.’s conclusion that we included a “mélange of studies” is unfounded.

Table 1 Articles included in the meta-analysis
Full size table

Critique 3

Decision to collapse effect sizes across different outcomes.

This comment has been the focus of debates across fields where such an approach was used in meta-analysis. Thus, we consider this comment not to be specific to our meta-analysis but to all meta-analyses in general. For example, would it be appropriate to include “depression” as an outcome, regardless of the depression measure? How about “distress,” using a definition that might include anxiety, worry, depression, etc.? The problem of multiple outcome measures per study in meta-analysis has received attention in methodological research. The optimum approach of accommodating several effect sizes (ES) per study without inflating its impact is a multivariate model [8]. However, this technique requires reliable estimates of inter-measure correlations which is usually not available for all combinations or based on normative samples not sufficiently similar to the study population [9]. Thus, averaging multiple ES to obtain one effectiveness estimator per study is recommended [10, 11]. Coyne and colleagues can question the comparability of various instruments collapsed into one ES in our meta-analysis [2]; yet, all measures incorporated were related to mental well-being, partly specific (e.g., CES-D), partly comprehensive (e.g., POMS). Our goal was to evaluate the overarching effectiveness of psychological interventions for mental health problems, and, thus, we consider our strategy to be appropriate. ES as a standardized index represents a metric of change independent of the original scale; we consider our method suitable to assess this question. Furthermore, if we had calculated ES separately for different domains, we would have had an overall lower number of studies because not all studies assessed all types of domains leaving us with fewer studies to investigate potential moderators.

Critique 4

Coyne et al. raised a concern that samples with even one non-BC patients were considered as heterogeneous and that this definition was flawed and misleading to resolve whether it is better for BC patients to be mixed with other cancer patients or not. In response to editorial feedback, we revised Table 1 by removing the type of cancer in the heterogeneous group. For this reason, the other cancer types are not outlined in detail, but we are glad that we have the opportunity to provide this information now: there is no study with a sample of almost all BC patients with 1–2 other cancer patients as questioned by Coyne et al. In 82% of the heterogeneous studies, the samples consist of more than 50% of patients with other cancer types than BC. There were four studies with less than 50% of other cancer types, but at least 20% of these samples had a diagnosis other than BC. Had Coyne et al. checked the studies before criticizing our approach, they would have seen that their concerns were unfounded. On a broader level, any time it is necessary to dichotomize a variable, there is always judgment involved as to where to make the cutoff; in our case, both conceptually and pragmatically, the cutoff we employed is appropriate.

Critique 5

Coyne et al. criticized that the methodological quality of studies was not taken into account. We believe that to conclude that earlier studies were of poorer methodological quality without reliably and validly assessing that quality is inappropriate. To our knowledge, a precise agreed-upon method for how to handle the problem of methodological heterogeneity of studies combined in meta-analysis is nonexistent. Our reluctance to integrate methodological quality stems from extensive disagreement in the literature on how to assess trial quality which has lead to a multitude of different scales and checklists [12], providing hardly any fool-proof rationale on how to select among such strategies.

Critique 6

Coyne et al. pointed out what they view as two critical errors in calculating ES: (a) for calculating the ES from the Helgeson et al. study [13], we used standard errors instead of standard deviations, and (b) we double counted two studies which were subsequently incorporated [5, 14]. There were no hints in the study of McQuellon and colleagues [5] that part of the sample was already reported in Wells and colleagues paper [14]. Quite the contrary, the samples of the two studies received different interventions which make it unlikely that one sample was incorporated in the other. The critique that different outcome measures for the same sample were entered separately is only for two studies [15, 16]; yet, there were no indications in those investigations of using the same samples for the Maguire and colleagues study [17, 18] or for Goodwin et al. [19–21]. It remains unclear why Coyne et al. concluded that multiple studies were double counted in our analysis. It is correct that in two cases data from the same study were included separately. We recalculated the ES without the two double-counted studies. No changes were found for the overall ES. All the studies were classified as supportive and homogeneous. A reanalysis of the ES for supportive showed a slight change of the ES from d = .13 to d = .12; for homogeneous from d = .14 to d = .13. Two studies were classified as nurse-led interventions and two as psychologist-led interventions. The ES changed from d = .15 to d = .13 for nurse-led interventions and from d = .30 to d = .31 for psychologist-led interventions. Therefore, these minor changes do not warrant a new interpretation of our results.

Coyne et al. criticized that we misclassified the Goodwin study [20] as cognitive behavioral. However, we consider an intervention including development of coping strategies, improvement of communication skills, and confrontation techniques as cognitive-behavioral because it teaches “various active coping strategies and changes specific thoughts or behaviors” as defined in our meta-analysis (p. 227). We are rather confused and surprised that we would be accused of “sloppily done analyses.”

Coyne et al. also proposed that our conclusions would be primarily based on outliers. However, we recalculated the ES excluding two outlier studies (with ES 2.66 and 2.30) already in response to a reviewer comment during the publication process. And whereas the ES’s for both CBT and EDU decrease somewhat, the difference in ES does not change at all. Given that for moderation analyses we want to explore variability across studies, we have retained these investigations in the analyses. Due to page limitations, we have not included these additional analyses in the final published version of the current meta-analysis.

The study of Nezu et al. [22] was not misclassified as psychologist-led as suggested by Coyne et al. because “individuals who conducted the clinical interviews included advanced clinical psychology graduate students who had the equivalent of a master’s degree and participated in a 10-h training protocol led by either Arthur M. Nezu or Christine Maguth Nezu” (p. 1038) which warrants classification as “psychologist-led” from our perspective.

Critique 7

Our Conclusions

First, we conclude that some aspects of the Coyne et al. critique are relevant to the procedure of meta-analyzing empirical studies in general. We have clearly laid out our methodology in the manuscript and remain confident in the conclusions we drew. Yet, given the evidence we obtained, other researchers might come to different conclusions, which is one of the reasons for separating “Results” from “Discussion” in a journal article, with the latter obviously containing interpretational remarks. We thank Coyne et al. for implicitly pointing to the necessity for thorough reading and understanding of research papers rather than skipping the tedious statistics and only focusing on the authors’ interpretation of the results in the discussion section. We are confident both of our results, and we, along with very thoughtful reviewers, have concluded that our interpretations of our results are appropriate.

Second, we conclude that some of the critiques that Coyne et al. apply specifically to our meta-analyses are a matter of opinion or ongoing scientific dispute. We would hope that reasonable differences in interpretation of results and ongoing debates about meta-analysis could be approached in a professional, respectful manner.