I thought I would never write this in “public” but one of the paradoxical problems with our mental health services is that they are not visibly harmful. Most of the research shows that treatment as usual (TAU) is neither harmful nor effective. Ineffective mental health services do not usually produce dramatic negative outcomes. In most cases the client will drop out of services, especially in the public system where the no show and dropout rates are phenomenally high. Certainly there are suicides and assaults and sometimes killings. However these incidents are usually attributed to inadequate supervision or custody rather than to ineffective treatment. Unlike the surgeon who leaves an instrument in a patient or a plumber who leaves a leaky fitting, there is no trail left by unsuccessful treatment. But poor outcomes are not buried only because they are insufficiently dramatic or visible. We hide the ineffective services from ourselves and the public by not collecting information that measures their effectiveness. Even the clients may not be aware that they have received ineffective services.

TAU is a Commodity without Consideration of its Quality or Effectiveness

If you were in the market for a car, would you buy it based only on the following information: It is located on 5th and Main, costs $20,000, is 2 years old, is a Ford, and seats six? Whenever I give that example in a talk I don’t get a single potential buyer. The reason is that there is not any information about important factors such as its performance. Now let’s look at what we know about mental health services. We know the location, the charges, how long the organization has been inexistence, whether it is inpatient, day treatment or outpatient and just maybe how many clients it serves. Like the car, we know nothing about its quality, performance, or durability. Yet that is how we buy and sell mental health services. Additional information may provide the impression of quality such as accreditation, licensing, clinician degrees and years of experience, but these have yet to be related to effectiveness. Clinical care is a commodity, like barrels of oil that sell by the unit without regard to individual quality characteristics. That is not to say there are no differences in prices. An inpatient facility may charge $50,000 a month but would be challenged to provide scientific evidence of its cost effectiveness. Recommendations for treatment are made based on less information then we possess about my imaginary car or even a toaster. We know little to nothing on how to even describe TAU and even less about its effectiveness. Moreover, what we do know does not provide support for assuming they are effective. Why?

Are There Technical Barriers to Measuring Effectiveness?

An acceptable reason for why we know so little about the effectiveness of services would be we don’t know how to judge them. However, the measurement of clinical outcomes has been possible for decades. True, many of the measures and techniques were not suited for real-world services but there is little demand from the field to develop such measurement systems. In the last several years a small number of researchers have been fielding systems that provide not only measurement of progress but clinical feedback as well. Such systems, which I simply call measurement feedback systems (MFS), have not been greeted with much enthusiasm and support. While these systems are far from perfect, my own sense of frustration with their low acceptance is fueled by the low priority placed on the adoption of MFSs by funding agencies, providers, consumers, and clinicians. Even relatively simple outcome measurement is rarely used to evaluate the effectiveness of services.

Do We Assume Services Are Effective?

Another suitable reason for not measuring outcomes would be that we already know that services are effective so we don’t have to monitor them. I think only those who have very effective systems of denial could believe this. I am specifically addressing the child and adolescent field as my knowledge of the effectiveness of adult mental health services is more limited. Since this is an editorial I will not review the literature in this area so you will need to investigate this yourself or take my word for this conclusion. Regardless, I don’t think anyone (at least I hope not) believes that mental health services delivered in the real world are foolproof and work without monitoring, even though it is our current situation. There is little reason to believe that attending a workshop on evidence based treatment (even for 2 days) will result in significant changes in practice without monitoring both adherence and client outcomes. The continuing education approach to changing practice has not been successful in education or general health. Clinician proofing treatment will not be any more effective than teacher proofing curriculum. We are long way from infallible treatments. We should not avoid outcome measurement because we assume that services are effective.

We may think it is too difficult or already believe that mental health services are effective but I don’t think those are the major reasons for the lack of measurement of outcomes. I will describe four reasons why I think there is a dearth of outcome measurement and feedback or use of this information.

There is No Financial Benefit in Delivering More Effective Services

Most mental health services are delivered in an environment where there is no systematic measurement of outcomes and thus no valid way to measure effectiveness. Given lack of information it is apparently assumed that all services are equally effective or ineffective. It is rare for a service to be paid more if it delivers more effective services. Thus, avoiding a MFS is financially the only thing that makes sense. Implementing a MFS is going to cost more and thus reduce the profit of a profit-making company, or weaken the chances of survival of a non-profit in this era of tight funding. Increasing accountability by systematically measuring outcomes is a cost that has no benefit in the current system. Only the rare progressive organization would take on this challenge. These organizations can hope that better outcomes or even the mere existence of a MFS will eventually be rewarded by funding sources. There is beginning to be some movement in that direction by some managed care companies.

Everybody benefits with the lack of measurement of effectiveness that is everyone except the client. Funding agencies can point to inputs instead of outcomes as a source of pride. How many people were treated, how much money was spent, and how many different services were used appear to be acceptable markers of the delivery of good services. Service delivery organizations can, of course, use the same “outcomes” but also add another measure that is often confused with a clinical outcome–satisfaction. But more importantly, they do not have to be accountable for outcomes and thus they avoid the problems and costs of not only implementing a MFS but dealing with the problems that it will certainly uncover. Clinicians can continue to believe in their effectiveness without any contrary evidence. The consumer may obtain some short-term gain from believing in the effectiveness of treatment but this placebo effect, if it occurs, does not endure. Consumers need to press for outcome measurement and not just more services. Increasing access to ineffective services is not a desirable goal.

Evidence Based Treatments (EBTs) and Protocols Serve as a Substitute for a MFS

EBTs and treatment protocols appear at face value to be significant improvements over TAU. However, there is a danger that these innovations will serve to displace the need to measure outcomes. Ongoing monitoring of adherence is required if EBTs are going to have a chance of being effective. Moreover, since many of the EBTs were developed under conditions that do not represent the environments, clinicians, or clients it is probable that they may not be of optimal effectiveness without local adaptations. Without a MFS such needed information to know whether adaptations are needed and/or successful will simply not be available.

The establishment of “approved” interventions by professional associations and government agencies is at best premature and at worst misleading. To label an intervention as effective without knowing anything about the moderators of the treatment is to assume that it works equally well under all conditions for all clients and with all clinicians. Moreover, the approach taken by some organizations presents a very low bar for being “approved.” For example, in one system the highest rating is being “well established” and requires only two independent studies and does not consider any contrary evidence. It is also questionable if “well established” is a dimension of effectiveness. Furthermore, the evidence that EBTs consistently outperform TAU is far from unequivocal.

Accreditation and Licensing Serve as Substitutes for a MFS

Two of the most widespread mechanisms designed to ensure quality of services do not have sufficient scientific evidence to demonstrate that they accomplish this aim. Very few of these licensing and accreditation processes require the systematic collection and use of outcome or relevant process measurements. Although they are useful for meeting some basic safety standards relying on them for assurance of effectiveness is misleading.

Experience and Clinical Judgment Serves as Substitutes for a MFS

There is no substantial evidence that clinical experience measured in number years bear any relationship to effectiveness. The debate over the fallibility of clinical judgment has continued for over 50 years and I don’t expect to settle it in a paragraph. However, we must recognize that clinical judgment is necessary since scientifically derived facts, be they EBTs or outcomes, cannot be applied without a heavy dose of clinical judgment. Simultaneously, we must also deal with the fallibility of clinical judgment and our lack of knowledge on how to integrate scientific data on an ongoing basis with treatment. In Vanderbilt’s current study, preliminary longitudinal data (which really means we will not stand behind it if it is later shown to be wrong) indicates that counselors have only a faint idea (very low correlations) on how youth and caregivers rate change in symptom severity and therapeutic alliance over time. I believe they can learn something important from systematic feedback from clients. Whether and how they can apply it to practice is another matter.

Real Change in the Real World is Real Hard

I have spent many hours talking to services researchers and progressive clinicians about the ineffectiveness of services. The impression I have is that more and more of them are abandoning hope that real change is possible. One experienced investigator conducted a study strongly supported by a state in which even funding regulations were changed that failed because of the inflexible contracts that agencies had with their clinicians. Another researcher who has been carefully implementing a statewide outcome system for a long period of time thinks that the efforts may be in vain since a new commissioner took office and agencies became more vocal and influential in their opposition to outcome measurement.

I don’t see the fruits of other reform efforts being very successful either. There are several groups funded by foundations or government agencies to foster change that appear not to have accomplished their goals. There was a flurry of interest in measuring outcomes several years ago (someone even labeled it outcome mania) but that seems to have been another passing fad in the mental health field.

In 2001 The Institute of Medicine identified what they called the “Quality Chasm” in healthcare. I have a slide in one of my presentations that has the following quotation, “You can’t cross a chasm in two small jumps.” Although I originally thought it was a quotation from Evil Knievel, it actually is from David Lloyd George, (1863–1945) a former British Prime Minister. Incrementalism has not been a successful strategy for change.

Implementing, adapting and sustaining a MFS is not as easy as I once thought. It is difficult to do this right since we do not even know which way(s) is right. However, a MFS has the potential to change the way treatment, and that includes any type of treatment, is provided and to help turn traditional service delivery organizations into true learning organizations. The widespread adoption of MFSs with the accompanying increase in accountability and support that they require is a large jump that needs to happen.