The Vulnerability of Experimental Findings to Misunderstanding, Misuse, Spin, and the Streetlight Effect

Abstract

This chapter examines why the results of basic income experiments are so easily misunderstood, and, therefore, vulnerable to spin, sensationalism, and other forms of misuse. These problems exist because of the inherent complexity of the material, the differences in background knowledge of the people involved, and the political nature of the issue.

This chapter attempts to help anyone involved in the discussion of the current round of experiments to avoid misunderstanding, misuse, spin, and the streetlight effect by explaining why UBI experiments are so vulnerable to those problems. Misunderstandings happen because the nonspecialists who create the demand for experiments and the specialists who conduct the experiments have great difficulty understanding each other, and they are separated by a long and difficult chain of connections. Essentially, we’re playing the telephone game, in which one person tells another person a story; the second person passes it to a third, and so on. Each degree of separation adds potential for misunderstanding, and the story gets less and less accurate the more it is passed on.

The telephone game is especially difficult for UBI experiments because we’re playing it with inherently difficult information, and the people involved don’t always have a shared set of background assumptions. All research—any discussion—involves background assumptions, but when information moves from groups with differing sets of background assumptions, misunderstandings creep in.

It’s common but wrong to distract attention from background assumptions as if they were unchallengeable truths. People do this both consciously and unconsciously. Even if people explain them, background assumptions are easily lost because they’re not the most interesting part of the story to pass along to the next person. A lack of understanding about the background assumptions that go into research can lead to the impression that it is more definitive than it actually is.

Consider how the chain of connections affects UBI experiments. The citizens who create the demand for trials might not know what questions experiments can and cannot address. I’ve argued that most citizens are interested in the big questions, an overall verdict on UBI’s efficacy. They will probably count on researchers conducting studies to decide what questions to address and how to address them, and they might presume or at least hope that these experts will be able to anticipate the questions they want answered and translate that evidence into the right answers.

Politicians , rather than the people most closely involved in the UBI discussion, usually make the decision to have a trial. Only a few of them will be closely connected to that discussion. They might be interested in a different definition of UBI than the one used by most supporters. Whatever UBI model politicians decide to test, they cannot be counted on to know what questions are most relevant to the citizens involved on any side of the public discussion. Often, they seem to have no specific questions in mind, and when they do, their questions might differ from the questions most important to the public discussion.

Once politicians make the decision to have an experiment, they designate a government department to work on it. Appointed public servants in that department might in turn hire managers or consultants specifically for the project, and one of those groups appoints social scientists to design and conduct the study. These social scientists are, therefore, separated from the public discussion by several degrees, each of which has potential to add misunderstanding.

The researchers hired to conduct the trial might or might not be well-versed in the dialogue. There are researchers, like myself, who are heavily involved with the public discussion of UBI, but hiring those researchers increases the risk of confirmation bias. Researchers who are not involved in the UBI discussion will almost certainly research UBI as a policy, but they might not always research the public discussion of it or consult closely with people involved in that discussion. Although research will most likely be conducted by good scientists who will attempt to make a positive contribution to the body of knowledge about UBI, there are vulnerable to misunderstanding and likely to focus on aspects of the issue that depart substantially from the aspects that most interest people involved in the public discussion. Consider five reasons.

First, social scientists are not one united group with an automatically shared set of background assumptions. Specializations in many different fields and subfields are relevant to UBI and UBI experiments. Social scientists have to make an effort to develop a shared set of background assumptions across disciplinary barriers before they can develop a shared understanding with their nonspecialist audience.

Second, as discussed in the puzzle analogy in the introduction, social scientists tend to look at research questions very differently than nonspecialists. Nonspecialists tend to want a verdict, up or down. Social scientists know that no single study is very likely to produce a decisive verdict on any social science issue and tend to want to add to the existing body of knowledge about UBI.

Third, social scientists have no particular expertise in discovering the questions that concern others. Their expertise is in applying the tools they know to questions those tools are most suited to address. Politicians hired them, knowing their area of expertise is to conduct an experiment that can address some questions better than others. Social scientists might reasonably assume that they have been hired to do what they (and their experimental tools) do best. But, of course, the streetlight effect simply is the focus on what researchers and/or experiments do best instead of the questions that most need to be answered.

Fourth, social scientists have a strong interest in being seen by their peers as doing something scientific. The general climate in most of the social sciences is that quantitative research is somehow more scientific than qualitative research. Studies reporting numbers—the more quantifiable the better—are seen as more scientific than those reporting less quantifiable observations. In addition, RCTs are seen as being more scientific than saturation studies, even if a saturation study produces more relevant results to the issue being studied.

Fifth, specialists—like everyone else, including you and me—tend to have self-serving bias, in this case toward believing what they do is important. If so, they are likely to believe that whatever questions their experiment can address are more important than they actually are. They might underemphasize (to themselves and to others) the importance of all those questions that the experiments cannot address or the differences between experimental findings and their implications about the centrally important questions in the evaluation of UBI as a policy.

This analysis indicates the possibility that specialists conducting UBI experiments will be most interested in different questions than the nonspecialist citizens and policymakers involved in the discussion of UBI. This difference in concern would not be crucial if everyone understood it. Nonspecialists might be disappointed to learn the extent to which, instead of a decisive, overall evaluation of the policy, UBI experiments produced a small improvement in the existing knowledge about a few of the questions relevant to that overall evaluation, but as long as they learn enough about the research and its implications, research findings will improve their understanding of the evidence about UBI.

Unfortunately, the telephone game begins again as experimental findings make their way back into the public discussion.

Researchers usually take other researchers as their primary audience. When they do, they write in the exacting academic terms familiar to other researchers and leave out the background knowledge familiar to other researchers in their respective fields but not necessarily familiar to people outside of their field. As Chap. 1 mentioned, many excellent researchers are not very good at communicating with nonspecialists.

The US and Canadian experiments released findings in the 1970s and early 1980s mostly in specialist-to-specialist publications, such as academic monographs and journals,¹ which are dense and difficult for nonspecialists.

Hopefully, the new round of studies will produce at least some reports aimed at general audiences. They might even employ science communication specialists to report the results in language that nonspecialists can best understand. However, even the best-written reports might not attempt to bridge the most important gaps in understanding. Research reports often aim to help nonspecialists understand scientific findings on their own terms more than they aim to help relate those findings to the questions that most concern nonspecialists. For example, reports might help people understand how the behavior of the control group differed from the behavior of the experimental group in the ways that researchers were capable of studying. For at least four reasons, reports might not attempt the very complex and difficult effort required to explain how much (and how little) these differences say about the likely overall effects of a national UBI in the areas of most concern to nonspecialists.

First, it’s not necessarily their job. Unless specifically instructed, it is not usually the job of researchers or of science communication specialists to find out what questions interest other people. Their job is to conduct research and explain the findings of that research. If our political process hires specialists to do job A, we cannot blame them for neglecting our unspoken need for them to do job B as well.

Second, what is obvious to specialists is not always obvious to nonspecialists, who share few background assumptions with specialists. These studies are short term. They do not capture community effects. They produce indirect and partial inferences about the national implementation of a policy. They do not address all of the important claims needed to fully evaluate UBI as a policy. From one specialist to another this list might seem too obvious to mention, or it might seem to merit no more than a dry list of caveats so that other specialists know that the researchers conducting the study were aware of these limitations. If specialists are unaware how poorly nonspecialists understand these issues, they might not even mention them, much less work through the difficult effort needed to connect experimental results to the questions nonspecialists want answered.

Third, self-serving comes into play again. We all tend to believe what we do is important. A report emphasizing all the barriers between the experimental results and the things we really want to know would make the experiments look less valuable than they would look in a report that ignored or downplayed those differences. Similarly, a report emphasizing how much theory and data from other sources were necessary to connect the experimental results to the evaluation of the actual effects of a national policy would make the experiments themselves look less valuable.

Fourth, the pressure for social scientists to be seen doing something scientific (often conflated with doing something quantitative) also comes into play again. The effort to discuss the limitations of experimental findings in order to connect them with answers to the questions nonspecialists most want answered will involve doing more qualitative and nonacademic discussion.

Whether for these reasons or others reports about the US experiments in the 1970s overwhelmingly stressed the differences between the behavior of the control and experimental groups rather than the part these play in understanding how to evaluate BIG as a potential national policy.²

Even if research reports do address the big questions that most concern nonspecialists, the effort to help create a good, shared understanding will be difficult. No matter how well-written reports might be, they face the inherent problem that the information they contain is complex and difficult. After all, any nonspecialist who learns what specialists know becomes a specialist. Some amount of the complex implications of a UBI trial simply will be missed by most nonspecialists. The trick is to get them to understand enough. That task is not usually impossible, but it is seldom easy. Weeding through the complexity of the issue to determine what is enough and figuring out how to communicate it is intrinsically difficult.

People reading about UBI experiments might be biased toward oversimplification just because they’re looking for something they can understand. They also might be biased in this direction by what we might call “professional deference.” By this, I mean the mistaken belief that expert findings are more definitive than even the specialists themselves believe. In everyday conversations, if one person says several negative things about an idea, they are implying that the idea itself is bad and should be rejected. Research reports by contrast are to be taken at face value. If they don’t include statements about the big questions, that means they don’t have answers to those questions. Unfortunately, not all readers will understand that. At least some of them will probably take every positive-sounding result as the experts’ vote for and every negative-sounding result as the experts’ vote against the policy. Even a clear caveat warning readers against making such inferences might be ignored by some readers or journalists.

Whether or not researchers conducting experiments produce reports attempting to explain that complexity directly to nonspecialists, most nonspecialists (i.e. most citizens and politicians) will get their information about the study not from research reports but from popular writers, such as journalists, bloggers, and columnists,³ creating yet another degree of separation, and one that involves opportunities for spin and sensationalism.

Popular writers might well be professional writers, but few of them are professional social scientists. Only a few of them will have much more expertise than the public they write for. They might struggle to understand research reports even on their own terms. They might be incapable of doing the complex analysis necessary to relate reported differences between the control and experimental groups to probable outcomes for a national UBI. That is, they might have some of the same problems as their readers in understanding the results of UBI trials.

Popular writers, especially if their understanding is limited or oversimplified, are likely to be biased toward sensationalism. The reporting in the 1970s on the NIT experiments was overwhelmingly sensational.⁴ Whether it is out of professional deference, a desire to attract more readers or the inherent difficulty of the material, many recent reports about the UBI experiments getting underway now have been sensational.⁵ For example, Matt Reynolds recently debunked a significant amount of sensational reports saying that Finland cancelled its UBI experiment, when it simply decided not to extend the experiment.⁶

Most likely, some writers, politicians, and even some researchers will—consciously or unconsciously—spin the results to the advantage of one side or other in the debate. “Spin”—as I use the term—is not necessarily deceptive. To spin is to present information in a way that favors one or another interpretation of it. A person (like me) who is convinced UBI is a good idea cannot present what they know honestly without also putting UBI in a favorable light. The same is true for opponents. Honest spin is not unethical, but it is a source of misunderstanding as information goes through the telephone game.

Spin becomes dishonest when people knowingly overemphasize one side of the issue over another. This kind of spin can still be unconscious if it stems from a bias toward recognizing favorable evidence as more important than unfavorable evidence, but it is deceptive and can be a big source of misunderstanding. Spin becomes extreme when people look at evidence not as a way to improve their understanding but as a source of ammunition to use to defend their preconceived position.

Most citizens will get their information from popular articles. As those citizens absorb that information, they add another degree of separation to the telephone game. They might add a layer of misunderstanding or oversimplification to what might already exist in the article.

All this adds up to a great danger that even well-conducted experiments will fail to increase the understanding of evidence among people engaged in the public debate. This risk doesn’t require any of the people involved to be fools or fakers; this risk exists because a lot of people are involved in a long chain of transmission of very complex information, about which they share few background assumptions. I’ve argued that communication problems like these had a detrimental effect on the discussion of the NIT experiments in the 1970s. It’s important not to let that happen to the current round of UBI experiments.

1 Working Backward from the Public Discussion and Forward to It Again

I’ll put off most of the discussion of how to combat misunderstanding, misuse, spin, and sensationalism until the concluding chapter. But I will say one thing now. People commissioning, designing, and conducting UBI experiments should work backward from an understanding of the public discussion to the experiment by identifying the claims that are important to the public discussion and attempting to relate all their findings to those claims. Then, they should work forward again, explaining the relevance of the experimental findings to the issues that are important to the discussion.

Working backward from the debate does not require experiments to test everything everyone wants to know about UBI. It requires researchers to try their best to identify the questions that interest the public, especially the big bottom-line questions, and relate the things they can find to the issues that are most valuable to the public evaluation of UBI as a policy. Not only can this effort help researchers design experiments that are better understood, but it will also help them design experiments that are more genuinely useful to the public decision of whether to introduce UBI.

Once the experiment is complete, researchers and others writing about experiments should work forward again from the test to the public discussion, explaining carefully what the experiments findings do and do not imply about the issues of interest to the public discussion of UBI. This effort involves calling attention to the limits of experiments, and it might, therefore, make the experiments seem less valuable. But a good understanding of what experiments cannot do is essential to the understanding of what they can do.

The next seven chapters consider the process of working backward from the debate to the design of the experiment. It’s a daunting but worthwhile task.