Arrow’s theorem and theory choice

Abstract

In a recent paper (Okasha, Mind 120:83–115, 2011), Samir Okasha uses Arrow’s theorem to raise a challenge for the rationality of theory choice. He argues that, as soon as one accepts the plausibility of the assumptions leading to Arrow’s theorem, one is compelled to conclude that there are no adequate theory choice algorithms. Okasha offers a partial way out of this predicament by diagnosing the source of Arrow’s theorem and using his diagnosis to deploy an approach that circumvents it. In this paper I explain why, although Okasha is right to emphasise that Arrow’s result is the effect of an informational problem, he is not right to locate this problem at the level of the informational input of a theory choice rule. Once the informational problem is correctly located, Arrow’s theorem may be dismissed as a problem.

1 Theory choice as a form of aggregation

The history of science offers several instances of situations in which distinct scientific theories or models were proposed to deal with the same class of phenomena. Familiar examples are the Ptolemaic and Copernican system of astronomy or Lavoisier’s account of oxidation and the rival accounts based on phlogiston chemistry.

In such situations, as in any case in which competing theories co-exist, the working scientist has to select one of them, in which she will pursue her research: a problem of theory selection or theory choice arises. In a recent paper (Okasha 2011), Samir Okasha has proposed to look at this problem from a formal point of view, by thinking of theory selection as the application of an aggregation procedure. In order to make sense of this perspective, one needs to assume that the scientist who is to select a theory relies on a list of criteria against which she is able to evaluate the alternative rival theories available. One possible list of criteria has been suggested in Kuhn (1977), a paper that provides the motivation for Okasha’s discussion: the list includes accuracy, consistency, broad scope, simplicity and fruitfulness.¹

Using each one of the given criteria, a scientist may generate a ranking of competing theories: the result of this operation is a finite list of rankings or a profile. An aggregation procedure is a rule that makes it possible to compute an ’overall’ ranking of rival theories from a profile. For example, one could attach a numerical score to the position that a theory occupies in each one of the single-criterion rankings and find its overall score by summing up these partial scores. The overall score of each theory determines the aggregate ranking. Other aggregation procedures may be used: because of this, it is in principle possible that scientists adopting the same list of criteria diverge in their actual choices of a theory from a set of competing alternatives. Okasha (2011) sees this as a problem for scientific theory choice that was already highlighted in Kuhn (1977)² and may be read as a potential threat to scientific rationality, if one assumes that there should be a unique, objective method of theory selection.³ Failure of uniqueness is, however, less threatening than failure of existence and Okasha argues that one can go beyond Kuhn and meaningfully ask whether rationally acceptable procedures of theory selection exist. This question is interesting because, given the characterisation of theory choice as an aggregation procedure, i.e., a collective decision method⁴, a result from collective decision theory known as Arrow’s theorem, according to which no aggregation procedure that satisfies a list of seemingly desirable properties exists, straightforwardly applies to theory choice.

In this paper I am going to argue that one must dismiss the idea that Arrow’s theorem poses any serious threat to scientific rationality in the first place. In order to reach this conclusion, I offer, in Sect. 2, a preliminary discussion of both Arrow’s theorem and the strategy proposed in Okasha (2011) to avoid it. In Sect. 3 I explain why one of the assumptions on which Arrow’s theorem rests must be rejected in the context of theory choice. In Sect. 4 I criticise Okasha’s diagnosis of the source of Arrow’s theorem: in particular, I point out that Okasha is right to emphasise that Arrow’s theorem arises because of an informational problem, but he is not right to locate this problem at the level of the informational input of a theory choice rule. Once the informational problem is correctly located, Arrow’s theorem may be dismissed as a threat. I conclude with some remarks on Kuhn’s view of theory choice in Sect. 5.

2 Arrow’s theorem and a (partial) way out

Suppose that three theories $a,\,b,\,c$ (a concrete example might involve the Ptolemaic, Tychonic and Copernican system) are to be evaluated on the basis of the five criteria presented in Kuhn (1977) and mentioned in Sect. 1. If each criterion generates an ordering⁵ of $a,\,b,\,c$, their application gives rise to a profile that is made up of five distinct orderings. A theory selection procedure is a rule that computes an ordering from any one profile. I will henceforth assume that such a procedure computes a strict ordering, i.e., that it incorporates a method to break ties and, thus, selects exactly one theory. Without explicitly describing an aggregation procedure, one may restrict the family of relevant procedures by requiring that they satisfy certain properties. One of them, which is quite natural to require, is unanimity, according to which, if every criterion ranks $a$ strictly above $b$, then the aggregate ranking of $a,\,b$ must be the same, in symbols $a\,>\,b$.

A further, seemingly plausible, property, known as independence of irrelevant alternatives or binary independence, states that any two competing theories must be judged on their relative merits alone: equivalently, the aggregate ranking of $a,\,b$ exclusively depends on the way each criterion ranks this pair. As a consequence, if two profiles agree on their relative $(a,\,b)$-rankings, their aggregate ranking of $a,\,b$ must be the same.

Finally, since more than a single criterion is involved in the aggregation, it makes sense to restrict attention to aggregation procedures that do not always agree with the ordering of a single, fixed criterion: this property has been called non-dictatorhip in the literature on Arrow’s theorem. One way to formulate the latter result in the light of the foregoing preliminaries is the following:

Theorem

(Arrow (1951), adapted to scientific theory selection)

If there are at least three theories and at least two criteria are involved in their evaluation, then every theory selection procedure that satisfies unanimity, binary independence and non-dictatorship has a cyclic aggregate ordering for some profile.⁶

In the case of three theories, one can prove that, if an aggregation satisfies Arrow’s conditions, there are profiles of (transitive) orderings that give rise to the aggregate cyclic outcome: $a\,>\,b,\,b\,>\,c,\,c\,>\,a$ [see for instance Saari (1995, p. 93), (1998, p. 252)], which violates transitivity. The reason why cycles are unwelcome is that the existence of a cyclic outcome amounts to the fact that the choice is not defined for the profile that generates such an outcome. Equivalently, the criteria employed to carry out theory choice are insufficient to actually make the choice. Since Arrow’s theorem holds for any number of criteria $\ge$2, as long as there are at least three rival theories, no list of criteria, however sophisticated, will be able uniformly to determine a choice.

A straightforward way to avoid cyclic outcomes is to restrict the set of profiles, disallowing a priori the possibility that theory choice criteria should generate certain combinations of orderings:⁷ such a move seems to me less than desirable, because ruling out possible evaluations of competing theories by fiat does not prevent, but only ignores, their hypothetical occurrence.⁸

Okasha does not pursue this route to circumvent Arrow’s negative result, but a subtler one inspired by Sen (1977): Sen points out that Arrow’s framework presupposes what he calls ordinal noncomparability within profiles. In other words, the profiles for which the theorem holds are lists of orderings, which no defined relation links together and in which no more than the relative position of distinct alternatives is recorded. The expression ’ordinal noncomparability’ is meant to specify the amount of information carried by a profile. Sen’s thought, which is pursued by Okasha in the context of theory choice, amounts to the suggestion that profiles encoding suitably stronger information might suffice to prevent the appearance of cyclic outcomes. As Okasha points out, the move to informationally stronger profiles looks especially plausible in the context of theory choice: when the evaluation of competing theories is at stake, one is often in a position to generate a ranking of alternatives that encodes information about how much better one theory is than another in some respect, i.e., information stronger than ordinal. If, in addition, orderings can be related by a suitable comparability condition, this may open a way to avoiding Arrow’s theorem for theory choice, by showing that the evaluation criteria involved in the aggregation procedure generate informationally rich profiles of interrelated orderings⁹ that prevent the emergence of cyclic aggregate rankings of scientific theories.

Even without considering this proposal in further detail, it is possible to highlight two reasons why it is not wholly satisfactory. The first reason is already alluded to by Okasha:

It may be that for the ‘large scale’ theory choices that Kuhn was interested in, ordinal comparisons are all that can be achieved. But it seems clear that in other more humdrum cases, particularly where the problem may be formulated statistically, we may have much more than ordinal information at our disposal (Okasha 2011, p. 103).

Although there are interesting classes of cases in which theory choice does not fall prey to Arrow’s theorem, this does not rule out the scenarios in which only ordinal information may be available. If such cases arise, Sen’s strategy or Okasha’s adaptation of this strategy do not work and the problem posed by Arrow’s theorem remains intact.

The second, and more important, reason why Sen’s strategy is not satisfactory is that it does not provide an accurate diagnosis of the reason why Arrow’s theorem obtains. More precisely, Sen’s emphasis on information is very important but he focuses exclusively on the information encoded in profiles, wholly neglecting the way in which information is processed under an aggregation procedure. The reason why information-processing is of the essence can be spelled out by considering again a scenario in which exactly three competing theories $a,\,b,\,c$ are given. If there are five criteria, a profile for this scenario is a list of five sequences of three items, namely $a,\,b,\,c$. The information provided may be called sequenced, ordinal information.

Now any aggregation procedure that relies upon binary independence only makes use of pairwise ordinal information, not of sequenced ordinal information: equivalently, what is a sequence of three alternatives¹⁰ for a profile, is treated as if it were merely a triple of pairs for an aggregation procedure. The reason why this seemingly trivial distinction actually makes a difference is that, by adopting procedures that process triples of pairs, one is bound to obtain cyclic outcomes whereas, by adopting procedures that deal with sequenced triples, one may rule them out completely. The existence of cycles or, equivalently, Arrow’s theorem, thus critically depends on the way the information available in a profile is, or fails to be, processed by an aggregation procedure and not particularly on the informational strength of a profile since, as Okasha himself points out (Okasha 2011, p. 101), a stronger informational content may fail to affect Arrow’s result.

This point is crucial and may be illustrated by looking at what happens to a three-alternative case when the type of information-processing carried out by an aggregation is modified.

In the present case, binary independence may be replaced with ternary independence [see Saari (2001, p. 151)], according to which two profiles with the same $(a, b, c)$-rankings must give rise to the same $(a, b, c)$-outcome. If exactly three alternatives are involved, it is clear that transitive profiles can never give rise to cyclic outcomes, since any cyclic profile differs from any transitive profile.

The switch from binary independence to ternary independence does not alter the informational content of a profile, which remains of the ordinal noncomparability type, in Sen’s terminology. On the other hand, ternary independence exploits the sequenced ordinal information contained in a profile, whereas binary independence could only exploit a fragment of that information, i.e., pairwise rankings.

Thus, ordinal profiles encode sequencing information that is not used by binary independent aggregations but may well be used by other aggregations. This suggests that, if the full informational content of a profile matters to theory choice, then the rejection of binary independence is mandatory.

3 The status of binary independence

The concluding remarks from the last section suggest that Arrow’s theorem holds not because the ordinal information provided by a profile is insufficient to prevent the emergence of cycles but because binary independent aggregations cannot make use of the ordinal information provided.

In presence of exactly three alternatives, ternary independence forces an aggregation to make use of the fact that a profile is made up of ordered sequences of alternatives: this is enough to prevent cycles. Binary independent aggregations, on the contrary, operate with pairs of alternatives, considered in isolation: Arrow’s theorem shows that doing so is sufficient to disrupt the uniform possibility of reconnecting separately aggregated pairs into a transitive ordering. Cyclic outcomes are bound to emerge.

The same observations apply to cases in which more than three alternatives are involved: a profile based on $n$ alternatives is a finite list of $n$ orderings and, thus, encodes betweenness information concerning the way in which $n$ alternatives are arranged in a sequence by each criterion. Binary independence, as well as a number of stronger forms of independence (up to ($n$ – 1)-ary independence), which only apply to overlapping parts of the given profile, gives rise to cyclic outcomes.

Thus, if one demands that the outcome of an aggregation depend on all of the information made available by a profile, one should also demand that the aggregation procedures employed be able to make use of that information: it follows that binary independence in particular, since it cannot make use of sequencing or betweenness information, is to be regarded as a condition not to be imposed on a viable theory choice procedure. To put it somewhat paradoxically, theory choice procedures should be designed to violate binary independence since, as soon as they satisfy it, they automatically give rise to cyclic outcomes that could be avoided by introducing alternative conditions that are more sensitive to the full ordinal structure of profiles.

In order to clarify this point and make apparent why binary independence is informationally defective, let me look at aggregation procedures from a slightly different point of view.

Take a general aggregation procedure to be a function A from a set $P$ of profiles (of orderings or cycles) into a set of outcomes, which may be orderings or cycles. If $p, p'$ are two profiles, denote their outcomes under A by $\mathrm{A}(p), \mathrm{A}(p')$. The relation $p \approx p'$, defined on profiles by the condition $\mathrm{A}(p) = \mathrm{A}(p')$, is an equivalence relation that partitions the space of profiles. The elements of each equivalence class are profiles indiscernible under A, since the latter discriminates profiles only by yielding distinct outcomes. Now suppose that A satisfies unanimity and binary independence: it follows that a profile in which every criterion generates the same cycle yields under A a cyclic outcome c. What Arrow’s theorem shows¹¹ is that, for every cycle c, binary independence makes available a transitive profile p such that $p \approx \mathrm{c}$. This is to say that, even when the space of profiles is restricted to transitive ones only, a binary independent aggregation treats some transitive profiles as if they were unanimously cyclic.

In order to avoid this undesired result, one may look for a type of aggregation procedure B such that it generates a certain finer partition of the space $P$ than A does. In general, a B-partition is finer than an A-partition if and only if every B-equivalence class is included in some A-equivalence class. The finer partition of interest is any one in which no equivalence class includes a cyclic and a transitive profile at once. If the number of alternatives over which the procedure of theory selection is fixed and $n$, a trivial solution is to replace binary independence by $n$-ary independence in order to achieve this goal. More interestingly, a modification of binary independence originally proposed in Saari (1994), (1995) provides the required refinement for every $n$. This or any equivalent modification is necessary, since the informational basis of a profile is richer than a collection of pairwise comparisons and dealing with this fragment of information only is deliberately to ignore the additional information whose processing rules out the emergence of cycles.

In order to take into account the sequencing of alternatives, i.e., their position within each component ordering of a profile, Saari proposed to replace binary independence by a condition to the effect that the aggregate ranking of a single pair of alternatives should depend not only upon the individual rankings of that pair in the profile but also of the alternatives’ degree of separation in each component of the ranking. The degree of separation between two alternatives is the number of alternatives that lie strictly between them in the ordering generated by a criterion. Saari calls the degree of separation the intensity of a pairwise ranking: it is important to stress that ’intensity’ is a purely ordinal concept, since it only depends on the particular way in which certain alternatives lie between others in a profile ordering.

If one replaces binary independence by its intensity variant, then there are procedures that satisfy unanimity, non-dictatorship and only give rise to orderings when applied to profiles of orderings [see Saari (1995, p. 97)].¹²

4 Evaluating Okasha’s strategy

The previous section showed that the threat to the rationality of theory choice posed by Arrow’s theorem can be neutralised simply by taking seriously the need to exploit the full information encoded in a profile. That this need should be taken seriously follows from the fact that a profile is generated by the application of evaluation criteria that have been selected in order to make a theory choice: to neglect, if only partially, the ordinal information provided by these criteria signals an inconsistent attitude whereby one at the same time decides to make use of certain criteria and to ignore the information carried by them. Thus, consistency in the application of evaluative criteria suffices to reject binary independence and, as a consequence, to dismiss Arrow’s theorem as a problem for theory choice.

This conclusion follows from the informational analysis of Arrow’s theorem articulated in Sects. 2 and 3, but cannot follow from the informational analysis proposed by Okasha. To see this, consider the following passage, which comes at the end of Okasha’s analysis:

Sen’s work demonstrates clearly that Arrow’s impossibility result is in large part a consequence of the impoverished information he feeds into his social choice rule. Enriching the informational basis, while retaining Arrow’s four conditions [...] is sufficient to avoid the impossibility (Okasha 2011, p. 101).

The first sentence is inaccurate: Arrow’s impossibility result entirely depends on the fact that the theorem deals with social choice rules that do not react to enrichments in profile information. For instance, the enrichment obtained by moving from profiles of pairwise rankings to ordinally stronger profiles of sequences does not affect Arrovian aggregations, nor are they affected by stronger forms of enrichment: this highlights the fact such aggregations are simply not responsive to informational enrichment, unless one aggregates profiles endowed with comparability relations, i.e., objects other than profiles. Because Arrow’s theorem continues to hold under profile enrichment and stops holding as soon as aggregations are required to be responsive to enrichment, Okasha’s diagnosis of the source of Arrow’s result cannot be accepted. In particular, this diagnosis cannot be relied upon to defend Okasha’s suggestion that the informational basis should be enriched. Furthermore, the informational basis fed into a procedure cannot be enriched a priori. If it is only ordinal, as Okasha envisages may be the case [see Okasha (2011, p. 103), quoted above], no enrichment will be possible and Sen’s strategy will not be viable. In order to avoid these difficulties and take seriously the idea that Arrow’s theorem arises from an informational problem, it is necessary to recognise something that both Sen and Okasha overlook, i.e., the fact that the informational strength of an aggregation procedure does not depend only on the information provided by its input but also on the way in which this information is treated by the aggregation rule to compute an output. If one focuses exclusively on the informational content of profiles, the only available informational strategy is enrichment, regarded as the imposition of more profile-structure, which may not always be possible. If, however, one also examines the type of information-processing characteristic of an aggregation rule, a further possibility arises, which amounts to replacing the given rule with one that is responsive to the information provided by the input. This is not a form of enrichment because it does not introduce more information that an aggregation rule can make use of. Rather, it is a form of fine-tuning between an aggregation rule and the informational content that the aggregation is required to process. Such a fine-tuning may be called informational adjustment.

Arrow’s theorem is not a problem of informational poverty, as Okasha would suggest after Sen, but a problem of informational maladjustment. The informational approach to the resolution of this problem is not enrichment but adjustment: it consists in the replacement of binary independence with an alternative condition that determines a finer partition of the space of profiles and, thus, can distinguish between transitive profiles, in which the degree of separation between any two alternatives matters because every ordering in the profile has an orientation, and cyclic profiles, in which no orientation is specified.

5 Concluding remarks

The last section sought to show that Okasha (2011) supports an informational approach to the resolution of Arrow’s theorem that is based on an inaccurate diagnosis of this theorem’s source.

The correct diagnosis, which I spelled out relying on Saari’s work in Sect. 3, supports a distinct informational strategy, which directly leads to the dismissal of the challenge raised by Arrow’s result. This conclusion is not unrelated to the approach to Arrow’s theorem articulated by Sen and set by Okasha in the context of theory choice: both Sen and Okasha correctly conjecture that Arrow’s theorem depends on an informational limitation but take too restrictive a point of view when seeking to locate it, since they only consider limitations connected to the structure of profiles, as opposed to the informational fit between profiles and aggregation rules.

Because the structural enrichment of profiles may not be viable and, even if it is, it may not affect Arrow’s result, the informational strategy outlined by Okasha after Sen is not entirely secure or desirable. The alternative informational strategy, which amounts to an improvement of fit between profiles and aggregation rule, follows from Saari’s analysis of the mathematics of Arrow’s theorem and removes its threat by making use of the full profile information provided.

In other words, when the informational approach to Arrow’s theorem is made to rest on an accurate isolation of the informational shortcoming that gives rise to it, the natural conclusion is the dismissal of the theorem itself as a serious problem for theory choice. What this reveals about theory choice is that there are no inherent difficulties with thinking about it in algorithmic terms, i.e. as a form of aggregation, in the sense that it is entirely possible to impose general conditions on theory selection algorithms that are satisfied by some explicitly specifiable procedure.

This leaves intact the issue pointed out by Kuhn, namely the fact that different choices of algorithm are possible. Although a discussion of the non-uniqueness problem goes beyond the scope of the present paper, it is worth pointing out that two immediate responses to it are available. One is to accept that non-uniqueness is a problem and add plausible conditions to unanimity, non-dictatorship and the intensity version of binary independence that can be satisfied by a unique procedure. In fact, if one requires the aggregation to use a scoring rule known as a positional method¹³, only one procedure, known as the Borda Count¹⁴, satisfies the above conditions [see Saari (1995, p. 97)]. Insofar as there are no immediate objections to the employment of scoring rules in a theory choice algorithm, at least one way of solving the non-uniqueness problem is open.

The other response consists in arguing, in the light of scientific practice, that the co-existence of distinct methods of theory selection is not to be regarded as a problem. This is what Kuhn claimed in his 1977 paper on theory choice when, describing the transition from an old to a new scientific theory, he remarked that:

$[{\ldots }]$ before the group accepts it, a new theory has been tested over time by the research of a number of men, some working within it, others within its traditional rival. Such a mode of development, however, requires a decision process which permits rational men to disagree, and such disagreement would be barred by the shared algorithm which philosophers have generally sought (Kuhn 1977, p. 332).

A shared algorithm, Kuhn continues, might impose on the entire scientific community choice criteria that are too loose, thus causing working scientists to move from a comprehensive, but vague, world picture to another; alternatively, an algorithm whose standards were too stringent would deny emerging theories any breathing space and constantly support a single tradition of normal science.

This suggests that, even if one thinks of theory choice as a practice that is driven by the application of an algorithm, the existence of many distinct algorithms need not prompt a search for a unique ’correct’ algorithm. On the contrary, the very feasibility of theory choice, algorithmically conceived, hangs on the question whether it is possible to find any acceptable algorithms. Whilst Arrow’s theorem may be invoked to suggest a negative answer to the last question, the foregoing discussion has shown that the negative suggestion can simply be dismissed on informational grounds.¹⁵