Evolution of cooperation in a particular case of the infinitely repeated prisoner’s dilemma with three strategies

Abstract

We study a population of individuals playing the infinitely repeated prisoner’s dilemma under replicator dynamics. The population consists of three kinds of individuals adopting the following reactive strategies: ALLD (individuals which always defect), ATFT (almost tit-for-tat: individuals which almost always repeat the opponent’s last move) and G (generous individuals, which always cooperate when the opponent cooperated in the last move and have a positive probability q of cooperating when their opponent has defected). Our aim is studying in a mathematically rigorous fashion the dynamics of a simplified version for the computer experiment in Nowak and Sigmund (Nature 355:250–253, 1992) involving 100 reactive strategies. We see that as the generosity degree of the G individuals varies, equilibria (rest points) of the dynamics appear or disappear, and the dynamics changes accordingly. Not only we prove that the results of the experiment are true in our simplified version, but we also have complete control on the existence or non-existence of the equilbria for the dynamics for all possible values of the parameters, given that ATFT individuals are close enough to TFT. For most values of the parameters the dynamics can be completely determined.

1 Introduction

Although cooperation is ubiquitous in human societies and also in biological systems, cooperating individuals usually have to pay a cost for the benefit of other individuals. It is therefore an interesting question to understand how cooperation can evolve in the light of Darwinian natural selection. Sigmund, Nowak and collaborators have studied in several contributions the evolution of cooperation, see e.g. Nowak (2006a, b) for some basic information and more references.

The essence of the problem can be grasped by the famous prisoner’s dilemma (PD) stated in different forms by Nowak (2012) and Sigmund (2010). In its one-shot version, individuals interact only once and each individual has only two strategies: cooperate (C) or defect (D). Whatever be the choice of his opponent, the pay-off for a defector is larger than for a cooperator. Using the jargon of game theory (Nowak 2006a), D is a strict Nash equilibrium, whereas C is not a Nash equilibrium. Rational individuals must choose D and cooperation cannot evolve in the one-shot PD (Nowak 2006a).

If individuals are given the opportunity of interacting many times before they receive their pay-offs, a reciprocity mechanism can favor cooperation. We are then in the realm of the repeated PD (RPD) and in this setting C and D are not the only possible strategies and we have the problem of selecting among a huge number of strategies combining the choice of C or D for each round of the game. In some cases, as in this paper, we may even repeat the PD infinitely, obtaining the so-called infinitely repeated PD (IRPD) (Akin 2012).

In the 1970’s Axelrod (1984) studied strategies for the RPD. He organized two RPD tournaments, in which contestants could submit any conceivable strategies, and in both realizations the winning strategy was the simplest among all submitted strategies: tit-for-tat (TFT). TFT is the strategy which repeats the previous move of its opponent. After Axelrod’s work much has been done in finding strategies with interesting properties for the RPD, as well as in other games. As the space of all available strategies is huge, many results were obtained in the subset of the so-called memory-one strategies (Akin 2012). The recent discovery (Press and Dyson 2012) of a new remarkable class of memory-one strategies for the IRPD—the so-called zero-determinant strategies—stimulated renewed interest in the field, as exemplified by Adami and Hintze (2013), Hilbe et al. (2013), Stewart and Plotkin (2013), Akin (2012), Stewart and Plotkin (2014) and Hilbe et al. (2015).

Game theory was introduced in Biology by Maynard Smith and coworkers, see Maynard Smith and Price (1973) and Maynard Smith (1974), creating the subject of evolutionary game theory. In evolutionary game theory pay-offs are viewed as biological fitness and population frequencies of individuals adopting strategies with larger pay-offs have the tendency to increase. The dynamics for the changing population frequencies is usually specified by a system of ordinary differential equations (ODEs) (Taylor and Jonker 1978) called replicator equations.

Reactive strategies (Nowak and Sigmund 1989), the ones considered in the present work, are the subset of the memory-one strategies in which the next move of an individual (C or D) is stochastically determined just by the last move of his opponent. A reactive strategy is characterized by a vector (p, q) such that p is the probability of playing C after the opponent played C and q is the probability of playing C after an opponent’s D. We may think of p as a loyalty parameter, q as a forgiveness parameter. Some simple strategies are recognized as reactive: ALLD, individuals which always defect, is denoted as (0, 0) and TFT is (1, 0). The pay-off for reactive strategy $s \equiv (p,q)$ against reactive strategy $s'\equiv (p',q')$ may be defined in the IRPD (Nowak and Sigmund 1990) if

$$\begin{aligned} (p-q)(p'-q') \ne 1 . \end{aligned}$$

(1)

This paper originates with the seminal computer experiment performed in Nowak and Sigmund (1992). In that experiment the authors took 99 reactive strategies with (p, q) randomly chosen in the square $[0,1] \times [0,1]$. To that sample, suggested by Axelrod’s results, they added by hand strategy (0.99, 0.01), an almost TFT (ATFT) strategy. TFT was not selected because (1) implies that the pay-off of TFT against itself is not well-defined in the IRPD. All 100 strategies were considered as having equal fractions at the initial time and then replicator dynamics was numerically evaluated. Results of this evolution—illustrated at one figure in Nowak and Sigmund (1992)—are described as follows:

• 1. Initially, strategies far from (0, 0) have their frequencies strongly depleted and it seems that the strategy closest to ALLD will extinguish all the others.

• 2. After some time the frequency of ATFT starts increasing, and it looks like it will be the ultimate winner.

• 3. After a much longer period the ATFT frequency decreases and a surprising strategy named generous TFT (GTFT) finally drives all other strategies to extinction. With the parameter values of the experiment, the winner was the strategy closest to $(1, \frac{1}{3})$.

GTFT seems to have been discovered in Molander (1985) and rediscovered precisely in Nowak (1990) and Nowak and Sigmund (1990), to reappear in Nowak and Sigmund (1992). Whereas TFT only reciprocates cooperation, GTFT does more than that, it is also forgiving. According to Molander (1985), GTFT defines an optimal level of generosity up to which it is safe to cooperate “without risk of exploitation by the other party”.

The purpose of this paper is to provide precise mathematical arguments supporting the results found in Molander (1985) and in Nowak and Sigmund (1992). Molander’s paper considers a situation in which there is no dynamics at all, only pairwise comparison between pay-offs obtained using different strategies in the IRPD. Furthermore, strategies considered in his paper are not reactive, but mixed strategies (Hofbauer and Sigmund 1998). Unable at this time to prove results for numbers of strategies as large as 100, we simplify the initial situation of the experiment in Nowak and Sigmund (1992) and consider the IRPD with only the three more prominent strategies in their experiment. Our results are valid for any suitable choice of the many parameters of the problem, not a fixed choice, and any initial condition for the population.

The replicator dynamics with three strategies is relatively easy for a fixed choice of the pay-offs, see e.g. the many examples treated in Hofbauer and Sigmund (1998). Despite that, we have seen no rigorous result in the literature as ours, dealing with many parameters and different dynamics according to the values of the parameters. An interesting and non-trivial example of a rigorous analysis with four strategies but fixed parameters is Zeeman (1981). Although not completely rigorous, an interesting paper about three strategies with a flavor similar to ours is Adami et al. (2012).

More concretely, in this paper we consider a population of individuals under the replicator dynamics playing the IRPD with three reactive strategies:

• Strategy 1 is an arbitrary ATFT, i.e. strategy $(1-\epsilon _1, \epsilon _2)$, where $\epsilon _1$ and $\epsilon _2$ are positive and small enough. Differently of Nowak and Sigmund (1992), we need not consider $\epsilon _1=\epsilon _2$.

• Strategy 2 is ALLD, i.e. (0, 0).

• Strategy 3, which will be called Generous (G), is (1, q), with $q>(\epsilon _1+\epsilon _2)^{1/2}$, i.e. perfectly loyal individuals and more forgiving than the considered ATFT.

We prove the existence of a maximum amount of forgiveness $q_{GTFT}$ and a region of initial conditions with positive area such that, as in Nowak and Sigmund (1992), only strategy 3 will survive after infinite time. But we will also see what happens for larger values of the forgiveness q of the G strategists and find out that if $q>q_{GTFT}$ we may still have some weaker form of cooperation evolution.

The methods used are exact calculations of the pay-off matrix for the IRPD with the mentioned three strategies and analysis of its entries. Some of the results depend on asymptotic analysis in parameters $\epsilon _1$ and $\epsilon _2$. A combination of parameters $G_1$, defined in (7), will appear naturally with its sign being a separator of different cases. For each q and sign of $G_1$ we will determine all equilibria of the replicator dynamics and find out the few phase portraits as in the classification in Zeeman (1980) or Bomze (1983) which are compatible with the existent equilibria and the dynamics at the boundary of the relevant region. In most cases only a single phase portrait of the complete table in Bomze (1983) is compatible. In such cases the dynamics will be completely determined. In some other cases we were unable to completely determine the dynamics, because more than one phase portrait in Bomze (1983) was found compatible with the information available to us. In such cases we formulate some conjectures about the dynamics.

The results of this work are summarized in Tables 1, 2 and 3, which state which of the equilibria exist for any value of q and sign of $G_1$, and the corresponding type of cooperation evolution. In Sect. 2 we introduce the pay-off matrix, the replicator dynamics and define some important concepts and notations regarding the equilibria for the replicator dynamics. In Sect. 3 we prove some simpler properties of the entries of the pay-off matrix. Section 4 contains the most important results of this paper. There we prove existence and properties of the thresholds $q_{AD}, q_{AG}$, $q_{DG}$ and $q_{int}$, which appear in the mentioned tables. In Sect. 5 we define the various types of evolution of cooperation, relate the existent equilibria for each value of q and sign of $G_1$ with the phase portraits in Bomze (1983) and use this information to conclude about the type of evolution of cooperation in each case. The paper is closed by a conclusions section.

2 Pay-off matrix and notations

In a game among n strategies, the pay-off matrix A is such that element $a_{ij}$ is the pay-off received by an individual playing strategy i confronted by an individual playing strategy j, $i,j \in \{1,2, \dots , n\}$. In the one-shot PD, if strategy 1 is C (cooperate) and strategy 2 is D (defect) then the pay-offs are

$$\begin{aligned} A_{os} \,=\, \left( \begin{array}{cc} R&{}S\\ T&{}P \end{array} \right) , \end{aligned}$$

(2)

where, by definition, entries satisfy inequalities

$$\begin{aligned} T>R>P>S . \end{aligned}$$

(3)

We will also assume the following inequalities as hypotheses for the results in this paper:

$$\begin{aligned} P<\frac{S+T}{2}<R. \end{aligned}$$

(4)

The upper bound for $(S+T)/2$ is a natural condition to ensure that alternating between C and D is not as good as a steady C for a pair of players, and has appeared at least since Molander (1985) in almost all works dealing with the RPD. The lower bound is a relatively novel condition, necessary for some of our proofs. We have seen it appearing only in Hilbe et al. (2015). As a consequence of this new assumption we will have in Proposition 1 that any amount of forgiveness in a reactive strategy will result in a pay-off larger than P for that strategy against itself.

Consider now the IRPD and reactive strategies $s=(p,q)$ and $s'=(p',q')$. As moves of the players are stochastically determined by their states at the previous instant, then their successive states follow a Markov chain. If and only if (1) holds, a limit distribution independent of the initial state of the Markov chain exists (Nowak and Sigmund 1990), and the mean pay-off per move in the IRPD may be defined precisely using this limit distribution. It can be shown that

$$\begin{aligned} c\,=\,\frac{(p-q) q'+q}{1-(p-q)(p'-q')}\;\;\;{\mathrm {and}}\;\;\; c'\,=\,\frac{(p'-q') q+q'}{1-(p-q)(p'-q')} \end{aligned}$$

(5)

are respectively the equilibrium probabilities that s cooperates with $s'$ and vice-versa. As a consequence, in Nowak and Sigmund (1990) the IRPD pay-off $E(s,s')$ of s against $s'$ is written as

$$\begin{aligned} E(s,s') \,=\, G_1 c c'+ (S-P) c + (T-P) c'+ P , \end{aligned}$$

(6)

where

$$\begin{aligned} G_1\,=\,(R-T)+(P-S) \end{aligned}$$

(7)

is a parameter which will have great importance in this work. Notice that inequalities (3) and (4) do not fix the sign of $G_1$. The special case of the PD with $G_1=0$ is known as donation game (Hilbe et al. 2013).

Notice that, apart the unimportant case of the paradoxical strategies (Nowak 1990) $s=s'=(0,1)$, condition (1) is not satisfied only if $s=s'=(1,0)$, i.e. both players are TFT. It can be seen that pay-offs for TFT players must depend on their initial moves and are also heavily affected by small amounts of noise, see Molander (1985) and also our Proposition 1. This is why ATFT—a noisy TFT—was naturally included in the experiment in Nowak and Sigmund (1992) and also in this work.

The pay-off matrix for the IRPD among the three strategies ATFT, ALLD and G numbered and specified in Sect. 1 may be calculated in a lengthy but straightforward fashion using (6) and (5). The result is

$$\begin{aligned} A_{IRPD}=\left( \begin{array}{c@{\quad }c@{\quad }c} F(\frac{\epsilon _1}{\epsilon _2}) &{} (1-\epsilon _2)P+\epsilon _2 S &{} a_{13}(q) \\ (1-\epsilon _2)P+\epsilon _2 T &{} P &{} (1-q)P+qT \\ a_{31}(q) &{} (1-q)P+qS &{} R \end{array} \right) , \end{aligned}$$

(8)

where

$$\begin{aligned} F(\rho )= & {} \frac{P\rho ^2+(S+T)\rho +R}{(1+\rho )^2} , \end{aligned}$$

(9)

$$\begin{aligned} a_{13}(q)= & {} G_1\frac{\epsilon _{1}^{2}(1-q)}{[q+(1-q)(\epsilon _1+\epsilon _2)]^2}\nonumber \\&- \frac{\epsilon _1}{q+(1-q)(\epsilon _1+\epsilon _2)}\frac{2R-S-T+(T-R)(\epsilon _1+\epsilon _2)}{1-\epsilon _1-\epsilon _2}\nonumber \\&+\frac{R(1-\epsilon _2)-\epsilon _1 S}{1-\epsilon _1-\epsilon _2}\; \end{aligned}$$

(10)

and

$$\begin{aligned} a_{31}(q)= & {} G_1\frac{\frac{\epsilon _{1}^{2}}{1-\epsilon _1-\epsilon _2}}{[q+(1-q)(\epsilon _1+\epsilon _2)]^2}\nonumber \\&-\frac{\epsilon _1}{q+(1-q)(\epsilon _1+\epsilon _2)}\frac{2R-S-T-[\epsilon _1(T-P)+\epsilon _2(R-S)]}{1-\epsilon _1-\epsilon _2}\nonumber \\&+\frac{R(1-\epsilon _2)-\epsilon _1 T}{1-\epsilon _1-\epsilon _2}. \end{aligned}$$

(11)

We stress that the combination of parameters $G_1$ in (7) reappears in (10) and (11).

Let $\mathbf {x}= (x_1, x_2, x_3)$ denote the fractions of individuals in the population using strategies 1 to 3. The fitness of strategy i, see e.g. Nowak (2006a), Hofbauer and Sigmund (1998) or Sigmund (2010), is defined as

$$\begin{aligned} f_i(\mathbf {x})= (A_{IRPD} \mathbf {x})_i, \end{aligned}$$

(12)

the i-th element of a matrix product. The mean fitness of the population is then

$$\begin{aligned} \phi \,=\, \sum _{i=1}^3 f_i(\mathbf {x}) x_i. \end{aligned}$$

(13)

The replicator dynamics (Taylor and Jonker 1978), which specifies how strategy frequencies change with time is

$$\begin{aligned} \dot{x}_i \,=\, (f_i(\mathbf {x})- \phi ) x_i, \end{aligned}$$

(14)

$i=1, 2, 3$. It can be shown (Hofbauer and Sigmund 1998) that the simplex

$$\begin{aligned} S_3=\left\{ \mathbf {x} \in {\mathbb {R}}^3; x_i \ge 0, \sum _{i=1}^3 x_i=1\right\} \end{aligned}$$

is invariant under the replicator dynamics.

Instead of studying the replicator dynamics in the simplex $S_3$, in this paper we will project it onto the $(x_1,x_2)$ plane. As the resulting dynamics will be well-defined everywhere in this plane, but we are only interested in its restriction to the projection of $S_3$, we define the biological region as this projection, i.e. the closed triangle B with vertices $E_1 \equiv (1,0)$, $E_2\equiv (0,1)$ and $E_3 \equiv (0,0)$. The sides of B will be denoted by $L_1$, $L_2$ and $L_3$, where $L_i$ is the side on which $x_i=0$. Of course, because of a one to one natural correspondence between B and $S_3$, B is invariant under the projected replicator dynamics.

For $i \ne j$ we denote

$$\begin{aligned} n_{ij}=\{(x_1,x_2)\in {\mathbb {R}}^2;\, f_i(x_1,x_2,1-x_1-x_2)=f_{j}(x_1,x_2,1-x_1-x_2)\}\; \end{aligned}$$

the straight lines in which two fitnesses are equal. We will also denote $P_{ijk}$ the point at which the line $n_{ij}$ intercepts the $x_k=0$ line. Notice that the coordinates for the $P_{ijk}$ can be easily calculated in terms of the entries in the pay-off matrix (8).

From general arguments, see Hofbauer and Sigmund (1998), the equilibria for the replicator dynamics with three strategies can be:

• Points in which only one strategy is present, i.e. the vertices $E_1$, $E_2$ and $E_3$ of B.

• Points in which one strategy is absent and the other two have the same fitness, i.e. $P_{123}$, $P_{132}$ and $P_{231}$, whenever they lie in B.

• One point in which all three strategies have the same fitness. This is the intersection of the three lines $n_{12}$, $n_{13}$ and $n_{23}$, whenever it lies in the interior of B, and will be denoted Q. Notice that if two among these lines cross at a point, then the third line must also pass through this point.

The following definition is given to rule out the cases when the above cited equilibria in the plane $(x_1,x_2)$ do not correspond to points in the simplex $S_3$:

Definition 1

We will say that equilibria $P_{123}$, $P_{132}$ and $P_{231}$ are biological whenever they lie in B, but not coincide with any of the vertices. We will say that equilibrium Q is biological whenever it lies in the interior of B.

As a mnemonic tool for dealing with the $n_{ij}$ and the $P_{ijk}$ equilibria, we will use A for strategy 1 (ATFT), D for strategy 2 (ALLD) and G for strategy 3. Thus $n_{12}$ will be referred to as the AD-isocline, $n_{13}$ will be the AG-isocline and $n_{23}$ the DG-isocline. Equilibrium $P_{123}$ will be termed AD-equilibrium, $P_{132}$ the AG-equilibrium and $P_{231}$ the DG-equilibrium. Equilibrium Q will be simply called interior equilibrium.

We will always be interested in positive values for $\epsilon _1$ and $\epsilon _2$. We define polar coordinates r and $\theta$ in the $(\epsilon _1, \epsilon _2)$ plane, so that

$$\begin{aligned} \epsilon _1= r \cos \theta \;\;\;{\mathrm {and}} \;\;\; \epsilon _2=r \sin \theta . \end{aligned}$$

(15)

Throughout this paper, r and $\theta$ will always be used with this meaning.

Many times we will use $\epsilon$ to refer to vector $(\epsilon _1, \epsilon _2)$. We define the phrase “property P holds if $\epsilon$ is small enough” as meaning “there exists $r_0>0$ such that property P holds if $0<r<r_0$”.

The overwhelming majority of the intermediate and final results in this paper will hold if $\epsilon$ is small enough. From now on, as with (3) and (4), we will assume as a hypothesis for the rest of this paper that $\epsilon$ is small enough. In the beginning we will be explicit in stating this hypothesis, because we want the reader to be aware of it, but with time we will be increasingly more relapse in reminding it.

In some instances we will also use the notation $O(r^{\alpha })$ standard in asymptotic analysis. For definiteness, if f is some function depending on $\epsilon$, we will write $f=O(r^{\alpha })$ if there exist $r_0>0$ and a constant K independent of r such that $|f/r^{\alpha }|<K$ for $0<r<r_0$.

3 Some properties of the entries of the pay-off matrix

We start by considering the pay-off $F(\frac{\epsilon _1}{\epsilon _2})$ of strategy ATFT against itself, with function F being given by (9).

Proposition 1

• (i) $F(0)=R$.

• (ii) $\lim _{x \rightarrow \infty }F(x)=P$.

• (iii) F is a decreasing function in $[0,+\infty )$.

• (iv) There exist positive constants $K_1, K_2$ such that

  $$\begin{aligned} \frac{K_1}{r}\le \frac{F(\frac{\epsilon _1}{\epsilon _2})-P}{\epsilon _2}\le \frac{K_2}{r}. \end{aligned}$$

  (16)

• (v) There exist positive constants $K_3, K_4$ such that

  $$\begin{aligned} \frac{K_3}{r}\le \frac{R-F(\frac{\epsilon _1}{\epsilon _2})}{\epsilon _1}\le \frac{K_4}{r}. \end{aligned}$$

  (17)

Proof

The first two items are direct consequences of (9). The third item follows easily by calculating the derivative of F and using both inequalities in (4).

Using polar coordinates (15) we get

$$\begin{aligned} \frac{F(\frac{\epsilon _1}{\epsilon _2})-P}{\epsilon _2} =\frac{1}{r} \; \frac{(R-P) \sin \theta +(S+T-2P)\cos \theta }{(\cos \theta +\sin \theta )^2}. \end{aligned}$$

By using (3) and (4) the function of $\theta$ in the right-hand side is clearly strictly positive and continuous in the compact $[0, \frac{\pi }{2}]$. Letting $K_1$ be its minimum and $K_2$ its maximum, assertion (iv) is proved.

Item (v) can be proved in an analogous way. $\square$

Items (i), (ii) and (iii) in Proposition 1 prove that the pay-off of an ATFT against another ATFT may be any number in (P, R) regardless of the smallness of $\epsilon$.

If $q=0$, strategy G becomes TFT. As a consequence of this, see Nowak and Sigmund (1990),

$$\begin{aligned} a_{13}(0)= a_{31}(0) = F\left( \frac{\epsilon _1}{\epsilon _2}\right) . \end{aligned}$$

(18)

Also at $q=1$ both formulas for $a_{13}$ and $a_{31}$ simplify and we obtain

$$\begin{aligned} a_{13}(1)\,=\, R+(T-R)\epsilon _1 \;\;\;{\mathrm {and}}\;\;\; a_{31}(1)\,=\,R-(R-S)\epsilon _1. \end{aligned}$$

(19)

Other important properties of $a_{13}$ and $a_{31}$ are:

Proposition 2

• (i) $a_{13}(q)>a_{31}(q)\;\forall q\in (0,1]$.

• (ii) $a_{13}(q)-a_{31}(q)$ is an increasing function in [0, 1].

• (iii) If $\epsilon$ is small enough, then both $a_{13}(q)$ and $a_{31}(q)$ are increasing functions in [0, 1] .

• (iv) $a'_{13}(0) \mathop {\rightarrow }\limits ^{r \rightarrow 0} \infty$ and $a'_{31}(0) \mathop {\rightarrow }\limits ^{r \rightarrow 0} \infty$.

• (v) $a'_{13}(1) \mathop {\rightarrow }\limits ^{r \rightarrow 0} 0$ and $a'_{31}(1) \mathop {\rightarrow }\limits ^{r \rightarrow 0} 0$.

• (vi) If $\epsilon$ is small enough, $a''_{13}(q)$ and $a''_{31}(q)$ are both negative in [0, 1].

Proof

After some easy manipulations with (10) and (11), we obtain

$$\begin{aligned} a_{13}(q)-a_{31}(q)=\frac{\epsilon _1(T-S)q}{q+(1-q)(\epsilon _1+\epsilon _2)}, \end{aligned}$$

which proves assertion (i). Differentiating the above equation proves (ii).

To prove (iii), we define first an auxiliary variable

$$\begin{aligned} x \equiv q + (1-q)(\epsilon _1+\epsilon _2) \end{aligned}$$

(20)

which leads us to

$$\begin{aligned} a'_{31}(q) \,=\, \epsilon _1 \, \frac{2R-S-T- \epsilon _1(T-P) - \epsilon _2(R-S)}{x^2} \, -\, \frac{2G_1 \epsilon _1^2}{x^3} . \end{aligned}$$

(21)

Notice then that $a'_{31}(q)/ \epsilon _1$ is a continuous function of $\epsilon$, positive at $\epsilon =0$. As a consequence, $a'_{31}(q)$ is positive if $\epsilon _1>0$, provided that $\epsilon$ is small enough. Using (ii) the analog result is obtained for $a_{13}(q)$.

To prove (iv), we substitute $q=0$, thus $x= \epsilon _1+\epsilon _2$, in (21). Using polar coordinates and (7), we get

$$\begin{aligned} a'_{31}(0) \,=\, \frac{1}{r} \, \frac{(S+T-2P) \cos ^2 \theta + (2R-S-T) \sin \theta }{(\cos \theta + \sin \theta )^3} \, +\, O(1) . \end{aligned}$$

As the function of $\theta$ multiplying 1 / r is positive, then the result for $a'_{31}(0)$ is proved. Using again (ii), we prove the same for $a'_{13}(0)$.

The proofs of (v) and (vi) follow similar ideas. $\square$

Although formulas (10) and (11) are complicated, Proposition 2 tells a lot about these functions. In particular, properties (iv) and (v) show that both $a_{13}$ and $a_{31}$ grow very fast for q close to 0 and then saturate before $q=1$.

To close this section, a simple and important

Corollary 1

If $\epsilon$ is small enough and $\alpha \in (F(\frac{\epsilon _1}{\epsilon _2}),R+(T-R)\epsilon _1)$, then equation $a_{13}(q)=\alpha$ has a unique root q in interval (0, 1). Analogously, if $\beta \in (F(\frac{\epsilon _1}{\epsilon _2}),R-(R-S)\epsilon _1)$, then $a_{31}(q)=\beta$ has a unique root in (0, 1).

4 Locating the equilibria

We start this section by studying the AD-equilibrium $P_{123}$, the simplest among the equilibria in which only two strategies coexist, because its location is independent of the variable q, as shown by the following result.

Proposition 3

The AD-equilibrium $P_{123}$ is independent of q and always biological.

Proof

Equating fitnesses $f_1$ and $f_2$, given by (12), and writing $x_2=1-x_1$, which is equivalent to $x_3=0$, we obtain

$$\begin{aligned} x_1(P_{123})=\ \frac{1}{1- \frac{T-P}{P-S}+\frac{1}{P-S} \, \frac{F(\frac{\epsilon _1}{\epsilon _2})-P}{\epsilon _2}} , \end{aligned}$$

(22)

which is indeed independent of q. Using (iv) in Proposition 1 we see that the denominator in the above equation is dominated by a positive term of order 1 / r. Thus $x_1(P_{123})>0$ and as small as we want if $\epsilon$ is small enough. $\square$

The next result will be important when showing that the interior equilibrium will become biological for some intervals in q, because it states that the intercepts of lines $n_{12}$, $n_{13}$ and $n_{23}$ appear always in the same order on $L_3$. Notice the appearance for the first time of a hypothesis stating that forgiveness q of individuals adopting strategy 3 must not be too close to 0. This will happen in other parts of this section and has the clear meaning that strategy 3 must be more forgiving than strategy 1 for some of the results to be true. If it were otherwise, then what we are calling strategy 3 would be closer to TFT than strategy 1 and strategy 1 more generous than strategy 3, their names ATFT and G appearing reversed.

Lemma 1

(Order on $L_3$) If $q\in [(\epsilon _1+\epsilon _2)^{1/2},1]$, then $0< x_1(P_{123}) <x_1(P_{233}) < x_1(P_{133})<1$.

Proof

For ease of comparison, we may rewrite all three quantities in the form $x_1(P_{ij3})= \frac{1}{1+\tilde{c}_{ij}}$, where formulas for the $\tilde{c}_{ij}$ will be presented. We will show that $0<\tilde{c}_{13}<\tilde{c}_{23}<\tilde{c}_{12}$, from which the claim will be a trivial consequence.

An easy calculation leads to

$$\begin{aligned} \tilde{c}_{13}=\frac{a_{31}(q)-F(\frac{\epsilon _1}{\epsilon _2})}{q(P-S)} \end{aligned}$$

and

$$\begin{aligned} \tilde{c}_{23}=\frac{a_{31}(q)-P-\epsilon _2(T-P)}{q(P-S)}. \end{aligned}$$

$\tilde{c}_{12}$ may be obtained in (22). As $a_{31}(q)$ is increasing and $a_{31}(0)= F(\frac{\epsilon _1}{\epsilon _2})$, then $\tilde{c}_{13}>0$. If the ratio $\frac{\epsilon _1}{\epsilon _2}$ is fixed and $\epsilon$ is taken small enough, we obtain, using (ii) and (iii) in Proposition 1, that $F(\frac{\epsilon _1}{\epsilon _2})> P + \epsilon _2 (T-P)$, thus proving that $\tilde{c}_{13}<\tilde{c}_{23}$ for small enough $\epsilon$ and $q>0$.

Using (19) and, again, the fact that $a_{31}(q)$ is increasing, we may see that, if $q>(\epsilon _1+\epsilon _2)^{1/2}$,

$$\begin{aligned} \tilde{c}_{23} < \frac{R-(R-S)\epsilon _1-P-\epsilon _2(T-P)}{(\epsilon _1+\epsilon _2)^{1/2}(P-S)} , \end{aligned}$$

which increases as $r^{-1/2}$ when $r \rightarrow 0$. On the other hand, by (22) and (iv) in Proposition 1, we see that $\tilde{c}_{12}$ increases as $r^{-1}$. We conclude that $\tilde{c}_{23}<\tilde{c}_{12}$ for small enough $\epsilon$. $\square$

We may now define two important thresholds for q related to when the AG and DG equilibria become or cease to be biological:

Definition 2

According to Corollary 1,

$$\begin{aligned} a_{13}(q)=R \end{aligned}$$

has a unique root in (0, 1). Let $q_{AG}$ be this root.

Let also

$$\begin{aligned} q_{DG}= \frac{R-P}{T-P} \end{aligned}$$

(23)

be the unique root of equation $a_{23}=R$.

With these definitions we prove an important general result:

Theorem 1

Let $q \in (\epsilon _2,1]$. Then:

• The DG-isocline intercepts $L_1$ if and only $q\le q_{DG}$ and intercepts $L_2$ if and only if $q \in [q_{DG},1]$. In particular, the DG-equilibrium is biological if and only if $q < q_{DG}$.

• The AG-isocline intercepts $L_1$ if and only $q\le q_{AG}$ and intercepts $L_2$ if and only if $q \in [q_{AG},1]$. In particular, the AG-equilibrium is biological if and only if $q > q_{AG}$.

Proof

After easy calculations we get

$$\begin{aligned} x_2(P_{231})= & {} \frac{1}{1+ \frac{q(P-S)}{R-P-q(T-P)}}, \end{aligned}$$

(24)

$$\begin{aligned} x_1(P_{232})= & {} \frac{1}{1- \frac{a_{31}(q)-P - \epsilon _2 (T-P)}{R-P-q(T-P)}}, \end{aligned}$$

(25)

$$\begin{aligned} x_1(P_{132})= & {} \frac{1}{1+ \frac{a_{31}(q)-F(\epsilon _1/\epsilon _2)}{a_{13}(q)-R}} , \end{aligned}$$

(26)

and

$$\begin{aligned} x_2(P_{131}) \,=\, \frac{1}{1 - \frac{(q-\epsilon _2)(P-S)}{a_{13}(q)-R}} , \end{aligned}$$

(27)

all in the form $1/(1+X)$, which will be in (0, 1) if and only if the corresponding X is positive. In all four cases the proof that the necessary X is positive if and only if the respective condition on q is satisfied is trivial. In the cases related to the AG-isocline, we must use item (iii) in Proposition 2. $\square$

Besides $q_{DG}$ and $q_{AG}$ we will define a threshold $q_{AD}$, which will signal the passage of the AD-isocline through the origin. In order to do that, let $\mu (q)$ be defined by

$$\begin{aligned} \mu (q)=a_{13}(q)-P-q(T-P). \end{aligned}$$

(28)

In terms of this new function we may easily obtain

$$\begin{aligned} x_2(P_{121}) \,=\, \frac{\mu (q)}{\mu (q)+ \epsilon _2 (P-S)} \end{aligned}$$

(29)

and

$$\begin{aligned} x_1(P_{122}) \,=\, \frac{\mu (q)}{\mu (q)- [F(\epsilon _1/\epsilon _2)-P- \epsilon _2 (T-P)]} , \end{aligned}$$

(30)

which show that the AD-isocline passes through the origin of the $(x_1, x_2)$ plane whenever $\mu$ has a zero.

The following lemma proves existence and uniqueness of such a zero:

Lemma 2

Function $\mu$ defined by (28) has a single critical point $\overline{q}$ and a single zero $q_{AD}$ in (0, 1) such that $q_{AD}>\overline{q}$. Furthermore, $\overline{q}$ is a maximum point, $\mu$ is positive in $(0,q_{AD})$ and negative in $(q_{AD},1]$.

Proof

As $\mu '(q)=a'_{13}(q)-(T-P)$, then (iv) and (v) in Proposition 2 imply that $\mu '(0)>0$ and $\mu '(1)<0$ if $\epsilon$ is small enough. Then $\mu$ has at least a critical point $\overline{q} \in (0,1)$. Item (vi) in the same proposition proves uniqueness for $\overline{q}$ and that it must be a maximum point.

As $\mu (0)=F(\frac{\epsilon _1}{\epsilon _2})-P>0$, then $\mu (\overline{q})>0$. And as $\mu (1)=R-T-(T-R)\epsilon _1<0$, then $\mu$ has a unique zero in (0, 1) located in $(\overline{q},1)$. The assertion on the signs of $\mu$ follows from the fact $\mu '(q_{AD})<0$. $\square$

It is now time to start displaying important results in which the sign of $G_1$ defined in (7) plays an important role. The first thing to notice is that formula (10) for $a_{13}(q)$ is notably simplified when $G_1=0$. Solving equation $a_{13}(q)=R$ becomes then trivial and we get, for $G_1=0$,

$$\begin{aligned} q_{AG}= \frac{2R-S-T}{R-S}. \end{aligned}$$

If we calculate the difference between this value and $q_{DG}$ we discover the identity

$$\begin{aligned} \frac{2R-S-T}{R-S} - q_{DG} \,=\, \frac{G_1(T-R)}{(R-S)(T-P)}, \end{aligned}$$

(31)

which shows that $q_{AG}$ and $q_{DG}$ coincide when $G_1=0$. As $a_{13}(q_{AG})=R$, we discover that $\mu (q_{AG})= (q_{DG}-q_{AG})(T-P)$, from which we can deduce that $q_{AD}$ also coincides with $q_{AG}$ and $q_{DG}$ when $G_1=0$.

If $G_1 \ne 0$, although more complicated, equation $a_{13}(q)=R$ leads only to a second-degree equation in q and a closed formula for $q_{AG}$ can also be obtained. If we solve the equation in the auxiliary variable x defined in (20) and notice that q and x differ by O(r), we prove in general that

$$\begin{aligned} q_{AG}=\frac{2R-S-T}{R-S}-\frac{G_1(T-R)}{(R-S)(2R-S-T)}\epsilon _1+O(r^2), \end{aligned}$$

(32)

with the interesting consequence that the exact calculated value of $q_{AG}$ for $G_1=0$ holds as a good approximation for $q_{AG}$ even when $G_1 \ne 0$.

By using the ideas above we can easily prove

Theorem 2

• (i) If $G_1=0$, then $q_{AG}=q_{DG}=q_{AD}$.

• (ii) If $G_1<0$, then $q_{AG}<q_{DG}<q_{AD}$.

• (iii) If $G_1>0$, then $q_{AG}>q_{DG}>q_{AD}$.

Equation (32) will be useful later to guarantee that $q_{AG}$ does not tend to 0 when $r \rightarrow 0$. We will also need to prove the same for $q_{AD}$. This is an easy consequence of the next result.

Proposition 4

$$\begin{aligned} q_{AD}=q_{DG}+O(r) \end{aligned}$$

(33)

Proof

Using (10) in (28), we may rewrite equation $\mu (q)=0$ as

$$\begin{aligned}&\frac{R(1-\epsilon _2)-\epsilon _1 S}{1-\epsilon _1-\epsilon _2}- \frac{\epsilon _1[2R-S-T+(T-R)(\epsilon _1+\epsilon _2)+G_1 \epsilon _1]}{1-\epsilon _1-\epsilon _2} \, \frac{1}{x}\\&\quad + \frac{G_1 \epsilon _{1}^{2}}{1-\epsilon _1-\epsilon _2} \, \frac{1}{x^2} = P \,+\, \frac{x-\epsilon _1-\epsilon _2}{1-\epsilon _1-\epsilon _2}\,(T-P), \end{aligned}$$

which solution $x_{AD}$ in x will yield $q_{AD}$ through (20). When $\epsilon _1=\epsilon _2=0$, the solution is simply $x_{AD}=x_0=q_{DG}$.

Substituting $x_0$ in the above equation, then $x_{AD}$ is implicitly given by the solution of $g(x,\epsilon )=0$, where

$$\begin{aligned} g(x, \epsilon )= & {} -\frac{\epsilon _1[2R-S-T+(T-R)(\epsilon _1+\epsilon _2)+G_1 \epsilon _1]}{1-\epsilon _1-\epsilon _2} \, \frac{1}{x}\\&+ \frac{G_1 \epsilon _{1}^{2}}{x^2} \,+\, x_0-x + \epsilon _1(T-S)+ \epsilon _2(T-R) . \end{aligned}$$

As $g(x_0,0)=0$ with $\frac{\partial g}{\partial x}(x_0,0)=-1+O(r) \ne 0$ for small enough r, the implicit function theorem proves that in some neighborhood of $\epsilon =0$ the root $x_{AD}$ of $g(x,\epsilon )=0$ is a differentiable function of $\epsilon$. Differentiability implies that $x_{AD}=x_0+O(r)$. Noticing that $q_{AD}$ and $x_{AD}$ differ by O(r) leads to (33). $\square$

The first result in the next lemma shows that if q does not tend to 0 as $r \rightarrow 0$, then the difference $a_{13}(q)-a_{13}(0)$ also remains large compared to r. The second result in the lemma shows that point $P_{132}$ moves very slowly unless q is very close to 0.

Lemma 3

Let $q_0 \in (0,1]$ be fixed and independent of $\epsilon$. Then:

• $a_{31}(q_0)-F(\frac{\epsilon _1}{\epsilon _2})$ does not tend to 0 when $r \rightarrow 0$.

• If $q \in [q_0,1]$, $\frac{d}{dq}x_1(P_{132})= O(r)$.

Proof

Using variable x, defined in (20), and (11) we may write

$$\begin{aligned}&a_{31}(q)-F\left( \frac{\epsilon _1}{\epsilon _2}\right) \,=\, R-F\left( \frac{\epsilon _1}{\epsilon _2}\right) +\frac{(R-T)\epsilon _1}{1-\epsilon _1-\epsilon _2}\\&\quad -\frac{\epsilon _1}{1-\epsilon _1-\epsilon _2}\frac{2R-S-T-[(T-P)\epsilon _1+(R-S)\epsilon _2]}{x} +\frac{G_1 \epsilon _{1}^{2}}{(1-\epsilon _1-\epsilon _2)x^2}. \end{aligned}$$

Remember that if $q_0$ is fixed, the corresponding x does not tend to 0 when $r \rightarrow 0$. So, except for the term $R-F(\frac{\epsilon _1}{\epsilon _2})$, the remaining terms in the right-hand side do tend to 0 when $r \rightarrow 0$. On the other hand, (v) in Proposition 1 proves that $R-F(\frac{\epsilon _1}{\epsilon _2})$ is positive and does not tend to 0. This proves the first part.

In order to prove the second part, notice that

$$\begin{aligned} \frac{d}{dq}x_1(P_{132})= & {} \frac{a'_{13}(q)\left( a_{31}(q)-F(\frac{\epsilon _1}{\epsilon _2})\right) +(R-a_{13}(q))a'_{31}(q)}{\left[ a_{13}(q)-R+a_{31}(q)- F(\frac{\epsilon _1}{\epsilon _2})\right] ^2}. \end{aligned}$$

If $q \ge q_{0}$, then (10) implies that $a_{13}(q)-R$ is O(r). Moreover, by the first part of this Lemma, $a_{31}(q)- F(\frac{\epsilon _1}{\epsilon _2})$ does not tend to 0 when $r \rightarrow 0$. Thus the denominator in the expression above does not tend to 0 when $r \rightarrow 0$. In the numerator both terms are O(r), as can be seen in (21) and its analog for $a_{13}$. As a consequence, the derivative of $x_1(P_{132})$ is O(r). $\square$

The following monotonicity result will also turn out to be fundamental ahead.

Proposition 5

If $q \in [q_{AD},1]$, then

$$\begin{aligned} \frac{d}{dq}(x_1(P_{122})-x_1(P_{132}))>0. \end{aligned}$$

Proof

Formulas for $x_1(P_{122})$ and $x_1(P_{132})$ have already been given, see (30) and (26).

Equation (33) guarantees that $q_{AD}$ does not tend to 0 as $r \rightarrow 0$. Then, by Lemma 3, using e.g. $q_0=1/2 q_{DG}<q_{AD}$, we conclude that $\frac{d}{dq}x_1(P_{132})$ is O(r) in $[q_{AD},1]$.

By an easy calculation, we have

$$\begin{aligned} \frac{d}{dq}x_1(P_{122}) \,=\,\frac{[(T-P)-a'_{13}(q)][F(\frac{\epsilon _1}{\epsilon _2})-P- \epsilon _2(T-P)]}{[F(\frac{\epsilon _1}{\epsilon _2})-P-\epsilon _2(T-P)-\mu (q)]^2}. \end{aligned}$$

(34)

It follows that

$$\begin{aligned} \frac{d}{dq}x_1(P_{122}) \,>\, \frac{(T-P)-a'_{13}(q_{AD})}{F(\frac{\epsilon _1}{\epsilon _2})-P-\epsilon _2(T-P)}, \end{aligned}$$

because in $[q_{AD},1]$ we have $\mu (q)\le 0$, $a'_{13}(q)<a'_{13}(q_{AD})$ and $F(\frac{\epsilon _1}{\epsilon _2})-P-\epsilon _2(T-P) >0$. In this last expression $a'_{13}(q_{AD})$ is O(r) and the denominator does not tend to 0. So there exists a positive constant C independent of r such that $\frac{d}{dq}x_1(P_{122})>C$.

We conclude that for small enough $\epsilon$, $\frac{d}{dq}(x_1(P_{122})-x_1(P_{132}))>0$ for $q \in [q_{AD},1]$. $\square$

At this point we start separating the case $G_1>0$ of the other cases $G_1=0$ and $G_1<0$. Our first result is on the interior equilibrium:

Proposition 6

If $G_1 \ge 0$, the interior equilibrium is biological for all $q \in (q_{AD},1]$. If $G_1>0$ the conclusion extends also to $q=q_{AD}$.

Proof

First of all, if $G_1 \ge 0$, by Theorem 2 we know that $q_{AD} \le q_{AG}$. For $q=q_{AD}$ we then know that the AG-isocline intercepts the sides $L_3$ (Lemma 1) and $L_2$ (Theorem 1) and then we must have $x_1(P_{132})\le 0$ for $q=q_{AD}$. But $x_1(P_{122})=0$ at $q_{AD}$. By Proposition 5 we discover that $x_1(P_{122})>x_1(P_{132})$ for all $q \in (q_{AD},1]$. Comparing this order in which the isoclines intercept side $L_1$ with the corresponding order on side $L_3$ given by Lemma 1, we see that the AD and AG isoclines must cross at the interior of B for all $q \in (q_{AD},1]$. If $G_1=0$ we already knew that the AD and AG isoclines crossed exactly at the origin for $q=q_{AD}$. But for $G_1>0$, $x_1(P_{132})<0$ already at $q=q_{AD}$, and the isoclines cross in the interior. $\square$

Our task is now to find out when the interior equilibrium first becomes biological and if it ever loses its biological status. We start with a general result:

Proposition 7

For any value of $G_1$ and $q=(\epsilon _1+\epsilon _2)^{1/2}$ we have $x_2(P_{121})>x_2(P_{131})$.

Proof

The coordinates for the intercepts of the $n_{ij}$ lines with $x_1=0$ may all be written as

$$\begin{aligned} x_2(P_{ij1})\,=\,\frac{1}{1+b_{ij}(q)}. \end{aligned}$$

where, after easy calculations, we get

$$\begin{aligned} b_{12}(q)\,=\,\frac{\epsilon _2 (P-S)}{\mu (q)} \;\;\;{\mathrm {and}} \;\;\; b_{13}(q)=\frac{(q-\epsilon _2)(P-S)}{R-a_{13}(q)}. \end{aligned}$$

(35)

As $a_{13}((\epsilon _1+\epsilon _2)^{1/2})= R- O(r^{1/2})$ and $\mu ((\epsilon _1+\epsilon _2)^{1/2})= R- P-O(r^{1/2})$, then by the above expressions our assertion is true. $\square$

The order just proved between $x_2(P_{121})$ and $x_2(P_{131})$ at $q=(\epsilon _1+\epsilon _2)^{1/2}$ is reversed at $q=q_{AD}$ if $G_1>0$. In fact, if $G_1>0$ and $q=q_{AD}$, we have $x_2(P_{131})>0$, because $q_{AG}>q_{AD}$, and $x_2(P_{121})=0$. It turns out that the AG and AD isoclines must cross on $L_1$ for at least one $q \in ((\epsilon _1+\epsilon _2)^{1/2},q_{AD})$. We will show that this crossing is indeed unique, defining $q_{int}$. If $G_1=0$, we saw that the AG and AD isoclines crossed at $q=q_{AD}$. We will also show that no other crossing will happen if $q \in ((\epsilon _1+\epsilon _2)^{1/2},q_{AD})$.

If $x_2(P_{121})-x_2(P_{131})$ were monotonic in $((\epsilon _1+\epsilon _2)^{1/2},q_{AD})$ the assertions in the preceding paragraph would be trivial. As this does not happen, we must work a bit harder, starting with

Proposition 8

Equation

$$\begin{aligned} \mu (q)= (\epsilon _1+\epsilon _2)^{1/2} \end{aligned}$$

(36)

has a single root $\tilde{q} \in [0,q_{AD}]$, which is asymptotically given by

$$\begin{aligned} \tilde{q} \,=\, q_{DG} - \frac{(\epsilon _1+\epsilon _2)^{1/2}}{T-P} + O(r) . \end{aligned}$$

(37)

In particular, $\tilde{q}$ does not tend to 0 as $r \rightarrow 0$.

Proof

Let $\overline{q}$ be the critical point of $\mu$ as in Lemma 2. As $\mu (0)> (\epsilon _1+\epsilon _2)^{1/2}$ if $\epsilon$ is small enough, and $\mu$ is increasing in $[0, \overline{q}]$, then Eq. (36) has no solution in that interval. On the other hand, as $\mu$ is decreasing in $(\overline{q},q_{AD}]$ with $\mu (q_{AD})=0$, then (36) must have one root exactly in $(\overline{q},q_{AD})$.

In order to obtain (37) we rewrite (36) using definition (28) along with (10) written in terms of variable x defined by (20). Putting $\epsilon _1=\epsilon _2=0$ we obtain the approximate solution $x \approx q_{DG}$, which suggests us to define a new auxiliary variable y as $y = (x-q_{DG}) r^{-1/2}$. Substituting $x= q_{DG}+r^{1/2} y$ in (36) and making several simplifications, we get that (36) is equivalent to $H(r,y)=0$, where

$$\begin{aligned} H(r,y)= & {} G_1 r^{3/2} \cos ^2 \theta (1-r^{1/2} y - q_{DG}) \\&- \,\frac{r^{1/2} \cos \theta [2R-S-T-r \cos \theta (T-P) - r \sin \theta (R-S)]}{q_{DG}+ r^{1/2}y}\\&+\, r^{1/2} [(T-S) \cos \theta +(T-R) \sin \theta )] \\&- \,(\cos \theta + \sin \theta )^{1/2} [1-r(\cos \theta +\sin \theta )] - (T-P)y . \end{aligned}$$

Repeating the argument with the implicit function theorem as in the proof of Proposition 4 we obtain (37). $\square$

We now prove a monotonicity argument for $x_2(P_{121})-x_2(P_{131})$, but restricted to $(\tilde{q}, q_{AD})$:

Proposition 9

If $G_1 \ge 0$, $x_2(P_{121})-x_2(P_{131})$ is a decreasing function of q in $(\tilde{q}, q_{AD})$.

Proof

$$\begin{aligned} \frac{d}{dq} x_2(P_{131}) \,=\, -\frac{ (q -\epsilon _2)(P-S) a'_{13}(q) + (R-a_{13}(q)) (P-S)}{[R-a_{13}(q)+(q -\epsilon _2)(P-S)]^2} \end{aligned}$$

shows that $\frac{d}{dq} x_2(P_{131})$ is negative and O(r) in $(\tilde{q}, q_{AD})$. In order to conclude that, we are using the fact proved in Proposition 8 that $\tilde{q}$ does not tend to 0 when $r \rightarrow 0$, which implies that both $R-a_{13}(q)$ and $a'_{13}(q)$ are O(r) for $q \ge \tilde{q}$. The denominator is of course O(1) due to the $(q -\epsilon _2)(P-S)$ term.

For $x_2(P_{121})$ we have

$$\begin{aligned} \frac{d}{dq} x_2(P_{121}) \,=\, \frac{\epsilon _2(P-S) \mu '(q)}{[ \mu (q)+\epsilon _2(P-S)]^2} \,=\, \frac{\mu '(q)}{\epsilon _2(P-S)} \left[ \frac{1}{1+\frac{\mu (q)}{\epsilon _2(P-S)}}\right] ^2. \end{aligned}$$

In $(\tilde{q},q_{AD})$, $\mu '(q) =a'_{13}(q)-(T-P) <-\frac{1}{2}(T-P)$, where we are using again that $a'_{13}(q)$ is O(r). Also,

$$\begin{aligned} \frac{1}{1+\frac{\mu (q)}{\epsilon _2(P-S)}}> & {} \frac{1}{1+\frac{\mu (\tilde{q})}{\epsilon _2(P-S)}}\,=\, \frac{\sin \theta }{\sin \theta + \frac{r^{-1/2} (\cos \theta +\sin \theta )^{1/2}}{P-S}}\\> & {} \frac{\sin \theta }{r^{-1/2}[\sin \theta + \frac{ (\cos \theta +\sin \theta )^{1/2}}{P-S}]} \ge K \sin \theta \, r^{1/2} , \end{aligned}$$

where $K= \max _{\theta \in [0, \pi /2]} \left( \sin \theta +\frac{(\cos \theta + \sin \theta )^{1/2}}{P-S}\right) >0$.

Finally, we obtain, for $q\in (\tilde{q},q_{AD})$,

$$\begin{aligned} \frac{d}{dq} x_2(P_{121}) < - \frac{1}{2}\, \frac{T-P}{P-S}\,K^2 \sin ^2 \theta , \end{aligned}$$

from which it turns out that $\frac{d}{dq} (x_2(P_{121})-x_2(P_{131}))<0$. $\square$

Putting together all known facts, we can now prove

Theorem 3

(Interior equilibrium, $G_1>0$) If $G_1>0$ and $q\in ((\epsilon _1+\epsilon _2)^{1/2},1]$, there is a single value $q_{int} \in (\tilde{q}, q_{AD})$ such that the AG, AD and DG isoclines cross on the border of B. Moreover, the crossing is on $L_1$.

Proof

By the properties of $\mu$ proved in Lemma 2, and noticing that

$$\begin{aligned} x_2(P_{121}) = \frac{1}{1+ \frac{\epsilon _2 (P-S)}{\mu (q)}}, \end{aligned}$$

it is clear that the minimum of $x_2(P_{121})$ in $[0,\tilde{q}]$ is attained at one of the boundaries of the interval. But as $\mu (0)=O(1)$, and $\mu (\tilde{q})=(\epsilon _1+\epsilon _2)^{1/2}$, the minimum is attained at $\tilde{q}$ and its value is thus $1-O(r^{1/2})$.

An easy calculation shows that the derivative of $x_2(P_{131})$ is negative in $[(\epsilon _1+\epsilon _2)^{1/2}, q_{AD})$. It can be seen also that $x_2(P_{131})= 1-O(1)$ at $q= (\epsilon _1+\epsilon _2)^{1/2}$. Thus the maximum of $x_2(P_{131})$ is less than the minimum of $x_2(P_{121})$ in $[(\epsilon _1+\epsilon _2)^{1/2}, \tilde{q}]$. This proves that the AG and AD isoclines do not cross on $L_1$ for $q \in [(\epsilon _1+\epsilon _2)^{1/2}, \tilde{q}]$.

On the other hand, they do cross for q somewhere in $(\tilde{q},q_{AD})$ because we have already seen that at $q=q_{AD}<q_{AG}$ we have $x_2(P_{131})>0=x_2(P_{121})$. We have also just proved that the reverse holds at $q=\tilde{q}$. Uniqueness of this crossing in $(\tilde{q},q_{AD})$ follows from Proposition 9. Uniqueness in $((\epsilon _1+\epsilon _2)^{1/2},1]$ is a consequence of Proposition 6. $\square$

This result settles the question of the existence of a threshold $q_{int}$ for the appearance of the interior equilibrium for $G_1>0$, and the fact that $q_{int}<q_{AD}$. We remember that the question of whether the other equilibria are biological or not is already solved in Proposition 3 and Theorem 1, and the order of the corresponding thresholds is established in Theorem 2. The results on which equilibria are biological for $G_1>0$, all justified, are summarized in Table 1. The two rightmost columns in that table (and in the other two tables) will still be the subject of the next section.

Table 1 The bullets indicate which equilibria are biological at each interval in the case $G_1>0$

As in the case $G_1>0$, the results on the equilibria for the case $G_1=0$ (donation game) are summarized in Table 2. All we need for justifying these results was already proved. It remains for us just the task remembering the needed results. First of all, in the case $G_1=0$ we define $q_{int}$ to be equal to the common value $q_{DG}=q_{AG}=q_{AD}$. We then have

Theorem 4

(Equilibria for $G_1=0$) If $G_1=0$, besides equilibria at the vertices of B and the AD-equilibrium, which are always biological, these are the biological equilibria at each interval:

• The DG-equilibrium is biological if and only $q \in [0, q_{int})$.

• If $q > (\epsilon _1+\epsilon _2)^{1/2}$, the AG and interior equilibria are biological if and only if $q \in (q_{int}, 1]$.

Proof

The assertions for the DG and AG equilibria were already proved in Theorem 1. In Theorem 6 we have already proved that the interior equilibrium is biological for $q \in (q_{int}, 1]$. The only thing remaining to be proved is that the AG, AD and DG isoclines do not cross on the border of B for $q \in ((\epsilon _1+\epsilon _2)^{1/2}, q_{int})$.

In fact, by Proposition 7 and the same argument in the proof of Theorem 3, we show that there is no crossing for $q \in ((\epsilon _1+\epsilon _2)^{1/2}, \tilde{q})$. No crossing for $q \in (\tilde{q},q_{AD})$ is a consequence of Proposition 9. Finally, for $(q_{AD},1]$ the argument is Proposition 6. So the AD, AG and DG isoclines only cross at the origin for $q=q_{int}$. $\square$

Table 2 The bullets indicate which equilibria are biological at each interval in the case $G_1=0$ (donation game)

The arguments necessary for proving validity of the equilibria results of the remaining case $G_1<0$ are similar and simpler than the ones used for the other two cases, so that we will leave them to the reader. The results are summarized in Table 3.

Table 3 The bullets indicate which equilibria are biological at each interval in the case $G_1<0$

5 The dynamics

Zeeman (1980) studied the replicator dynamics for n strategies from the point of view of the theory of Dynamical Systems. He addressed mainly the robust cases, where robust means cases in which the dynamics remains unchanged for arbitrarily small changes of parameters. In particular, for $n=3$ Zeeman showed that it was possible to obtain the phase portraits of replicator dynamics by only knowing about the existence or not of equilibria at each face of the simplex $S_3$, existence or not of an interior equilibrium, and the stability of each equilibrium. In this case the number of possible phase portraits was small enough so that each possibility could be shown. Bomze (1983) continued Zeeman’s work including also the non-robust cases, thus obtaining a complete classification of all possible 47 phase portraits for the replicator dynamics with three strategies.

As we will also be interested in some non-robust cases, in this section we will refer to the classification by Bomze. In particular, we will see that, for most among the possible intervals and values of $G_1$ in Tables 1, 2 and 3, our knowledge up to now can associate a single diagram in Bomze (1983), thus fully determining the dynamics for that interval for q and sign for $G_1$. The enumeration of the diagrams in our tables is the same as in Bomze (1983). In only one row in each table, because we were unable to determine whether the interior equilibrium was asymptotically stable, neutral or unstable, more than one diagram was found to be compatible. Of course, for fixed values of the parameters, it is straightforward to linearize the dynamics around the interior equilibrium, and by calculating eigenvalues of a $2 \times 2$ matrix, discover the missing stability information. But we found the exact expression for the eigenvalues depending on the many parameters too complicated for a rigorous analysis.

Another information necessary for reading our Tables 1, 2 and 3 is that the letter R in front of the number of a diagram in Bomze (1983) means that one should take the corresponding diagram with all arrows reversed. In fact, the reader should notice that replacing matrix A by $-A$ in Eq. (14) has only the effect of reversing the orientation of all orbits.

The last column in each of the tables refers to the type of evolution of cooperation, a consequence of the dynamics for that case. We now define each of the possible types of evolution of cooperation.

Definition 3

We will say that the population admits

• full evolution of cooperation if equilibrium $E_3$ is asymptotically stable. In other words, if there is a region $R_G \subset B$ with positive area, such that for any initial condition in $R_G$ only individuals adopting strategy G will survive after infinite time.

• partial evolution of cooperation if $E_3$ is unstable, but the AG-equilibrium $P_{132}$ is biological and asymptotically stable.

• weak evolution of cooperation if $E_3$ is unstable, $P_{132}$ is unstable or not biological and there is a region $R_{ADG} \subset B$ with positive area, such that for any initial condition in $R_{ADG}$ individuals adopting all the three strategies will survive for infinite time.

• no evolution of cooperation if the dynamics leads to extinction of the G individuals for any initial condition in a region of total area contained in the triangle B.

We now start describing how we obtained the “Diagrams” column in each of the tables. In order to do that, we must know about the dynamics restricted to the sides of B. On each side of B one of the strategies is absent and we only need to study the one-dimensional replicator dynamics for two strategies. Results for this case are rather trivial, see e.g. (Nowak 2006a, p. 50), and only depend on the strategies being or not Nash equilibria. The results enumerated below are simple consequences of pairwise comparisons between elements of the pay-off matrix (6).

• 1. In the absence of strategy 3, strategies 1 and 2 are both strict Nash equilibria. Thus, for the replicator dynamics restricted to side $L_3$, the AD-equilibrium is unstable. Moreover, for any $q \in ((\epsilon _1+\epsilon _2)^{1/2},1]$ the fitness ranking is fixed, see Lemma 1, so we can divide this side in the following four regions:

  • In the region between $E_2$ and $P_{123}$, we have $f_2>f_1>f_3$.

  • In the region between $P_{123}$ and $P_{233}$. we have $f_1>f_2>f_3$.

  • In the region between $P_{233}$ and $P_{133}$. we have $f_1>f_3>f_2$.

  • In the region between $P_{133}$ and $E_1$. we have $f_3>f_1>f_2$.

  In particular, between $E_2$ and $P_{233}$, $f_3$ is the smallest fitness. This implies that $f_3< \phi$ in a neighborhood of $P_{123}$, with the consequence that orbits in the interior of B close to $P_{123}$ must have $\dot{x}_3<0$, and flow towards $L_3$, which makes $P_{123}$ a saddle point. This property is particularly important, because in some cases it is necessary in order to discard some diagrams in Bomze (1983) otherwise compatible.

• 2. In the absence of strategy 1, we have two possibilities:

  • If $q<q_{DG}$, the DG-equilibrium is biological, and as strategies 2 and 3 are both strict Nash equilibria, then the DG-equilibrium is unstable if the dynamics is restricted to the $L_1$ side.

  • If $q \ge q_{DG}$, the DG-equilibrium is not biological, and only strategy 2 is a Nash equilibrium. Then all orbits on $L_1$ must flow into $E_2$.

• 3. In the absence of strategy 2, we also have two possibilities:

  • If $(\epsilon _1+\epsilon _2)^{1/2}<q \le q_{AG}$, then the AG-equilibrium is not biological, and between strategies 1 and 3, only 3 is a Nash equilibrium. All orbits on $L_2$ must flow into $E_3$.

  • If $q>q_{AG}$, then the AG-equilibrium is biological. Because neither strategy 1, nor strategy 3 are Nash equilibria, then the AG-equilibrium is asymptotically stable when dynamics is restricted to $L_2$.

Using an example we now explain how we have obtained all the results in the “Diagrams” column in the table. The example we take is the first line in all three tables, for its greater importance regarding understanding of the results in Nowak and Sigmund (1992). We know that for $(\epsilon _1+\epsilon _2)^{1/2}<q<\min \{q_{AG},q_{int}\}$, regardless of $G_1$ we will have as biological equilibria only the three vertices, and the AD and DG equilibria. By the above reasoning on the dynamics at the border of B, $E_1$ must be a saddle point, whereas $E_2$ and $E_3$ are attractors, the AD-equilibrium is a saddle with outgoing orbits on $L_3$, and the DG-equilibrium has outgoing orbits on $L_1$. Point Q is not in the biological region.

Among the diagrams in Bomze (1983) not a single one is compatible with the above situation. But if we reverse the orbits, then diagrams 37 and 38 become compatible. The only one which remains compatible when we take into account that interior orbits close to the AD-equilibrium must flow towards $L_3$ is 38R. In Fig. 1 we show a plot of some numerically calculated orbits for a choice of parameters in one of the cases, the first row in Table 1, leading to diagram 38R. Notice that all orbits below the separatrix joining the DG and AD equilibria lead to survival only of strategy G. We have full evolution of cooperation, where $R_G$ is the region below the mentioned separatrix. For orbits in $R_G$ starting very close to the AD-equilibrium $P_{123}$ we can see the occurrence of the phenomenon in the experiment of Nowak and Sigmund (1992): an initial population with majority of ALLD, some ATFT and a minority of G individuals evolves to a population where only the G individuals are present, after passing through a transient in which the ATFT comprise almost the entire population. The phenomenon is illustrated by the graphs of fractions $x_1$, $x_2$ and $x_3$ as functions of time in Fig. 1.

Fig. 1

Phase portrait and graph of the fractions $x_1$, $x_2$ and $x_3$ as functions of time for a choice of parameters corresponding to the first row in Table 1: $T=5$, $R=4$, $P=2$, $S=0$, $\epsilon _1=0.05$, $\epsilon _2=0.10$, $q=0.40$. The initial condition for the graphs is $(x_1,x_2)= (0.18, 0.76)$. Observe the full evolution of cooperation

All information about diagrams in our tables was obtained in a way similar to the example treated above.

As already mentioned, in the cases at the last line of each Tables 1, 2 and 3 we could not find a rigorous argument for proving which of diagrams 12, 12R and 13 is the correct one.

In the case $G_1>0$, we already know for interval $(q_{DG},q_{AG})$—see third row in Table 1—that the only compatible diagram is 15R, in which the interior equilibrium is unstable and, consequently, there is no evolution of cooperation. It is not reasonable that increasing q will foster cooperation. In fact, larger values of q will make the G individuals more susceptible to exploitation by ALLDs. Thus the natural conjecture is that if $G_1>0$ and $q \in (q_{AG},1]$ the interior equilibrium will still be unstable and no evolution of cooperation will happen. If this conjecture is true, then the associated diagram must be 12R. The conjecture is supported also by numerical calculation of the eigenvalues of the linearized dynamics around the interior equilibrium and by results in Fig. 2.

Fig. 2

Phase portrait and graph of the fractions $x_1$, $x_2$ and $x_3$ as functions of time for a choice of parameters corresponding to the last row in Table 1: $T=5$, $R=4$, $P=2$, $S=0$, $\epsilon _1=0.05$, $\epsilon _2=0.10$, $q=0.90$. The initial condition for the graphs is $(x_1,x_2)= (0.40,0.20)$. Observe that there is no evolution of cooperation for this set of parameter values

In the $G_1<0$ case (Table 3) we know that there is full evolution of cooperation until $q=q_{AG}$ and only partial evolution for $q_{AG}<q<q_{int}$, due to the AG-equilibrium entering the biological region and destabilizing $E_3$. For larger values of q the interior equilibrium becomes biological and we have no knowledge on its stability. Numerical calculation of eigenvalues suggests that in the case $G_1<0$ the interior equilibrium is asymptotically stable and will attract all orbits in a region of positive area, which means weak evolution of cooperation and that the correct diagram should be 12. This conjecture is illustrated in Fig. 3.

Fig. 3

Phase portrait and graph of the fractions $x_1$, $x_2$ and $x_3$ as functions of time for a choice of parameters corresponding to the last row in Table 3: $T=5$, $R=3$, $P=1$, $S=0$, $\epsilon _1=0.05$, $\epsilon _2=0.10$, $q=0.70$. The initial condition for the graphs is $(x_1,x_2)= (0.10,0.80)$. Observe the weak evolution of cooperation for this set of parameter values, with the interior equilibrium Q being asymptotically stable

Finally, in the $G_1=0$ case for $q>q_{int}$ we may expect a situation intermediate between the other two cases. Numerical calculations suggest that the real part of the eigenvalues of the linearized dynamics around the interior equilibrium may be null. Numerically calculated orbits around the interior equilibrium seem to be closed. The correct diagram is conjectured to be 13, in which the interior equilibrium is a center, leading to weak evolution of cooperation. Figure 4 illustrates this conjecture.

Fig. 4

Phase portrait and graph of the fractions $x_1$, $x_2$ and $x_3$ as functions of time for a choice of parameters corresponding to the last row in Table 2: $T=5$, $R=3$, $P=2$, $S=0$, $\epsilon _1=0.05$, $\epsilon _2=0.10$, $q=0.50$. The initial condition for the graphs is $(x_1,x_2)= (0.40,0.20)$. Observe the weak evolution of cooperation for this set of parameter values, with the interior equilibrium Q being possibly neutral

6 Conclusions

We have proved that the results of the computer experiment in Nowak and Sigmund (1992) are true for a simplified version of the situation in which, instead of 100 reactive strategies, we only have the three more prominent ones in the experiment: ATFT, ALLD and G. More precisely, if we define

$$\begin{aligned} q_{GTFT} \,=\, \min \{q_{AG}, q_{DG}\} \end{aligned}$$

(38)

then for $q\in ((\epsilon _1+\epsilon _2)^{1/2},q_{GTFT})$ there exists a region $R_{G} \subset B$ with positive area such that the orbit of the replicator dynamics for any initial condition in $R_G$ will converge to $E_3$, i.e. only the G individuals will survive. Using our Definition 3, we can rephrase this: we showed existence of a maximum forgiveness $q_{GTFT}$ given by (38) such that if $\epsilon$ is small enough and $q\in ((\epsilon _1+\epsilon _2)^{1/2}, q_{GTFT})$, irrespective of the sign of $G_1$, we will have full evolution of cooperation in our version with three strategies for the situation in Nowak and Sigmund (1992).

In Molander (1985), the statement of Theorem 1 introduces $q_{GTFT}$ as a value “arbitrarily close to”

$$\begin{aligned} \min \left\{ \frac{2R-S-T}{R-S}, q_{DG}\right\} \end{aligned}$$

(39)

for “low noise levels”, i.e. for small $\epsilon$. Although Molander was aware that the value in (39) should be corrected due to the noise, in his paper there is no expression such as (32) to quantify this correction.

In Nowak and Sigmund (1992), the same (39) appears as the definition of the GTFT strategy winner in their computer experiment. Although the arguments in the present paper cannot be naively extended to a case such as theirs with 100 strategies, we believe that the correct value for which the results in Nowak and Sigmund (1992) should hold is instead (38). For the parameters $T=5$, $R=3$, $P=1$, $S=0$ and $\epsilon _1=\epsilon _2= 0.01$, we find $q_{GTFT}\approx 0.339574$ by numerically solving equation $a_{13}(q)=R$. This value is, for the accuracy conditions of the experiment, indistinguishable of the 1 / 3 given by (39). Despite their possibly inexact definition in Nowak and Sigmund (1992), we should note that Nowak and Sigmund had been more precise in defining $q_{GTFT}$ in their previous paper (Nowak and Sigmund 1990) (see e.g. their first Theorem at page 258), although no formula such as (38) was given.

If we understand $q_{GTFT}$ as the maximum forgiveness such that there is full evolution of cooperation for any values of $\epsilon _1$ and $\epsilon _2$ in our simplified version of the experiment with only three strategies, then we are forced to take a limit of (38) when $\epsilon \rightarrow 0$ and (39) is recovered. If, on the other hand, we take a fixed ATFT strategy, i.e. fixed values of $\epsilon _1$ and $\epsilon _2$, then (38) should be thought as the rigorous version of (39) for only three strategies as in this paper. If we have a situation as in Nowak and Sigmund (1992) with more than three strategies and a fixed ATFT, we do not know which among (38), (39) or some other number is the right value for the maximum forgiveness. We conjecture that our value (38) might be the correct one. A repetition of the experiment in Nowak and Sigmund (1992) having this question in mind might help in solving this puzzle.

We have also partially understood the population dynamics for values of q larger than $q_{GTFT}$. We have seen that in some cases some weaker forms of cooperation evolution will still hold, but we have also seen that if $G_1>0$ and $q \in (q_{DG},q_{AG})$ no evolution of cooperation is possible, because for almost all initial conditions only ALLD individuals will survive. The same conclusion probably holds also for $q \ge q_{AG}$ and $G_1>0$.

As already stated in our Introduction section, much has changed in this field since the discovery of the so-called zero-determinant (ZD) strategies for the IRPD (Press and Dyson 2012). ZD strategies are memory-one strategies, a set which contains the reactive strategies considered in this study and much more. Among the various types of ZD strategies, extortion and generous strategies (Stewart and Plotkin 2013) deserve some mention here, because they seem to have roles similar respectively to ATFT and G when the evolutionary context is taken into account. According to Hilbe et al. (2013), extortion ZD strategies “can act as catalysts for the evolution of cooperation, similar to tit-for-tat, but $\cdots$ they are not the stable outcome of natural selection”. On the other hand, in the context of all memory-one strategies, Akin (2012) defines good strategies, also called partner strategies in Hilbe et al. (2015). In a context different from ours, Stewart and Plotkin (2013) claim that good ZD strategies, which includes GTFT, “forgive defecting opponents but nonetheless dominate in evolving populations”.

We feel that considering memory-one strategies would take us out of the original context of the reactive strategies as in the experiment in Nowak and Sigmund (1992). Moreover, it would also introduce technical difficulties, as formulas (5) and (6) would cease to be true. We believe that extending the results of this paper to the setting of the more general memory-one strategies, including thus the ZD strategies, is another important task to be accomplished by researchers in the near future.

As the main tool used in this work for predicting the dynamics from knowledge on the equilibria is the two-dimensional classification in Bomze (1983), increasing the number of considered strategies from three to only four would imply rather difficult technical difficulties. We also expect that these technical barriers may be overcome by future researchers.