1 Introduction

Cancer is characterized by uncontrolled growth of abnormal cells that appear as a result of a series of mutations of normal cells. To develop into cancer, the mutant cells should be able to successfully compete with the normal cells. It is highly surprising that this issue attracted significant attention; see, e.g., [3, 4], and literature therein.

In this paper, we focus at an initial stage of cancer and explore the conditions that are necessary for the initial fixation of the malignant mutant cells in a patient. Accordingly, we consider the dynamics of a simple mutation-selection model that comprises two interaction populations, namely, normal cells and mutant cancer cells. To address this issue, in this paper, we consider a version of the cancer evolution model suggested in [1, 2], where we consider only two classes, namely, the normal cells and the malignant cells.

2 Model

Let us consider a model described in [1]. The model postulates the existence of the normal cells and n cancer cell genotypes in the system. Let us denote the population size of the i-th genotype cell at time t by \(C_i(t)\) and the population size of the normal cells at the same time by \(C_0(t)\). The model is based upon the Lotka–Volterra model of competing populations and postulates that (i) all cells reproduce and die, (ii) there is a resource that limit populations growth, (iii) cells of different genotypes have to compete for this limited resource, (iv) in the process of mitosis, with some probability \(p_{ij}\), a cell of the i-th genotype can produce a mutant daughter cell of the j-th genotype, which subsequently goes to the j-th population, and (v) as a result of somatic mutation, with probability \(q_{i,j}\) a cell of the i-th genotype can move to the j-th genotype. This situation can be described by the following system of ordinary differential equations:

$$\begin{aligned} \dot{C_i}=\sum \limits _{j=0}^n \left( p_{ji} a_j C_j \left( 1-\frac{h_j}{K} \sum \limits _{k=0}^n b_{jk}C_k \right) \right) - d_i C_i\left( 1+\frac{g_i}{K}\sum \limits _{k=0}^n b_{ik}C_k \right) + \sum \limits _{j=0}^{n} q_{ji}C_j - \sum \limits _{j=0}^{n} q_{ij}C_i \, . \end{aligned}$$
(1)

Here, \(i=0,1,\ldots ,n\), \(a_i\) are the replication rates, \(d_i\) the death rates, K the carrying capacity, \(b_{ij}\) the competition factors, and \(h_i\) and \(g_i\) reflect the competition effects on the birth and death, respectively.

Our goal is to study the initial appearance and fixation of the mutant cells. Therefore, we consider only one type of mutant cells, or, what is the same, assume that all mutant cells are the same, and consider interaction of these with the normal tissue cells. Thus, we assume that \(n=1\) in model (1), and the model reduces to a two-dimensional system. Moreover, for simplicity, we assume that \(h_0=h_1=0\). Mathematically, this preserves the positive invariance of the first quadrant of the phase space; biologically, this means that the lack of resources increases the death rate, but does not inhibit the proliferation. For this case, denoting

$$ x=b_{00} C_0 / K , \qquad y= b_{11} C_1 / K , \qquad \tau = d_{0}g_0 t, $$
$$ A=p_{00}a_0-d_0 , \quad B=(a_1p_{10}+q_{10})b_{00}/b_{11} , \quad C=(a_0p_{01}+q_{01})b_{11}/b_{00} , \quad D=p_{11}a_1-d_1, $$
$$ \alpha =p_{00} a_0 h_0 + d_0 g_0 , \qquad \beta =d_0g_0 b_{01}/b_{11} , \qquad \gamma =d_1 g_1 b_{10}/b_{00} , \qquad \delta =p_{11} a_1 h_1 + d_1g_1, $$

the system (1) can be written as

$$\begin{aligned} \frac{dx}{d \tau }&= \frac{1}{\alpha } (Ax + By -\alpha x^2-\beta xy) \, , \end{aligned}$$
(2)
$$\begin{aligned} \frac{dy}{d \tau }&= \frac{1}{\alpha } (Cx + Dy -\gamma xy-\delta y^2) \, . \end{aligned}$$
(3)

Further, we analyze this system.

3 Equilibrium States of the Model

Equilibrium states of model (2) satisfy the following system of algebraic equations:

$$\begin{aligned} Ax + By -\alpha x^2-\beta xy =0 \, , \qquad Cx + Dy -\gamma xy-\delta y^2=0 \, . \end{aligned}$$
(4)

Accordingly, for this system, the equilibrium states correspond to the intersections of two conic curves defined by equalities (4). Of course, since the variables x and y represent sizes of populations, we are interested only in the intersections, which are located in the first quadrant of the phase space.

Let us start with some trivial observations expressed by the following lemmas:

Lemma 1

The origin \(P_0=(0,0)\) is always an equilibrium state of the system and is the only equilibrium state located on the coordinate axes, when B and C are strictly positive.

By this lemma, the system always has at least one nonnegative equilibrium state.

Lemma 2

The system (2) has from one to four equilibrium states.

Through some geometrical observation, it is possible to obtain the following results about the nullclines (4):

Lemma 3

Each of the nullclines (4) is either a hyperbola or a degenerate hyperbola (two intersecting straight lines).

Lemma 4

One of the two branches of each of the nullclines (4) has no points in the first quadrant.

Consequently,

Lemma 5

The system (2) has either one or two nonnegative equilibrium states.

That is, the system either has no positive coexisting equilibrium states at all, or have only one such equilibrium state.

3.1 Stability of Equilibrium State \(\varvec{P_0}\)

The local analysis of the system (2) near equilibrium state \(P_0\) immediately reveals that the eigenvalues of the linearized system are real numbers. Consequently, \(P_0\) cannot be a focus. Moreover, no Hopf bifurcation is possible at the origin. There are the following three possibilities:

  1. (i)

    if \(AD-BC<0\), then the origin is a saddle point;

  2. (ii)

    if \(AD-BC>0\) and \((A+D)>0\), then the origin is a repulsive (unstable) node;

  3. (iii)

    if \(AD-BC>0\) and \((A+D)<0\), then the origin is an attractive (stable) node.

In terms of the physical parameters, \((A+D)>0\) is equivalent to condition \(a_0 p_{00}+a_1 p_{11}>d_0 +d_1\). Please note that \(A+D<0\) is not impossible, as there can be a situation where one of the \(d_i\) is very large, whereas the corresponding \(a_i p_{ii}\) is sufficiently small. This means that the corresponding genotype could appear, but cannot sustain even without competition.

Biologically, this condition means that the sum of rates of birth to the same genotype cells is larger than the sum of death rates. Condition \(AD-BC<0\) is equivalent to the inequality \((a_0 p_{01}+q_{01})( a_1 p_{10}+q_{01}) >(a_0 p_{00} - d_0)(a_1 p_{11} - d_1)\).

3.2 Existence and Properties of the Positive Fixed Point

Let us consider the existence and the location of the possible positive equilibrium state \(P_*=(x_*,y_*)\).

Lemma 6

If either \(A\beta - B\alpha \ge 0\), or \( D\gamma - C\delta \ge 0\), then the positive equilibrium state \(P_*\) exists.

If \(A\beta -B\alpha <0\) and \(D\gamma -C\delta <0\), then the existence of \(P_*\) is uncertain. However, in either case, equilibrium state \(P_*\) can appear only as the result of a transcritical bifurcation that occurs at the origin. Therefore, \(P_*\) is generated when the two hyperbolas have a common tangent at the origin. This implies that the transcritical bifurcation occurs when \(AD-BC=0\). An intriguing fact is that the equilibrium state \(P_*\) can exist as when \(AD-BC<0\) as when \(AD-BC>0\).

Linearizing the system around \(P_*\), it is easy to see that this point can be either a saddle point, or an attractive node, or an attractive focus. It is possible to state the following result:

Lemma 7

No Hopf bifurcation is possible at \(P_*\).

Therefore, recalling that no Hopf bifurcation is possible at (0, 0), one can deduce the following.

Corollary 8

The model (2) admits no Hopf bifurcation.

Moreover, a sufficient (but not necessary) condition for the stability of \(P_*\) obtained via linearization is that \(\alpha \delta -\beta \gamma >0\). Thus, there are two sufficient conditions for the stability of \(P_*\):

  1. (i)

    \(\alpha \delta - \beta \gamma > 0\), or

  2. (ii)

    \(AD-BC<0\) and \(A+D<0\).

We already analyzed the second condition discussing the stability of equilibrium state \(P_0\). In terms of the physical parameters, the first condition is equivalent to \(b_{00}b_{11} - b_{01}b_{10}>0\), that is, \(\det B >0\). Biologically, it means that the system achieves the stable equilibrium when the competition effects among cells with the same genotype are stronger than the ones between the two different genotypes. It is very unlikely to occur in the case of cancer.

3.3 Global Properties of the Model

Denoting \(P(x,y)=\big ( (A-\alpha x) x + (B-\beta x)y\big )/\alpha \) and \(Q(x,y)=\big ( (C-\gamma y) x + (D - \delta y) y\big )/\alpha \), we can rewrite model (2) as follows:

$$\begin{aligned} \frac{dx}{d \tau } = P(x,y) , \qquad \frac{dy}{d \tau } = Q(x,y) \, . \end{aligned}$$
(5)

Let us study the direction of the vector field (PQ) at a point (xy) in the first quadrant. It is easy to see that, if \(x>\max \left\{ A/\alpha , B/\beta \right\} \), then \(P(x,y)<0\); analogously, if \(y>\max \left\{ C/\gamma , D/\delta \right\} \), then \(Q(x,y)<0\). Moreover, on the positive semi-axes, the vector flow is directed inside of the first quadrant. This means that the compact square

$$ S=\left\{ [0, \max \left( A/\alpha , B/\beta \right) ] \times [0, \max \left( C/\gamma , D/\delta \right) ] \right\} $$

is a positive-invariant set and an attractive region of the system. Therefore, by the Poincaré–Bendixson Theorem, it contains at least one stable limit cycle or at least one stable fixed point. For this system, it is possible to exclude the existence of a limit cycle. Moreover, by choosing \(\varphi (x,y)=\alpha \), one immediately obtains for the system (2):

$$ \text {div} (\varphi P, \varphi Q)=A+D - (2\alpha +\gamma ) x - (\beta +2\delta )y \, , $$

which is always negative if \(A+D<0\). Therefore, since S is a simply connected region, by the Bendixson–Dulac theorem, no limit cycle is possible within S when \(A+D<0\). It implies that all orbits tend to a stable fixed point. Please recall that for \(A + D<0\) the origin \(P_0\) is stable if and only if \(AD-BC>0\). Hence, for \(AD-BC<0\), positive equilibrium state \(P_*\) must exist and be stable. This means that for the case when \(A + D<0\) the necessary and sufficient condition for the existence of \(P_*\) is \(AD-BC<0\).