1 Introduction

The term “cancer” refers to a group of diseases, which can affect almost any tissue and organ and are characterized by the uncontrolled growth of abnormal cells, which cell cycle is much faster than that of the normal cells. Cancer appears as a result of a series of mutations of normal cells, which occur during the DNA replication process or as a result of a somatic mutation. Cancer cells are usually characterized by their genome instability, and as a consequence of this, by extremely high levels of mutability and evolvability. The genome instability, as well as the mutability and evolvability of cancer, is one of the cancer hallmarks [3,4,5]. As a result of this very high mutability, a typical tumor is composed of a very large number of cancer genotypes. Moreover, the mutability, evolvability, and the resulting genetic diversity of cancer make its treatment very difficult.

The mentioned genome instability, high mutability, and evolvability of cancer makes its study from the point of view of evolutionary biology essential. Accordingly, there is a growing interest and a certain progress in mathematical modeling of cancer evolution. Usually, a mathematical model of cancer evolution utilizes the idea of quasi-species (or multi-strain modeling) and is formulated in the form of a system of ordinary differential equations, partial differential equations, or integrodifferential equations (see, e.g., [9,10,11, 14, 15], and bibliography therein; the same conceptual ideas were also developed and applied for mathematical modeling of viral evolution [1, 2, 6,7,8, 13]). In this paper, we use these ideas to construct a reasonably simple mechanistic model of cancer evolution on the basis of the model of cancer and normal cells competition.

2 Model

In order to model cancer evolution, let us assume that there is a system composed of the normal cells and cancer cells of n different genotypes, where \(n\rightarrow \infty \) or is a very large number. Let us denote the size of cell population of the ith genotype at time t by \(C_i(t)\) and the size of the normal cells population by \(C_0(t)\). We assume that (i) all the cells reproduce and die; (ii) there are limited resources, which limit populations growth through inhibiting the reproduction and accelerating the death; (iii) cells of the different genotypes have to compete for these limited resources; (iv) in the process of mitosis, with some probability \(p_{ij}\), a cell of the ith genotype can produce a mutant daughter cell of the jth genotype, which, subsequently, goes to the jth population; and (v) as a result of somatic mutation, with probability \(q_{i,j}\) a cell of the ith genotype can move to the jth genotype.

We start at the Lotka–Volterra model of competing populations which is a usual basis for cancer modeling [12]:

$$\begin{aligned} \dot{C_i}(t)=r_i C_i\left( 1- \frac{1}{K}\sum \limits _{j=0}^n b_{ij} C_j \right) , \end{aligned}$$
(1)

where \(i=0,1,2,\ldots ,n\). In these equations, \(r=(r_i)_{i=0}^n\) represents the vector of the growth rates, the elements of matrix \(B=(b_{ij})_{i,j=0}^n\) represent the relative competitive capabilities of the genotypes and K is the carrying capacity of the system. To introduce into this model, a possibility of mutations that occur during DNA replication, it is necessary to separate the birth and the death rates, as these mutations occur in the process cell reproduction. For these two processes, we have

$$\begin{aligned} \text {birth rate of the }i\text {-th population}= a_i C_i\left( 1-\frac{h_i}{K} \sum \limits _{k=0}^n b_{ik}C_k \right) , \end{aligned}$$
(2)
$$\begin{aligned} \text {death rate of the }i\text {-th population}= - d_i C_i\left( 1+\frac{g_i}{K} \sum \limits _{j=0}^n b_{ik}C_k \right) . \end{aligned}$$
(3)

Here, \(a_i\) and \(d_i\) are the per capita birth and death rates of the ith genotype cells, and \(h_i\) and \(g_i\) are weights that fine-tune the relative impacts of the lack of resources and competition on the proliferation and death rates, respectively. These parameters are related to \(r_i\) by the equalities \(r_i=a_i-d_i\) and \(r_i=a_ih_i+d_ig_i\).

Then, using an approach suggested in [9], we can introduce a possibility of mutation into the model modifying the birth term (2). Then the growth of the ith genotype population is represented by the following equation:

$$\begin{aligned} \dot{C_i}=\sum \limits _{j=0}^n \left( p_{ji} a_j C_j \left( 1-\frac{h_j}{K} \sum \limits _{k=0}^n b_{jk}C_k \right) \right) - d_i C_i\left( 1+\frac{g_i}{K}\sum \limits _{k=0}^n b_{ik}C_k \right) + \sum \limits _{j=0}^{n} q_{ji}C_j - \sum \limits _{j=0}^{n} q_{ij}C_i \, . \end{aligned}$$
(4)

Here, the diffusion-like term \(\sum _{j=0}^{n} q_{ji}C_j-\sum _{j=0}^{n} q_{ij}C_i\) represents the somatic mutations. All the parameters of this model are positive real numbers, except for the weights \(h_i\) and \(g_i\) and the elements of probability matrices \(P=(p_{ij})\) and \(Q=(q_{ij})\), which can be zero.

This discrete variant space model is formulated as a system of ODEs. An equivalent continuous variant space model is formulated in [11].

To non-dimensionalize the system (4), we introduce nondimensional variables \(x_i(\tau )\) and \(\tau \) as the following:

$$\begin{aligned} x_i= b_{ii} C_i / K \, , \qquad \tau = T t \, , \qquad T= p_{00}a_{0}h_0 + d_{0}g_0. \end{aligned}$$
(5)

Please note that \(d_0\) and \(p_{00}\) are always positive, whereas \(h_0\) and \(g_0\) cannot be equal to 0 simultaneously and, hence, T is always positive. Substituting these variables into the system (4) and separating the linear and the nonlinear parts of the equations, we rewrite the system in the following form:

$$\begin{aligned} \frac{dx_i}{d\tau }=\sum \limits _{j=0}^n u_{ij}x_j -\sum \limits _{j=0}^n \sum \limits _{k=0}^n v_{ij} f_{jk} x_j x_k \, , \quad i=0,\ldots , n \, . \end{aligned}$$
(6)

Here,

$$\begin{aligned} u_{ij}={\left\{ \begin{array}{ll} \frac{ (p_{jj} a_j + q_{jj} - d_j - \sum _{k=0}^{n} q_{jk} ) e_{ij}}{T} &{} \text {if } j=i \, , \\ \frac{( p_{ji} a_j + q_{ji}) e_{ij}}{T} &{} \text {if } j\ne i \, , \\ \end{array}\right. } \qquad v_{ij}= {\left\{ \begin{array}{ll} \frac{( p_{jj} a_j h_j + d_j g_j ) e_{ij}}{T} &{} \text {if } j=i \, , \\ \frac{p_{ji} a_j h_j e_{ij}}{T} &{} \text {if } j\ne i \, , \end{array}\right. } \end{aligned}$$

and \(f_{ij}=b_{ij}/b_{jj}\) and \(e_{ij}=b_{ii}/b_{jj}\).

3 Simulations

To illustrate model behavior, we run numerical simulations. In these simulations, we assume that \(h_i=0\) for all \(i=0,\ldots ,n\). (That is, a shortage of the resources does not affect the reproduction; it is equivalent to an assumption that a decrease of new births is attributed to an increment of deaths.) Furthermore, we assume that \(Q=(q_{ij})=0\) for all \(i,j=0,\ldots ,n\). (That is, we disregards somatic mutations.) We assume that a cell of the ith genotype can produce a daughter cell only of ith, \((i-1)\)th, or \((i+1)\)th genotypes with probabilities given by matrix P:

$$ P=\left[ \begin{matrix} 0.9 &{} 0.1 &{} 0 &{} \cdots &{} \cdots &{} 0 \\ 0.1 &{} 0.8 &{} 0.1 &{} 0 &{} \cdots &{} 0 \\ \vdots &{} \vdots &{} \vdots &{} \vdots &{} \vdots &{} \vdots \\ 0 &{} \cdots &{} 0 &{} 0.1 &{} 0.8 &{} 0.1 \\ 0 &{} \cdots &{} \cdots &{} 0 &{} 0.1 &{} 0.9 \end{matrix} \right] . $$

In the simulations, the environment carrying capacity \(K=10^{5}\) cells, and time t is measured in days. Values of the other parameters, as well as, the initial conditions used in the three simulations are summarized the following table:

Parameter

Simulation # 1

Simulation # 2

Simulation # 3

n

50

50

200

\(a_i\)

2

10

\(4-2 e^{-i}\)

\(d_i\)

0.2

0.2

\(0.2-0.1 e^{-i}\)

\(g_i\)

9

51

19

\(b_{ij}\)

\(2- i/n\)

\(1+ i/n\)

\(2- i/n\)

\(C_i(0)\)

\((K-1,1,0, \ldots ,0)\)

\((K-1,1,0, \ldots ,0)\)

\((K-1,1,0, \ldots ,0)\)

Results of the simulations are depicted in Figs. 1, 2, and 3, respectively. In simulation #1, we used \(b_{ij}=2- i/n\) for all j. This implies that the Darwinian fitness of the genotypes grows as i increases. The initial conditions in the simulation #1 implies that initially, there was present only one mutant cell of the first mutant genotype. The formation of a traveling wave moving in the direction of increasing i is clearly seen in Fig. 1. This implies that in this simulation an average fitness of the tumor population steadily increases.

Fig. 1
figure 1

Simulation #1: variation of the genotype abundance in time

Full size image

Please note that in Fig. 1, the populations of genotypes \(i=\) 47 to 50 remain approximately constant after \(t\approx 120\) days: for this simulation, we consider a system of 50 genotypes and, hence, in this simulation, the Darwinian fitness of the 50th genotype is maximal. This enables this genotype, as well as close genotypes 47th to 49th, to eventually prevail in the system. Of course, in a real-life system, the number of possible mutant genotypes n is significantly higher, and, hence, no such steady prevalence of a particular mutant genotypes can be observed.

Figure 2 shows the results of simulation #2. For this simulation, in contrast to simulation #1, we take \(b_{ij}=1+ i/n\). This implies that the Darwinian fitness decreases as i grows. The simulation confirms an intuitive expectation that in such a case, for whatever large levels of mutation probabilities, the mutations are unable to fix in the system and the mutant cells will be eventually removed from the tissue.

Fig. 2
figure 2

Simulation #2: variation of the genotype abundance in time

Full size image

In the simulation #3, we consider the impact of the proliferation rates on the evolution. It is a well-known fact that cancer cells proliferate faster than the healthy cells and that the proliferation rates depend, above all, on the degree of differentiation. Accordingly, in the simulation #3, we used \(a_i=4-2 e^{-i}\) and \(d_i=0.2-0.1 e^{-i}\), in combination with the same \(b_{ij}\) as in simulation #1. Figure 3 shows the results of simulation #3. As one can expect, in this case, a traveling wave of evolution is forming as well. However, it moves faster than in the simulation #1, where the proliferation and death rates were constant. Moreover, it is easy to see that in this case the speed of the traveling wave notably decreases as i grows (and as \(a_i\) and \(d_i\) grow).

Fig. 3
figure 3

Simulation #3: variation of the genotype abundance in time

Full size image

It is hardly surprising that the system behavior in the simulations #1 and #3 is very similar: the only difference is the higher speed of the traveling wave in simulation #3. In both these simulations, the mean genotype number grows converging to the last genotype that has the highest level of the Darwinian fitness. The variance of the genotype distribution in the population initially grows until it reaches, at a certain time \(t^*\), its maximum value, and then it slowly decreases. Such a behavior is intriguing and counterintuitive.

The analysis of the simulation results suggests that the comparative values of the competition factors \(b_{ij}\) mostly determine the system behavior and that changes of these values can change the system qualitative behavior.