1 Introduction

The metastatic spread of a tumor to distant organs is the major cause of death from cancer. Tumor metastasis is often preceded by the development of morphological instability of a growing tumor where fingers, chains and sheets of cancer cells may form and invade the local microenvironment (Friedl and Wolf 2009; Friedl et al. 2012). Mathematical modeling shows that the parameters that control the tumor shape also control its invasive ability (Cristini et al. 2003, 2005; Frieboes et al. 2006; Macklin and Lowengrub 2007; Anderson and Quaranta 2008; Friedman and Hu 2007; Lowengrub et al. 2010). Therefore, analysis of tumor morphology can aid in tumor prognosis and in predicting the response to treatment. The transport of oxygen, nutrients and growth promoting factors, cell-death, cell–cell/cell-extracellular matrix (ECM) adhesion, soft tissue stress, cell proliferation, motility and death all regulate the shape of a growing tumor.

Over the past decade, there has been extensive progress in the development of mathematical and computational models of solid tumors that use a variety of approaches. In discrete models, tumors are treated as a collection of individual cells that interact with each other and with elements in their microenvironment such as ECM, blood vessels, etc. Continuum models access larger scales and use cell densities, or volume fractions, to characterize the growing populations. Hybrid models, which combine continuum and discrete models of solid tumors, have also been developed in an attempt to bridge the cell and tissue scales. We refer the reader to recent reviews (Lowengrub et al. 2010; Katira et al. 2013; Rodriguez-Brenes et al. 2013; Wang et al. 2015; Altrock et al. 2015; Tanaka 2015; Te Boekhorst et al. 2016) for references on these different approaches. In spite of these efforts, predicting tumor invasiveness and metastatic potential remains an unsolved problem, primarily because cancer is a complex system involving nonlinearly interacting processes at multiple scales.

In previous work (Pham et al. 2010) we studied the morphological stability of growing tumors using three continuum mathematical models and evaluated the consistency between theoretical model predictions from a linear stability analysis and experimental data from in vitro 3D multicellular tumor spheroids (Frieboes et al. 2006). In particular, we modeled the tumor as an incompressible fluid using a Darcy law, Stokes law and a combined Darcy–Stokes law to investigate the effects of viscous stress on the tumor dynamics. Our analysis suggested that the Stokes model, where the tumor is treated as a viscous fluid, was the most consistent with the experimental data and predicted that in vitro tumor spheroid growth is marginally stable. In related works (e.g., Chatelain et al. 2011; Amar et al. 2011), the authors used a multiphase Darcy law model (elastic fluid model) and identified conditions for morphological instabilities of melanoma lesions in the skin through a control parameter that relates cell proliferation, adhesion and the front propagation speed, finding that slower propagating tumors tend to be more unstable, in agreement with ours and earlier work (Cristini et al. 2003). See also the work by Lorenzi et al. (2017) where stability analyses of growing tumor spheroids were performed using an elastic fluid model. By Ciarletta et al. (2011), a general multiphase mixture model was developed that incorporated an incompressible hyperelastic solid model for the basal lamina (e.g. ECM) and a viscoplastic constitutive model for the interaction forces between the cells and basal lamina. Interestingly, the elastic properties of the basement membrane were not found to influence the growth patterns. Here, we extend our previous work (Pham et al. 2010) by using a Stokes fluid model to examine role of the viscosity of the host microenvironment (two-fluid model) and nonlinearity in the Stokes tumor dynamics using numerical simulations. Our model is at the tissue scale and thus the tumor and host represent mixtures of cells and extracellular matrix (ECM). The overall mixture is treated as two viscous fluids (tumor and host) with different viscosities. While we do not model the production of ECM explicitly, we account for different amounts of ECM in the tumor and host through the viscosities of the mixtures (e.g., a large viscosity is used to mimic a high ECM density). Although recent experiments show that tumor cells can be softer than normal cells (Cross et al. 2008), the overall tumor may be stiffer than the host tissue because of relative differences in ECM densities. While most tumors are found to be stiffer than the surrounding tissue (Butcher et al. 2009), which we model using a ratio of host and tumor viscosities less than one, we also consider cases in which the tumor is less stiff than the surrounding tissue. As in experiments, we find that the tumor dynamics is sensitive to these mechanical properties. We focus on 2D but we anticipate that the behavior in 3D should be similar, as predicted by linear stability analysis and as seen in results from other tumor models based on Darcy’s law (Butcher et al. 2009; Wise et al. 2008). The outline of this paper is as follows. In Sect. 2, we present the two-fluid tumor model, as well the non-dimensionalization, reformulation, and linear stability analysis. Then we present the boundary integral formulation in Sect. 3. Nonlinear simulation results, and comparisons with linear theory, are presented in Sect. 4. In Sect. 5, we summarize our findings and compare with experiments. Details on the linear stability analysis, boundary integral formulation and the numerical implementation are provided in Appendicies.

2 Mathematical model

We introduce a two-fluid flow model to study the dynamics of tumor spheroids. The tumor cell population is assumed homogeneous. We treat the tumor as an incompressible viscous fluid growing in a fully vascularized environment behaving like another incompressible viscous fluid. Both fluids are considered to be mixtures of cells and ECM with the viscosities mimicking the effect of varying densities of ECM. As by Pham et al. (2010), we employ Stokes flow as the constitutive law to model tissue stresses although here we consider viscosity of both the tumor and the host. By Pham et al. (2010), we considered only the limiting case where the viscosity of the host was negligible. Growth is regulated by an externally supplied cell substrate, e.g. oxygen, through cell proliferation and apoptosis. As by Cristini et al. (2003) and Pham et al. (2010), the tumors are assumed to be non-necrotic, cell–cell adhesion is modeled using surface tension at the tumor/host interface, and pressure due to cell proliferation acts as an expansive force.

2.1 Governing equations

Let \(\varOmega _-=\varOmega _-(t)\) be the tumor volume and \(\varSigma =\varSigma (t)\) be the interface separating the tumor from the host tissue microenvironment \(\varOmega _+=\varOmega _+(t)\). We assume that growth is regulated by cell substrates (e.g. oxygen, nutrients and growth factors), whose concentration \(\sigma (\mathbf{x },t)\) satisfies the reaction–diffusion equation:

$$\begin{aligned} \sigma _t = {\mathcal {D}} \nabla ^2\sigma + \varGamma ,&~~ \text {in} ~ \varOmega _-, \end{aligned}$$
(2.1)

where \({\mathcal {D}}\) is the diffusion coefficient and \(\varGamma \) is the rate at which cell substrates are added to \(\varOmega _-\), which accounts for all sources and sinks of substrates in the tumor volume.

These substrates are supplied by a vasculature at a rate \(\varGamma _B\) given by

$$\begin{aligned} \varGamma _B=C_B( \sigma _B-\sigma ), \end{aligned}$$
(2.2)

where \(C_B\) is the substrate transfer rate between the tissue and the blood, and \(\sigma _B\) is the substrate concentration (uniform) in the blood. The rate \(\varGamma \) is given by

$$\begin{aligned} \varGamma = -C_\sigma \sigma + \varGamma _B, \end{aligned}$$
(2.3)

where \(C_\sigma \) is the uptake rate. We assume that the tumor microenvironment environment is highly vascularized, so that the cell substrate concentration is constant and uniform in the host tissue \(\varOmega _+\):

$$\begin{aligned} \sigma = \sigma _\infty , ~~ \text {in} ~ \varOmega _+. \end{aligned}$$
(2.4)

Here \(\sigma _\infty \), the substrate concentration in the host tissue, may be smaller than \(\sigma _b\). Thus, at the tumor-host interface \(\varSigma \),

$$\begin{aligned} (\sigma )_\varSigma = \sigma _\infty . \end{aligned}$$
(2.5)

Incorporating substrate diffusion throughout the host tissue could be done (Cristini et al. 2009), but would not qualitatively change the behavior observed here. However, non-constant diffusivities could influence tumor morphological stability.

We assume constant cell density in the tumor domain, which is a good approximation away from areas of necrosis. Accordingly, mass changes correspond to volume changes. Let \(\mathbf{v}_-\) be the tumor cell velocity. Then the local rate of volume change is given by

$$\begin{aligned} \nabla \cdot \mathbf{v}_- = C_P,&~~ \text {in} ~ \varOmega _-, \end{aligned}$$
(2.6)

where \(C_P\) is the net cell-proliferation rate

$$\begin{aligned} C_P = b\sigma - C_A, \end{aligned}$$
(2.7)

where b and \(C_A\) are the rates of mitosis and apoptosis respectively, which are both assumed to be uniform. Let \(\mathbf{v}_+\) to be the velocity field of the host tissue. We assume that there is no relative mass gain or loss here, so that

$$\begin{aligned} \nabla \cdot \mathbf{v}_+ = 0,&~~ \text {in} ~ \varOmega _+. \end{aligned}$$
(2.8)

Note that in this model, we have not explicitly considered different cell compartments (e.g., quiescent, proliferating, necrotic). However, since the proliferation rate is proportional to the local substrate concentration, the cells in regions with low substrate levels will have low proliferation rates, which mimics quiescence. And, in these regions, the apoptosis rate may dominate proliferation, which mimics the effects of necrosis. Further, even though the cells have the same rate of substrate uptake, the total uptake is proportional to the cell proliferation rate.

2.2 The Stokes model

Modeling the tumor and the host tissue as highly viscous fluids, we assume

$$\begin{aligned} \nabla \cdot \mathbf{T }_\pm = 0, \end{aligned}$$
(2.9)

where \(\mathbf{T }_\pm \) are the tumor (−) and host (\(+\)) stress tensors given by

$$\begin{aligned} \mathbf{T }_\pm = \mu _\pm (\nabla \mathbf{v}_\pm + (\nabla \mathbf{v}_\pm )^T) + {\bar{\mu }}_\pm (\nabla \cdot \mathbf{v}_\pm )\mathbf{I }-p_\pm \mathbf{I }. \end{aligned}$$
(2.10)

The stress tensors take into account the local rate of strain, dilatation, and pressure. The parameters \(\mu _\pm \) and \({\bar{\mu }}_\pm \) are the shear and bulk viscosity coefficients and \(p_\pm \) are the pressures. Across the interface, the stress tensors are assumed to satisfy the Laplace–Young jump boundary condition:

$$\begin{aligned}{}[\mathbf{T }\mathbf{n}] = \mathbf{T }_+\mathbf{n}- \mathbf{T }_-\mathbf{n}= \tau \kappa \mathbf{n}, ~~ \text {on} ~ \varSigma , \end{aligned}$$
(2.11)

where \(\tau \) is the surface tension, which is used to model cell–cell adhesion and \(\kappa \) is the total local curvature. We also assume that velocity field is continuous across the interface:

$$\begin{aligned}{}[\mathbf{v}] = \mathbf{v}_+ - \mathbf{v}_- = 0. \end{aligned}$$
(2.12)

The tumor interface evolves according to the normal velocity V:

$$\begin{aligned} \frac{d\mathbf{x}}{dt}\cdot \mathbf{n}= V, \end{aligned}$$
(2.13)

where

$$\begin{aligned} V=\mathbf{v}_+\cdot \mathbf{n}=\mathbf{v}_-\cdot \mathbf{n}, ~~ \text {on} ~ \varSigma . \end{aligned}$$
(2.14)

Non-dimensionalization. We rescale the dimensional variables by their characteristic values and define the following non-dimensional parameters

(2.15)

where

$$\begin{aligned} L = {\mathcal {D}}^{1/2}(C_B+C_\sigma )^{-1/2},~~C_M = \left( b\sigma _\infty \right) ^{-1},~~{{{\bar{P}}}}_{\pm } = \mu _\pm C_M,~~ B = \frac{\sigma _B}{\sigma _\infty }\frac{C_B}{C_B + C_\sigma }.\nonumber \\ \end{aligned}$$
(2.16)

Here L is the nutrient penetration length (Cristini et al. 2003), \(C_M\) is the mitosis rate, \({{{\bar{P}}}}_{\pm }\) is the pressure scale, and B is a dimensionless parameter that measures the relative rates of blood tissue transfer in the tumor and substrate uptake (Cristini et al. 2003) and the ratio of the substrates concentration in the blood and tissue.

We assume \(C_M L^2/{\mathcal {D}}\ll 1\), i.e. nutrient diffusion is much faster than mitosis. This assumption allows an approximation of the dimensional reaction–diffusion equation (2.1) by a quasi-steady equation given in nondimensional form below in Eq. (2.20) by dropping the time derivative. Finally, we define the following non-dimensional parameters,

$$\begin{aligned} {\mathcal {A}}=\frac{C_A}{C_M} - B,~~\nu _-=\frac{{\bar{\mu }}_-}{\mu _-}, ~~ {{\mathcal {G}}}= \frac{\tau }{\mu _- C_M L}, ~~ \lambda =\frac{\mu _+}{\mu _-}, \end{aligned}$$
(2.17)

where \({{\mathcal {A}}}\) is the dimensionless relative rate of cell apoptosis to mitosis in the absence of tumor vasculature (\(B=0\)), \(\nu _-\) is the ratio between the two interior viscosity coefficients, \({{\mathcal {G}}}\) is the relative strength of cell–cell adhesion, and \(\lambda \) is the viscosity ratio of the host microenvironment to the tumor.

Dropping the bars and primes, we obtain the non-dimensional Stokes system is:

$$\begin{aligned} \nabla \cdot \mathbf{v}_-= & {} (1-B)\sigma -{\mathcal {A}}, ~~ \text {in} ~ \varOmega _-, \end{aligned}$$
(2.18)
$$\begin{aligned} \nabla \cdot \mathbf{T }_-= & {} 0, ~~ \text {in} ~ \varOmega _-, \end{aligned}$$
(2.19)
$$\begin{aligned} \varDelta \sigma= & {} \sigma , ~~ \text {in} ~ \varOmega _-, \end{aligned}$$
(2.20)

and

$$\begin{aligned} \nabla \cdot \mathbf{v}_+= & {} 0, ~~ \text {in} ~ \varOmega _+, \end{aligned}$$
(2.21)
$$\begin{aligned} \nabla \cdot \mathbf{T }_+= & {} 0, ~~ \text {in} ~ \varOmega _+, \end{aligned}$$
(2.22)
$$\begin{aligned} \sigma= & {} 1, ~~ \text {in} ~ \varOmega _+, \end{aligned}$$
(2.23)

where

$$\begin{aligned} \mathbf{T }_-= & {} \nabla \mathbf{{v}}_ - + (\nabla \mathbf{{v}}_ - )^T - p_ - \mathbf{{I}} + \nu _ - \left( {(1 - B)\sigma - {{{\mathcal {A}}}}} \right) \mathbf{{I}},\end{aligned}$$
(2.24)
$$\begin{aligned} \mathbf{T }_+= & {} \lambda (\nabla \mathbf{v}_+ + (\nabla \mathbf{v}_+)^T) - p_+ \mathbf{I }. \end{aligned}$$
(2.25)

On the tumor interface \(\varSigma \), we obtain the nutrient and velocity continuity conditions:

$$\begin{aligned} \sigma = 1,&~~ \text {on} ~ \varSigma . \end{aligned}$$
(2.26)
$$\begin{aligned} \mathbf{v}_+ - \mathbf{v}_- = 0,&~~\text {on} ~ \varSigma . \end{aligned}$$
(2.27)

The Laplace–Young jump boundary condition in Eq. (2.11) becomes

$$\begin{aligned} \mathbf{T }_+\mathbf{n } - \mathbf{T }_-\mathbf{n } ={{\mathcal {G}}}\kappa \mathbf{n }, ~~ \text {on} ~ \varSigma . \end{aligned}$$
(2.28)

The non-dimensional Stokes system shows that the evolution is governed by the dimensionless parameters \(\nu _-\), \(\lambda \), \({{\mathcal {A}}}\), \({{\mathcal {G}}}\) and B.

2.3 Reformulation

We next present a reformulation of the system to a new Stokes model in which the velocities are divergence free. This is helpful in the design of numerical methods. We first redefine the tumor cell velocity as

$$\begin{aligned} \mathbf{u}_- = \mathbf{v}_- - (1-B)\nabla \sigma + \frac{{{\mathcal {A}}}\mathbf{x}}{d}, \end{aligned}$$
(2.29)

where \(d=2\) is the spatial dimension. Then Eq. (2.18) reduces to

$$\begin{aligned} \nabla \cdot \mathbf{u}_- = 0. \end{aligned}$$
(2.30)

Recall the interior Stokes law in Eq. (2.19):

$$\begin{aligned} \nabla \cdot \mathbf{T}_- = \varDelta \mathbf{v}_- - \nabla (p_- - (1+\nu _-)\nabla \cdot \mathbf{v}_-) = 0. \end{aligned}$$
(2.31)

Redefining the interior pressure as \({\tilde{p}}_- = p_- - (\nu _-+1)\nabla \cdot \mathbf{v}_- - (1-B)\sigma \) and using the redefined velocity \(\mathbf{u}_-\), Stokes law reduces to

$$\begin{aligned} \varDelta \mathbf{u}_- = \nabla {\tilde{p}}_-, \end{aligned}$$
(2.32)

and the stress tensor \(\mathbf{T}_-\) becomes

$$\begin{aligned} \mathbf{T}_- = \mathbf{T}^u_- + 2(1-B)\nabla \nabla \sigma - 2(1-B)\sigma \mathbf{I } - \frac{{{\mathcal {A}}}}{d}(2-d)\mathbf{I }, \end{aligned}$$
(2.33)

where \(\mathbf{T}^u_- = \nabla \mathbf{u}_- + (\nabla \mathbf{u}_-)^T - {\tilde{p}}_-\mathbf{I }\).

Table 1 Summary of the main parameters used in the two-fluid model

Plugging the redefined variables back into Eq. (2.28), we obtain a new jump boundary condition for the stress field:

$$\begin{aligned} \mathbf{T}_+\mathbf{n}- \mathbf{T}^u_- \mathbf{n}= {{\mathcal {G}}}\kappa \mathbf{n}+ 2(1-B)\nabla \nabla \sigma \mathbf{n}- 2(1-B)\sigma \mathbf{n}- \frac{{{\mathcal {A}}}}{d}(2-d) \mathbf{n}. \end{aligned}$$
(2.34)

The reformulation requires the evaluation of \(\nabla \nabla \sigma \mathbf{n}\), which can be expressed in terms of the normal derivative of \(\sigma \) as

$$\begin{aligned} \mathbf{s}\cdot (\nabla \nabla \sigma (s)\mathbf{n})= & {} \frac{d}{ds}(\mathbf{n}\cdot \nabla \sigma (s)), \end{aligned}$$
(2.35)
$$\begin{aligned} \mathbf{n}\cdot (\nabla \nabla \sigma (s)\mathbf{n})= & {} 1-\kappa \mathbf{n}\cdot \nabla \sigma (s), \end{aligned}$$
(2.36)

where s is the arclength representation of the tumor-host interface. Since the exterior velocity field is already divergence-free, it is unnecessary to reformulate the exterior problem. The boundary condition for nutrient remains the same as in Eq. (2.26). However, in view of Eq. (2.29), the reformulated velocity field becomes discontinuous:

$$\begin{aligned} \mathbf{v}_+(\mathbf{x}_\varSigma ) = \mathbf{u}_-(\mathbf{x}_\varSigma ) + (1-B)\nabla \sigma |_\varSigma -\frac{{{\mathcal {A}}}\mathbf{x}_\varSigma }{2}, \end{aligned}$$
(2.37)

Note that the reformulation removes \(\nu _-\) from the model, showing the evolution is independent of this parameter. The nondimensional parameters and their values are given in Table 1.

2.4 Linear stability analysis

We begin our analysis of the tumor model by considering the behavior in the linear regime. We perturb a circular (2D) tumor interface \(\varSigma \) of radius R(t) as follows:

$$\begin{aligned} r(t,\theta )=R(t) + \delta (t) \cos l\theta , \end{aligned}$$
(2.38)

where \(\delta \) is the perturbation size, l and \(\theta \) are polar wavenumber and angle, respectively. In Appendix 1, we provide details of the analysis. Here we summarize the main results. The evolution equation for the unperturbed tumor radius is

$$\begin{aligned} \frac{dR}{dt} = (1-B)\frac{I_1(R)}{I_0(R)} - \frac{{\mathcal {A}}R}{2}, \end{aligned}$$
(2.39)

where \(I_0(R)\) and \(I_1(R)\) are the modified Bessel functions of the first kind with indices 0 and 1 respectively. In the insert of Fig. 1, we plot the stationary radius in terms of \({{\mathcal {A}}}\) when \(dR/dt=0\). This equation is the same for the Darcy model (Cristini et al. 2003), and thus is unaffected by the tumor and host viscosities. The tumor velocity \(\displaystyle {\frac{dR}{dt}}\), with the vascularization parameter \(B=0\), is shown in Fig. 1 for different choices of \({{\mathcal {A}}}\). Observe that when \({{\mathcal {A}}}=0\), the tumor grows without bound (no apoptosis). When \({{\mathcal {A}}}>0\), the tumor evolves to a steady state with \(dR/dt=0\). The shape perturbations evolve according to

$$\begin{aligned} \bigg (\frac{\delta }{R}\bigg )\frac{d\big (\frac{\delta }{R}\big )}{dt} = \frac{1}{\lambda +1}\left( -\frac{{{\mathcal {G}}}l}{2 R} + \lambda {{\mathcal {A}}}+(1-B) \bigg (1-\frac{I_{l+1}(R)}{I_l(R)} - \frac{2}{R}(\lambda +1)\bigg )\frac{I_1(R)}{I_0(R)} \right) ,\nonumber \\ \end{aligned}$$
(2.40)

which depends on the parameters \({{\mathcal {A}}}\), \({{\mathcal {G}}}\), B and \(\lambda \). Observe that the right-hand side of the equation increases with increasing \({{\mathcal {A}}}\) (high cell death) and decreases with increasing \({{\mathcal {G}}}\) (high cell adhesion), implying that \({{\mathcal {A}}}\) promotes shape instability while \({{\mathcal {G}}}\) promotes stable morphologies, consistent with previous work (Byrne and Chaplain 1996, 1997; Cristini et al. 2003; Pham et al. 2010). The parameters B and \(\lambda \) may promote or reduce instability, depending on the values of l and the radius of the tumor R. Thus, the morphology is determined by the competition between cell death, cell–cell adhesion, the tumor and host viscosity ratio and vascularization. The ratio \(\delta /R\) is also known as the shape factor as its magnitude measures the deviation of the tumor shape from a circle of varying radius.

Fig. 1
figure 1

The velocity of a circular (2D) tumor, \(\displaystyle {\frac{dR}{dt}}\) from Eq. (2.39), as a function of tumor radius R; \({{\mathcal {A}}}\) as labeled. The vascularization parameter \(B=0\) (color figure online)

A marginally stable (or critical) value of the adhesion parameter \({{\mathcal {G}}}_M(l,{{\mathcal {A}}},R,\lambda )\) is obtained by setting the time derivative of \(\delta /R\) in Eq. (2.40) to zero and thus separates stable (\({{\mathcal {G}}}>{{\mathcal {G}}}_M\)) from unstable growth (\({{\mathcal {G}}}<{{\mathcal {G}}}_M\)). Recall that \({{\mathcal {G}}}\) is proportional to cell–cell adhesion. By plugging Eq. (2.39) into Eq. (2.40), we find

$$\begin{aligned} {\mathcal {G}} _M = - \frac{{4\lambda }}{l}\left( {\frac{{dR}}{{dt}}} \right) + (1 - B)\frac{2R}{l}\left( {1 - \frac{{I_{l + 1} (R)}}{{I_l (R)}}\frac{{I_1 (R)}}{{I_0 (R)}} - \frac{2}{R}\frac{{I_1 (R)}}{{I_0 (R)}}} \right) , \end{aligned}$$
(2.41)

which means that the viscosity ratio \(\lambda =\mu _+/\mu _-\) influences the tumor stability through the rate of tumor growth dR / dt. When \(dR/dt>0\), \({{\mathcal {G}}}_M \) increases as the host is less viscous (\(\lambda \) decreases). That is, the growing tumor is less stable (and hence more invasive) as its viscosity increases (and/or the viscosity of the host decreases). When \(dR/dt<0\), \({{\mathcal {G}}}_M \) increases as the host is more viscous (\(\lambda \) increases). This is different from the classical Saffman–Taylor problem (Saffman and Taylor 1958) where instability occurs when a less viscous fluid displaces a more viscous fluid but not the other way around. A detailed comparison with the Saffman–Taylor problem shows that the difference is exactly due to the presence of a bulk source/sink term in the tumor model that effectively switches the roles of the low and high viscosity fluids. Interestingly, however, we find that in our numerical simulations that the largest values of the shape perturbations of the tumor-host interface (shape factor \(\delta /R\)) occur at the locations where the lower viscosity host penetrates the higher viscosity tumor and not at the fingers of the tumor that penetrate the host, which are much broader (e.g., see the blue dots in Fig. 6). Finally, observe from Eq. (2.41) that for fixed R, \({\mathcal {G}}_M\) tends to zero as l increases at a rate proportional to 1 / l (the rate depends on R). This indicates that the high modes are stabilized by a non-zero cell adhesion.

We consider the case with \(B=0\) (no tumor vasculature) and study the stability of tumor growth in the avascular stage. When \({{\mathcal {A}}}=0\), the circular tumor grows without bound. In the limit of large R and fixed l, the marginally stable value is given by

$$\begin{aligned} {{\mathcal {G}}}_M (l,{{\mathcal {A}}}=0,R,\lambda )=\frac{2}{l}\left( l+1-2\left( \lambda +1\right) \right) +O(R^{-1}) ~~~\mathrm {as}~~R\rightarrow \infty . \end{aligned}$$
(2.42)

Therefore, for each mode l, there is a critical viscosity ratio below which instability can occur. In particular, all modes with

$$\begin{aligned} l<2\lambda +1, \end{aligned}$$
(2.43)

are stable. Decreasing the viscosity ratio \(\lambda \), which corresponds to increasing the viscosity of the tumor relative to the host, enhances instability. This is in contrast to the Darcy model where instability requires \({{\mathcal {A}}}>0\) (Cristini et al. 2003). In the Stokes model, taking \(l=3\) for example, then instability occurs only if \(\lambda <1\). This behavior is summarized in Fig. 2a, b where in (a) the marginally stable values of \({{\mathcal {G}}}_M\) are shown with \(l=3\) for different viscosity ratios \(\lambda \). In Fig. 2b, \({{\mathcal {G}}}_M\) is shown for the fixed viscosity ratio \(\lambda =0.5\) for different modes l. Note the non-monotonic behavior of \({{\mathcal {G}}}_M\) as a function of R and l for small R and that adhesion stabilizes sufficiently large modes (e.g., compare modes \(l=5\), \(l=7\) and \(l=50\)).

Fig. 2
figure 2

The marginally stable value of the adhesion parameter \({{\mathcal {G}}}_M\) as a function of unperturbed radius R with \({{\mathcal {A}}}=0\). The evolution is stable if \({{\mathcal {G}}}>{{\mathcal {G}}}_M\) (high adhesion) and unstable if \({{\mathcal {G}}}<{{\mathcal {G}}}_M\) (low adhesion). In (a), results for a fixed mode \(l=3\) and different viscosity ratios (as labeled) are shown. In (b, results) for different modes (as labeled) and fixed viscosity ratio \(\lambda =0.5\) are shown. In both cases, the insets show close-ups (color figure online)

Another interesting case to consider is the behavior of perturbations about a steady-state tumor radius. Taking \({{\mathcal {A}}}={{\mathcal {A}}}_s=2\frac{I_1(R)}{R I_0(R)}\) for each R, we have \(dR/dt=0\) so that each R is a steady-state radius. In this case, for fixed l we get

$$\begin{aligned} {{\mathcal {G}}}_M (l,{{\mathcal {A}}}_s,R,\lambda )= & {} \frac{2R}{l}\left( 1-\frac{I_{l+1}(R)}{I_l(R)}-\frac{2}{R}\frac{I_1(R)}{I_0(R)}\right) \end{aligned}$$
(2.44)
$$\begin{aligned}= & {} 2\left( 1-\frac{1}{l}\right) + O(R^{-1})~~~\mathrm {as}~~R\rightarrow \infty , \end{aligned}$$
(2.45)

which strikingly is independent of the viscosity ratio. Figure 3 shows the corresponding \({{\mathcal {G}}}_M\) as a function of the unperturbed radius for different modes l. Note the non-monotonic dependence of \({{\mathcal {G}}}_M\) upon the wavenumber l.

Fig. 3
figure 3

The marginally stable values of the adhesion parameter \({{\mathcal {G}}}_M\) with \({{\mathcal {A}}}={{\mathcal {A}}}_s(R)\), the steady state value at each unperturbed radius R (see text), as a function of R for different modes (as labeled). The results are independent of viscosity ratio \(\lambda \) (see text). The inset shows a close-up of the behavior at small R (color figure online)

For arbitrary \({{\mathcal {A}}}\), recall Eq. (2.41), which implies that for fixed l we obtain

$$\begin{aligned} {{\mathcal {G}}}_M= & {} -\frac{4\lambda }{l}\frac{dR}{dt} +2\left( 1-\frac{1}{l}\right) +O(R^{-1})~~~\mathrm {as}~~R\rightarrow \infty . \end{aligned}$$
(2.46)

Note that \(\displaystyle {\frac{dR}{dt}=1-\frac{AR}{2} + O(R^{-1})}\) as \(R\rightarrow \infty \). Assuming \(dR/dt>0\), so that the tumor is growing, we observe that instability may only occur if

$$\begin{aligned} \lambda <\frac{1}{2}\left( l-1\right) \left( \frac{dR}{dt}\right) ^{-1}. \end{aligned}$$
(2.47)

Therefore, while the tumor is growing, instability is enhanced if \(\lambda \) is decreased as in the case \({{\mathcal {A}}}=0\). On the other hand if \(dR/dt<0\) so that the tumor is shrinking, then the evolution is always unstable since \(\displaystyle {\lambda \ge 0>-\left| \frac{dR}{dt}\right| ^{-1}\frac{1}{2}\left( l-1\right) }\) and instability is enhanced if \(\lambda \) is increased (assuming \(l>1\)). In Fig. 4, we illustrate this behavior by showing \({{\mathcal {G}}}_M\) with \({{\mathcal {A}}}=0.5\) for mode \(l=3\) and different viscosity ratios \(\lambda \). When \({{\mathcal {A}}}=0.5\), the steady state radius is \(R_s\approx 3.3255\), which corresponds to the point at which all the curves intersect that reflects the \(\lambda \)-independent behavior at the steady-state radius.

Fig. 4
figure 4

a The marginally stable value of the adhesion parameter \({{\mathcal {G}}}_M\) as a function of unperturbed radius R for different viscosity ratios (as labeled) with \({{\mathcal {A}}}=0.5\) and mode \(l=3\). b Close-up around the point at which the curves cross one another, which corresponds to the steady-state radius \(R_s=3.3255\) where stability is independent of the viscosity ratio \(\lambda \). The four points \(P_1(1.988,0.15)\), \(P_2(1.988,0.6)\), \(P_3(4.5,0.15)\), \(P_4(4.5,0.6)\) indicate parameter values at which nonlinear simulations will be performed (see Fig. 7) (color figure online)

3 Numerical methods

In order to investigate the nonlinear dynamics of the Stokes tumor model we use highly accurate and efficient numerical methods. We represent the tumor/host interface \(\varSigma \equiv \partial \varOmega (t)\) as a planar curve parametrized by \(\alpha \): \(\mathbf{x}(\alpha ,t)\equiv (x(\alpha ,t),y(\alpha ,t))\) with \(\alpha \in [0,2\pi ]\). We use boundary integral methods to solve the Stokes and diffusion equations for the cell velocity and nutrient concentration, respectively. In this approach, these systems are reformulated in terms of single and double layer potentials that are determined only by equations on the tumor/host interface and not in the full 2D domain. Spectrally accurate spatial discretizations are used to approximate the boundary integrals and derivatives are evaluated using the Fast Fourier Transform. The tumor/host interface is updated in time using a non-stiff time integration method (Hou et al. 1994; Leo et al. 2000). The overall method is similar to that used by Cristini et al. (2003) with several new features. To improve the accuracy of the algorithm, we use a spatial rescaling of the system to keep the area enclosed by the tumor constant in time in the rescaled frame. The boundary integral equations and numerical algorithms are described in detail in Appendicies 24. Here, we present the spatial rescaling and the rescaled boundary integral equations for Stokes flow as they are new.

3.1 Space rescaling

We introduce the following spatial scaling

$$\begin{aligned} \mathbf{x}= S(t){\tilde{\mathbf{x}}}(t,\alpha ), \end{aligned}$$
(3.1)

where \({\tilde{\mathbf{x}}}(t,\alpha )\) is the position vector of the scaled tumor/host interface. The scaling factor S(t) is chosen such that the area enclosed by the scaled interface is constant, that is

$$\begin{aligned} \frac{d|{\tilde{\varOmega }}_-|}{dt} = \int _{{\tilde{\varSigma }}}{\tilde{V}}d{\tilde{s}} = 0. \end{aligned}$$
(3.2)

The normal velocity in the scaled frame \(\left( \displaystyle {{{\tilde{V}}}= \frac{d{\tilde{\mathbf{x}}}}{dt}\cdot \mathbf{n}}\right) \) and the original frame \(\left( \displaystyle {V= \frac{d{\mathbf{x}}}{dt}\cdot \mathbf{n}}\right) \) are related by:

$$\begin{aligned} {\tilde{V}} = \frac{d{\tilde{\mathbf{x}}}}{dt}\cdot \mathbf{n}= \frac{V}{S} - \frac{{\dot{S}}}{S}{\tilde{\mathbf{x}}}\cdot \mathbf{n}, \end{aligned}$$
(3.3)

where \(\displaystyle {\dot{S}}=\frac{dS}{dt}\) and \(\mathbf{n}\) is the normal vector, which is the same in both the scaled and original domains. The dynamical equation for the scaling factor is

$$\begin{aligned} 2{\dot{S}}(t)=\frac{1-B}{S(t)|{\tilde{\varOmega }}|} \int _{{\tilde{\varSigma }}}\frac{\partial {\tilde{\sigma }}}{\partial {\tilde{\mathbf{n}}}}d{\tilde{\varSigma }} - {{\mathcal {A}}}S(t), \end{aligned}$$
(3.4)

where \({\tilde{\sigma }}({\tilde{\mathbf{x}}})=\sigma (\mathbf{x}({\tilde{\mathbf{x}}}))\) is the nutrient concentration in the scaled frame.

Letting \(\nabla =\frac{1}{S}{{\tilde{\nabla }}}\), \({\tilde{p}}_{\pm }({\tilde{\mathbf{x}}})=p_{\pm }(\mathbf{x}({\tilde{\mathbf{x}}}))\), and defining the velocities in the rescaled frame as

$$\begin{aligned} {\tilde{\mathbf{w}}}_-= & {} \frac{1}{S}{\mathbf{v}}_-(\mathbf{x}({{\tilde{\mathbf{x}}}})) -(1-B)\frac{1}{S^2}{\tilde{\nabla }}{\tilde{\sigma }} -{{\mathcal {A}}}\frac{{\tilde{\mathbf{x}}}}{d},\\ {\tilde{\mathbf{w}}}_+= & {} {\mathbf{v}}_+(\mathbf{x}({{\tilde{\mathbf{x}}}})), \end{aligned}$$

where \(d=2\) is the spatial dimension, the rescaled equations for the tumor and host become:

$$\begin{aligned} {\tilde{\nabla }}\cdot {\tilde{\mathbf{w}}}_-= & {} 0, ~~\text {in} ~ {\tilde{\varOmega }}_-, \end{aligned}$$
(3.5)
$$\begin{aligned} {\tilde{\varDelta }}{\tilde{\mathbf{w}}}_-= & {} {\tilde{\nabla }}{{\tilde{p}}}_-, ~~\text {in} ~ {\tilde{\varOmega }}_-, \end{aligned}$$
(3.6)
$$\begin{aligned} {\tilde{\varDelta }}{\tilde{\sigma }}= & {} S^2{\tilde{\sigma }}, ~~\text {in} ~ {\tilde{\varOmega }}_-, \end{aligned}$$
(3.7)

and

$$\begin{aligned} {\tilde{\nabla }}\cdot {\tilde{\mathbf{w}}}_+= & {} 0, ~~\text {in} ~ {\tilde{\varOmega }}_+, \end{aligned}$$
(3.8)
$$\begin{aligned} {\tilde{\varDelta }}{\tilde{\mathbf{w}}}_+= & {} {\tilde{\nabla }}{\tilde{p}}_+, ~~\text {in} ~ {\tilde{\varOmega }}_+, \end{aligned}$$
(3.9)
$$\begin{aligned} {\tilde{\sigma }}= & {} 1, ~~\text {in} ~ {\tilde{\varOmega }}_+. \end{aligned}$$
(3.10)

At the tumor/host interface \(\varSigma \), we have

$$\begin{aligned} {\tilde{\sigma }}&= 1, ~~\text {on} ~ {\tilde{\varSigma }}, \end{aligned}$$
(3.11)
$$\begin{aligned} {\tilde{\mathbf{w}}}_+-{\tilde{\mathbf{w}}}_-&= (1-B)\frac{1}{S^2}{\tilde{\nabla }}{\tilde{\sigma }}|_{{\tilde{\varSigma }}} - \frac{{{\mathcal {A}}}{\tilde{\mathbf{x}}}_{{\tilde{\varSigma }}}}{2}, ~~\text {on} ~ {\tilde{\varSigma }}, \end{aligned}$$
(3.12)
$$\begin{aligned} \lambda {\tilde{\mathbf{T}}}^w_+\mathbf{n}- {\tilde{\mathbf{T}}}^w_-\mathbf{n}&= {{\mathcal {G}}}\frac{{\tilde{\kappa }}}{S}\mathbf{n}+ 2(1-B)\frac{1}{S^2}{\tilde{\nabla }}{\tilde{\nabla }}{\tilde{\sigma }}\mathbf{n}- 2(1-B){\tilde{\sigma }}\mathbf{n}- \frac{{{\mathcal {A}}}}{d}(2-d)\mathbf{n}, ~~\text {on} ~ {\tilde{\varSigma }}, \end{aligned}$$
(3.13)

where \({\tilde{\mathbf{T}}}^w_+ = {\tilde{\nabla }}{\tilde{\mathbf{w}}}_+ + {\tilde{\nabla }}{\tilde{\mathbf{w}}}_+^T - {\tilde{p}}_+\mathbf{I }\) and \({\tilde{\mathbf{T}}}^w_- = {\tilde{\nabla }}{\tilde{\mathbf{w}}}_- + {\tilde{\nabla }}{\tilde{\mathbf{w}}}_-^T - {{\tilde{p}}}_-\mathbf{I }\) are rescaled stress tensors.

The evolution of the interface becomes

$$\begin{aligned} \frac{d{\tilde{\mathbf{x}}}}{dt} ={\tilde{\mathbf{w}}}_+ - \frac{{\dot{S}}}{S(t)}{\tilde{\mathbf{x}}} = {\tilde{\mathbf{w}}}_- + (1-B)\frac{1}{S^2}{\tilde{\nabla }}{\tilde{\sigma }}-({\dot{S}}/S + {{\mathcal {A}}}/d){\tilde{\mathbf{x}}}. \end{aligned}$$
(3.14)

In the rescaled frame we obtain a new boundary integral formulation for the tumor model, which is given below.

3.2 Rescaled boundary integral equations for Stokes flow

Let \(\mathbf{G }\) be the Stokeslet and \(\mathbf{T}\) be the tensor Stresslet (see Appendix 3 for their definitions), then the single layer potential at the interface \(\varSigma \) is

$$\begin{aligned} \mathbf{S }[\mathbf{f }]({{\tilde{\mathbf{x}}}}_\varSigma ) = \frac{1}{4\pi }\int _\varSigma \mathbf{G }({{\tilde{\mathbf{x}}}}_\varSigma ' -{{\tilde{\mathbf{x}}}}_{{{\tilde{\varSigma }}}})\mathbf{f }({{\tilde{\mathbf{x}}}}')d{{\tilde{s}}}', \end{aligned}$$
(3.15)

where \({{\tilde{\mathbf{x}}}}'_\varSigma ={\tilde{\mathbf{x}}}_\varSigma ({\tilde{s}}')\)and the double layer potential at \({{\tilde{\varSigma }}}\) is

$$\begin{aligned} \bar{\mathbf{D }}[\mathbf{u }]({{{\tilde{\mathbf{x}}}}}_{{{\tilde{\varSigma }}}}) = \frac{1}{4\pi } P.V.\int _{{{\tilde{\varSigma }}}}\mathbf{u}({{\tilde{\mathbf{x}}}}') \mathbf{T }({{\tilde{\mathbf{x}}}}_\varSigma ' -{{\tilde{\mathbf{x}}}}_{{{\tilde{\varSigma }}}}) \mathbf{n}({{\tilde{\mathbf{x}}}}')d{{\tilde{s}}}', \end{aligned}$$
(3.16)

where P.V. denotes the principle value integral. The boundary integral equation for the rescaled Stokes tumor model is

$$\begin{aligned} {\tilde{\mathbf{w}}}_+({\tilde{\mathbf{x}}}_{{\tilde{\varSigma }}}) - 2\frac{\lambda -1}{\lambda +1}\bar{\mathbf{D }}[{\tilde{\mathbf{w}}}_+] ({\tilde{\mathbf{x}}}_{{\tilde{\varSigma }}})=\frac{1}{\lambda +1}\tilde{\mathbf{F }}, \end{aligned}$$
(3.17)

where

$$\begin{aligned} \tilde{\mathbf{F }}= & {} -2{\mathbf{S }}\left[ \lambda {\tilde{\mathbf{T}}}^w_+ - {\tilde{\mathbf{T}}}^w_-\right] (\mathbf{x}_\varSigma ) +2\bar{\mathbf{D }}\left[ (1-B)\frac{1}{S^2}{\tilde{\nabla }}{\tilde{\sigma }} -\frac{{{\mathcal {A}}}{\tilde{\mathbf{x}}}}{2}\right] ({\tilde{\mathbf{x}}}_{{\tilde{\varSigma }}}) + \left. (1-B)\frac{1}{S^2}{\tilde{\nabla }}{\tilde{\sigma }}\right| _{{\tilde{\varSigma }}}\nonumber \\&-\frac{{{\mathcal {A}}}{\tilde{\mathbf{x}}}_{{\tilde{\varSigma }}}}{2}. \end{aligned}$$
(3.18)

Note that the single layer term on the right hand side of Eq. (3.18) is evaluated using Eq. (3.13), which requires the calculation of \({\tilde{\nabla }}{\tilde{\nabla }}{\tilde{\sigma }}\mathbf{n}\). Componentwise, this term can be expressed as

$$\begin{aligned} \mathbf{s}\cdot ({\tilde{\nabla }}{\tilde{\nabla }}{\tilde{\sigma }} \cdot \mathbf{n})= & {} \frac{d}{d{\tilde{s}}}(\mathbf{n}\cdot {\tilde{\nabla }}{\tilde{\sigma }}),\end{aligned}$$
(3.19)
$$\begin{aligned} \mathbf{n}\cdot ({\tilde{\nabla }}{\tilde{\nabla }}{\tilde{\sigma }} \cdot \mathbf{n})= & {} S^2 -{\tilde{\kappa }}\mathbf{n}\cdot {\tilde{\nabla }}{\tilde{\sigma }}. \end{aligned}$$
(3.20)

Assuming that \({{\tilde{\sigma }}}\) is given by a double layer potential (see Appendicies 2 and 4 ), the normal derivative can be written as (Colton and Kress 1992):

$$\begin{aligned} \mathbf{n}\cdot {\tilde{\nabla }}{\tilde{\sigma }}({\tilde{s}}) = \frac{d}{d{\tilde{s}}}\tilde{{\mathcal {S}}}({\tilde{\nu }}_{{\tilde{s}}}/S) - S \mathbf{n}\cdot \tilde{{\mathcal {S}}}(\mathbf{n}{\tilde{\nu }}), \end{aligned}$$
(3.21)

where \(\tilde{{\mathcal {S}}}\) is the rescaled single layer potential for the modified Helmholtz equation and is given by

$$\begin{aligned} \tilde{{\mathcal {S}}}(\mathbf{P })\equiv -\frac{S}{2\pi }\int _\varSigma \mathbf{P }({\tilde{s}}') K_0(S{\tilde{r}})d{\tilde{s}}', \end{aligned}$$
(3.22)

for any vector (or scalar) \(\mathbf{P }\), and \(K_0\) is the modified Bessel function of the first kind.

The discretization of the equations, the solution of the integral equations for the nutrient and cell velocities, and interface evolution algorithms are presented in Appendix 4.

4 Results

We next investigate the nonlinear dynamics of the early stages of tumor progression by focusing on the case of avascular tumors \(B=0\). We elucidate the effect of the three biophysical parameters \({{\mathcal {G}}}\) (cell adhesion), \({{\mathcal {A}}}\) (cell death/apoptosis), and \(\lambda \) (host tissue viscosity) on tumor evolution. We assess linear predictions of instability against nonlinear simulations by comparing the corresponding linear and nonlinear shape factors. The shape factor from linear theory is calculated by solving the system of differential equations (2.39) and (2.40). The nonlinear shape factor is calculated using

$$\begin{aligned} \delta /R = \max _j\bigg ( |\mathbf{x}_j/R|^2 - 1 \bigg )^{1/2} ~~~~~ j=1,...,N, \end{aligned}$$
(4.1)

where \(\mathbf{x}_j\) denote the discrete points that describe the tumor/host interface and R denotes the effective radius of the tumor, which is the radius of a circle with the same area as the tumor. Although the simulations are performed in the scaled frame, the results are presented in the original variables.

4.1 Unbounded growth (\({{\mathcal {A}}}=0\), no cell death/apoptosis)

We consider the evolution of a nearly circular tumor of radius \(R(0)=1.988\) perturbed by a cosine function with initial size \(\delta (0)=0.05\), mode \(l=3\):

$$\begin{aligned} (x(\alpha ,0),y(\alpha ,0)) = (1.988+{0.05}\cos (3\alpha ))(\cos \alpha ,\sin \alpha ). \end{aligned}$$
(4.2)

In the numerical scheme, the initial scale factor is taken to be \(S(0)=1.988\). We fix the cell–cell adhesion parameter \({{\mathcal {G}}}=0.15\) and consider two viscosity ratios: \(\lambda =0.05\) where the tumor is more viscous than the host and \(\lambda =2\) where the tumor is less viscous than the host. As shown earlier, linear stability analysis predicts the \(l=3\) wavenumber is only unstable when \(\lambda <1\). Further, for when \(\lambda =0.05\), \(l=3\) is the maximally unstable mode at \(R(0)=1.988\). In Fig. 5a, the morphologies of the nonlinearly growing tumors are shown with \(\lambda =0.05\) demonstrating unstable growth while the tumor with \(\lambda =2\) is growing stably. These results are consistent with linear theory (Fig. 5b), although when \(\lambda =0.05\) linear theory tends to underpredict the shape instability. This is because nonlinearity generates new modes that are also unstable according to linear theory (see also Supplementary Materials, which shows how the maximally unstable mode depends on tumor size).

Fig. 5
figure 5

Nonlinear tumor evolution with \({{\mathcal {A}}}=0\) and different viscosity ratios. a Larger tumor viscosity: \(\lambda =0.05\) (top row) and smaller tumor viscosity: \(\lambda =2\) (bottom row). b Comparison of the linear (dashed) and nonlinear results (solid) for the shape factor (\(\delta /R\)) as a function of time. The inset shows a close-up. The initial condition is a 3-mode perturbation of a circle given in Eq. (4.2). The adhesion parameter \({{\mathcal {G}}}=0.15\)

4.2 Growth and shrinkage (\({{\mathcal {A}}}=0.5\))

We next consider the dynamics of tumors that may grow or shrink depending upon their initial radius. We take \({{\mathcal {A}}}=0.5\) and consider two cell–cell adhesion parameters \({{\mathcal {G}}}=0.15\) and \({{\mathcal {G}}}=0.6\), two initial tumor radii \(R(0)=1.988\) and \(R(0)=4.5\), and we vary the viscosity ratio \(\lambda \). In particular, we consider evolution from the points \(P_1(1.988,0.15)\), \(P_2(1.988,0.6)\), \(P_3(4.5,0.15)\), \(P_4(4.5,0.6)\) in Fig. 4b where the first coordinate represents the initial tumor radius R(0) and the second represents the cell–cell adhesion parameter \({{\mathcal {G}}}\). When \({{\mathcal {A}}}=0.5\), theory predicts that circular tumors evolve to the stationary radius \(R_s\approx 3.3255\), while the stability of the evolution depends on \({{\mathcal {G}}}\). In particular, as seen from Fig. 4b, linear theory predicts that starting from the point \(P_1(1.988,0.15)\), where cell–cell adhesion is low, a 3-mode perturbation will be unstable as the tumor grows to the stationary radius \(R_s\). Note that mode \(l=3\) is the maximally unstable mode at \(P_1\) for all the viscosity ratios (see Supplementary Materials). Starting from the point \(P_2(1.988,0.6)\), where the cell–cell adhesion is higher, the growth of the same initially perturbed tumor to \(R_s\) is stable. See Supplementary Materials for an analysis of the maximally unstable mode for the parameters corresponding to \(P_2\).

Viscosity ratio (soft tumor) \(\lambda =2\). In Fig. 6a, we present the nonlinear evolution of a tumor starting from the point \(P_1\). The initial tumor/host interface is a 3-mode perturbation of a circle given by Eq. (4.2). The corresponding nonlinear shape factor \(\delta /R\) (solid) and linear shape factor (dashed) are shown in Fig. 6b. As predicted by linear stability theory, the mode \(l=3\) perturbation initially decreases (see inset) as the tumor grows until the effective radius of the tumor crosses the instability threshold radius, which is the radius at which the horizontal line \({{\mathcal {G}}}=0.15\) and the marginal stability curve \({{\mathcal {G}}}_M\) cross. After this, the perturbation starts to grow. Three buds initially form from a central core in directions dictated by the 3-fold initial data. These protrusions trap the healthy tissue in channels that penetrate the tumor central core, which provide the tumor center with more exposure to nutrients since the healthy tissue is assumed to be well vascularized (nutrient is constant in the host healthy tissue). This explains the larger center at \(t=70\). In the meantime, each protrusion continues to grow flatter at the top. At late times, secondary protrusions start to form at the base of the parent protrusions but are mostly directed inward because the environment is more viscous and the nutrient is more readily available. These nonlinearly generated secondary protrusions grow closer to one another leading to encapsulation of host tissue near the tumor center. The tumor grows well beyond its diffusion-limited size \(R_s\approx 3.3255\) due to the shape instabilities that develop, which increase the access of the cells to nutrients and enables the noncircular tumors to continue growing beyond the point at which their circular counterparts would stop.

Fig. 6
figure 6

a Nonlinear evolution for the case \(\lambda =2\). The initial tumor surface as in Eq. (4.2). The blue dots indicate the position on the tumor surface where maximum shape factor is obtained. b Shape factor (\(\delta /R\)) as a function of time from nonlinear simulations (solid line) and linear theory (dashed line). The blue dots give the shape factor at the interface positions in part (a). We consider parameter values are \({{\mathcal {A}}}=0.5\), \({{\mathcal {G}}}=0.15\), and \(l=3\) and resolutions \(N=512\) and \(\varDelta t=10^{-2}\) (color figure online)

Interestingly, the nonlinear shape factor is a non-monotonic function of time and achieves a maximum shortly before the buds apparently grow into one another. Because we have assumed that the nutrient is constant in the host, the core and buds continue to grow as they come into close contact. The point on the tumor/host interface where the largest shape factor occurs (blue circles) is in the central core, where there is an invagination of more viscous host tissue into the less viscous tumor. The maximum in \(\delta /R\) occurs because the invaginating finger tends to flatten out as the secondary protrusions start to form. As seen above, linear theory underpredicts the instabilities because of the generation of new unstable modes through nonlinear interactions.

The effect of viscosity ratio and initial tumor radius. In Fig. 7a, b the nonlinear morphologies of tumors starting from points \(P_1\) and \(P_2\), respectively, are shown. In each case, the initial tumor/host interface is a 3-mode perturbation of a circle given by Eq. (4.2). The viscosity ratio is varied from \(\lambda =0.05\) to \(\lambda =10\) as labeled. The figures show the nonlinear shape factors as functions of time; insets show the corresponding tumor morphologies at time \(t=20\) and at the final time \(t_f\) of the simulation, which depends on \(\lambda \) as described below. The colors of the tumor insets and the shape factor curves correspond. The effective radii of the tumors are plotted (solid curves) in Fig. 7e, f. When the cell–cell adhesion is small (\({{\mathcal {G}}}=0.15\), evolution from \(P_1\)), Fig. 7a, e show that the tumor grows unstably, and the time scale of unstable growth depends strongly on the viscosity ratio such that tumors with small \(\lambda \) are more unstable than those with large \(\lambda \). This is consistent with the predictions of linear stability theory (Fig. 4b). Note that the curves in (a) stop because higher numerical resolution is needed to resolve high curvature region. In addition, for \(\lambda \ge 0.1\), the tumor shape appears to want to reconnect, which would represent a mathematical singularity in this sharp interface model. To go beyond the times shown, more efficient adaptive numerical methods need to be used together with a physical regularization (e.g., viscosity solution or phase field type approach) to go beyond reconnection (Macklin and Lowengrub 2007; Wise et al. 2008). These are the subject of future work.

Fig. 7
figure 7

Nonlinear evolution with \({{\mathcal {A}}}=0.5\) with different viscosity ratios as labeled. The shape factor (\(\delta /R\)) is shown as a function of time, with accompanying tumor morphologies shown as insets. a Unstable growth corresponding to the point \(P_1\) from Fig. 4. b Stable growth corresponding to the point \(P_2\) from Fig. 4. In (a, b), the initial interface is given by three-mode perturbation of a circle from Eq. (4.2) where the initial unperturbed radius \(R_0=1.988\), and \({{\mathcal {G}}}=0.15\) in (a) and \({{\mathcal {G}}}=0.6\) in (b). Note that the curves in (a) stop because the tumor shapes experience reconnection, i.e. the lobes touch. Below the same in (c, f). c Unstable evolution from point \(P_3\) from Fig. 4. d Shrinkage from point \(P_4\) from Fig. 4. In (c, d), the initial interface is given by three-mode perturbation of a circle from Eq. (4.2) but with unperturbed radius \(R_0=4.5\) and \({{\mathcal {G}}}=0.15\) in (c) and \({{\mathcal {G}}}=0.6\) in (d). e The effective radii of the tumors from the unstable dynamics in Fig. 7a shown as the solid curves (\(R_0=1.988\)) and in Fig. 7c shown as the dashed curves (\(R_0=4.5\)). f The effective radii of the tumors from the stable dynamics in Fig. 7b and shown as the solid curves (\(R_0=1.988\)). The dashed curves correspond to the stable dynamics from Fig. 7d where \(R_0=4.5\). In (e), \({{\mathcal {G}}}=0.15\) and in (f), \({{\mathcal {G}}}=0.6\). The colors correspond to those used in Fig. 7a–d. Note that in (f), the dashed and solid curves lie on top of one another

When \(\lambda =0.05\), the tumor morphology evolves similarly to that observed when \({{\mathcal {A}}}=0\) as three broad buds form and high curvature regions develop in the dimples between the buds that limits our ability to simulate their evolution further. At slightly larger viscosity ratios (\(\lambda =0.1\) and \(\lambda =0.5\)), the buds split, creating a more complex shape. As the viscosity ratio increases, this splitting is suppressed and the tumors grow more compactly. As in the \(\lambda =2\) case shown above, for each \(\lambda \ge 0.1\), the nonlinear shape factor is a non-monotonic function of time and attains a maximum shortly before the final time \(t_f\), which is just before the buds seemingly collide with one another. Observe that the amount of trapped host tissue inside the tumor decreases as the viscosity ratio increases, due to the more compact shape of the tumors with larger \(\lambda \). Interestingly, the maximum values attained by the shape factors are increasing functions of \(\lambda \). This is because of the increasingly complex invaginations of host tissue within the tumor as \(\lambda \) increases.

When the cell–cell adhesion is increased to \({{\mathcal {G}}}=0.6\) (point \(P_2\)) in Fig. 7b, the \(l=3\) mode perturbation decreases and the tumor grows stably to its diffusion-limited size \(R_s\approx 3.3255\) (Fig. 7f). In this case, the effective radii–solid curves in Fig. 7f—collapse onto one another as predicted by theory [recall Eq. (2.39)].

Next, we increase the initial tumor radius to \(R(0)=4.5\) and consider the evolution from the points \(P_3\) (low adhesion, \({{\mathcal {G}}}=0.15\)) and \(P_4\) (higher adhesion, \({{\mathcal {G}}}=0.6\)). Linear stability theory (Fig. 4b) predicts that when \({{\mathcal {G}}}=0.15\), the 3-mode perturbation should increase for all viscosity ratios as the tumors shrink to the stationary size \(R_s\approx 3.3255\), with the smaller viscosities being more unstable than the larger ones. When \({{\mathcal {G}}}=0.6\), the \(l=3\) mode perturbations may grow initially, but as the tumor shrinks the perturbations eventually become stable as the radius decreases below the \(\lambda \)-dependent threshold radius. In contrast to the case with small \(R(0)=1.988\), the threshold radius is a decreasing function of \(\lambda \) so that tumors with small viscosities (large \(\lambda \)) must reach a smaller size in order for the 3-mode perturbations to decay. For example, when \(\lambda =10\), the threshold radius is nearly equal to the stationary radius \(R_s\). See Supplementary Materials for an analysis of the maximally unstable mode in this case.

As predicted by linear stability theory, when \({{\mathcal {G}}}=0.15\) (point \(P_3\)), the nonlinear tumor evolution is unstable (Fig. 7d). However, in contrast to the predictions of linear theory, the tumors are actually growing as the instabilities develop (dashed curves Fig. 7e). And, strikingly, the shapes are very similar to those obtained when the initial radius is smaller (evolution from \(P_1\): \(R(0)=1.988\) in Fig. 7a). An examination of the effective tumor radii in Fig. 7e reveals the mechanism behind this behavior. At early times, the tumors shrink as predicted by theory. However, as the tumors near the stationary size, shape perturbations drive nonlinear interactions among the modes. This causes the tumors to start growing instead of shrinking. Once the tumors start to grow, their dynamics is very similar to that obtained in Fig. 7a because the perturbations are still dominated by mode \(l=3\) and its harmonics.

When \({{\mathcal {G}}}=0.6\) (point \(P_4\)), the nonlinear behavior is well predicted by linear theory. The nonlinear evolution shows slight shape instabilities (increasing shape factors) at early times (Fig. 7d) with larger viscosity ratios being more unstable (mode \(l=3\) is maximally unstable at \(P_4\)). Eventually the perturbations decay to zero. Concomitantly, the effective radii decrease to the stationary radius at the same rate (dashed curve in Fig. 7f).

4.3 Evolution near the stationary radius (\({{\mathcal {A}}}=0.7\))

In this section, we increase the value of apoptosis to \({{\mathcal {A}}}=0.7\), which coincides with a stationary unperturbed tumor spheroid with radius \(R=1.988\). The ordered pair \((R_s,~{{\mathcal {G}}})=(1.988,~0.15)\) lies in the linearly unstable regime for \(l=3\) independent of the viscosity ratio (Fig. 3). Mode \(l=3\) is the maximally unstable mode at this \(R_s\) and \({{\mathcal {G}}}\) (see Supplementary Material). Here, we investigate the nonlinear evolution using the viscosity ratios \(\lambda =0.05\) and \(\lambda =0.5\) and the initial shape given in Eq. (4.2). The results are shown in Fig. 8.

Fig. 8
figure 8

Nonlinear evolution of tumors with \({{\mathcal {A}}}=0.7\), which is the stationary value for a tumor with radius \(R_0=1.988\) i.e. where \(dR/dt=0\), and the adhesion parameter \({{\mathcal {G}}}=0.15\). The initial tumor is a three-mode perturbation of a circle with radius \(R_0=1.988\) given in Eq. (4.2). In (a) the viscosity ratio \(\lambda =0.05\) and in (b) \(\lambda =0.5\). The red dots indicate the interface positions where maximal shape perturbation \(\delta /R\) is obtained. c Nonlinear evolution of a multimodal initial interface with \({{\mathcal {A}}}=0.7\), \({{\mathcal {G}}}=0.15\), \(R_0=1.988\) and \(\lambda =0.5\). The initial tumor surface is from Eq. (4.3). The red dots indicate the interface positions where maximal shape perturbation \(\delta /R\) is obtained (color figure online)

When \(\lambda =0.05\), the three initial protrusions grow and repeatedly split forming a branched structure that penetrates the host tissue. To extend the simulation beyond the time shown, higher numerical resolution is needed to resolve the high curvature region. On the other hand, when \(\lambda =0.5\), the three initial protrusions also grow but in a much more compact manner. Instead of forming a branched structure, increased host viscosity, and hence resistance to motion, tends to make the initial protrusions to flatten as the they grow and evolve toward one another leading to the encapsulation of a complex network of host tissue analogous to what we observed when \({{\mathcal {A}}}=0.5\) (Fig. 7a; e.g., point \(P_1\)). To go beyond the time shown, a physical regularization is needed to accommodate the interface reconnection (Macklin and Lowengrub 2007; Wise et al. 2008).

To investigate the dependence of the results on the symmetry imposed by the initial tumor morphology, we consider next the evolution of an asymmetric, multimodal initial tumor interface with \(\lambda =0.5\):

$$\begin{aligned} (x(\alpha ),y(\alpha ))= & {} (1.988+0.05\cos (2\alpha ) +0.1\cos (3\alpha )+0.08\sin (4\alpha )\nonumber \\&+\,0.12\cos (5\alpha ))(\cos \alpha ,\sin \alpha )/1.988 \end{aligned}$$
(4.3)

The nonlinear evolution (see Fig. 8c) shows that when multiple modes are present the same basic features are preserved, although the shape is not as compact as the single mode symmetric case indicating the mixing of branching and encapsulating behavior. Again, higher numerical resolution is needed to extend the simulation beyond the time shown.

5 Conclusions

We investigated the dynamics of avascular tumors in two dimensions using a two-fluid Stokes flow model where the viscosities of the tumor and host environment may be different. We performed a linear morphological stability analysis and used a boundary integral method to study the effects of nonlinearity. A novel reformulation of the equations enabled the use of standard boundary integral solvers. We focused on the effects of cell–cell adhesion, apoptosis, and the ratios of the host and tumor viscosities. As in other tumor growth models (e.g., Darcy law by Li et al. 2007) cell–cell adhesion stabilizes tumor morphologies while apoptosis is destabilizing. Unlike Darcy flow models (Macklin and Lowengrub 2007), however, when the tumors were more viscous than their environment, we found the development of invasive fingers that lead to a branch-like structure. As the relative ratio of the host to tumor viscosities is increased, the tumors tend to grow more slowly and more compactly. Further, under these conditions, the tumors also developed invaginations of healthy regions that became encapsulated in the tumor interior. To extend the simulations beyond these times, higher numerical resolution and physical regularizations of the sharp interface model are needed. These are the subject of future work.

There is much evidence now in the literature that mechanical forces in the tumor microenvironment play a key role in tumor progression and response to therapy (Butcher et al. 2009; Carey et al. 2012; Jain et al. 2014; Wei and Yang 2016). At a macroscopic scale, tumors are frequently found to be stiffer than normal tissue and this can increase invasiveness into soft tissues (Butcher et al. 2009). This is consistent with our findings that branching morphologies tend to develop when the host viscosity is less than that of the tumor. In addition, numerous in vitro experiments have shown that stiffer extracellular matrices lead to slower growth and more compact tumor shapes (Helmlinger et al. 1997; Cheng et al. 2009; Montel et al. 2011; Delarue et al. 2014), which is also consistent with our results. Although several experiments investigating the properties of cells with different invasiveness demonstrated that tumor cells are softer than normal cells (Lekka et al. 1999; Cross et al. 2008), the actual tumor may be stiffer than normal tissue because of the structure of the environment and surrounding tissues. For example, the tumor may contain a higher density of extracellular matrix than normal tissue and may also contain calcified regions. We note that our model does predict that instability may also occur when the tumor is less viscous that the surrounding normal tissue.

In experiments, mechanical stresses have been found to influence cell fates and motilities (Butcher et al. 2009), proliferation rates (Delarue et al. 2014) and apoptosis rates (Cheng et al. 2009). Here, we have not modeled these effects and we defer such studies to future work. In addition, because we use a Stokes flow constitutive model for the tumor and host tissue, elastic and viscoelastic effects are neglected. At long times, however, cell rearrangements may make a fluid-like treatment of the tissues appropriate although this needs to be further investigated. Another future research direction involves investigating the effects of non-constant cell substrate diffusivity since the diffusivity may depend on the properties of the local tissue (e.g., density, viscosity). Spatial variability in the delivery cell substrates could in turn influence tumor morphological stability. Finally, while we presented results in two dimensions, our results are expected to hold qualitatively in three dimensions as suggested by the similarity between the two- and three-dimensional stability analyses (Cristini et al. 2005; Pham et al. 2010; Lowengrub et al. 2009).