Keywords

1 Introduction

The mechanical contraction of a heart, which pumps the blood throughout the entire body, is caused by its electrical activity. In order to understand how the heart works, it is important to be able to simulate cardiac electrical processes. However, heart simulation is a complex multilevel (cell-tissue-organ) modeling task [9] that is very computationally intensive. Therefore, for a fast and accurate heart simulation, parallel computing is required.

However, the parallel heart simulation is impeded by the two obstacles. First, it requires a deep knowledge in a number of modern computational architectures and parallel programming technologies, which most of biophysicists do not possess. Secondly, sophisticated multilevel models are hard to implement in code, especially when the complex optimization for modern computational architectures is required. As a result, multilevel simulation software is very hard to support, while models and computational architectures are changing constantly.

Nowadays, automated scientific computing frameworks that allow software development using near-mathematical notation are becoming popular. Recently, the performance of such frameworks was significantly improved by the use of just-in-time compilers, highly efficient mathematical libraries, parallel computing, etc.

We propose an approach to simulation of the heart electrical activity using the scientific computing framework FEniCS [6], which provides the ability to automatically solve partial differential equations (PDE) using the finite element method (FEM) on MPI clusters. We use the FEniCS framework to study the space propagation of the membrane potential alternation over the left ventricle (LV) of a human heart using the ten Tusscher–Panfilov (TP06) cell model [10].

Due to the fact that FEM produces sparse matrices, computations heavily depend on the degree of sparsity because it allows to use various optimization techniques such as the compressed row storage, the sparse matrix-vector multiplication, or custom approaches [3]. Various models produce matrices with different degrees of sparsity. Hence, linear solvers and appropriate preconditioners demonstrate different results on parallel systems for various models. We evaluated which combination of the linear solver and the preconditioner is the most suitable for the simulation of a cardiac electrical activity using the TP06 model.

2 Model of the Heart Electrical Activity

There are many mathematical models of cardiac electrical activity. However, all of them contain the description of the action potential (AP), which is the difference of the potential between the intra- and extracellular space. We adopted the TP06 model [10, 11] for the simulation of the electrical activity in the LV of a human heart. This model uses the reaction-diffusion equations to describe the space and time evolution of the action potential (V):

$$\begin{aligned} C_{m} \cdot \frac{dV}{dt} = \nabla \cdot (D \nabla V) - I_{ions}, \end{aligned}$$
(1)
$$\begin{aligned} \frac{dS}{dt} = g(V,S), \end{aligned}$$
(2)

where \(C_{m}\) is the capacitance of a cell membrane, D is the \(3 \times 3\) diffusion matrix, \(I_{ions}\) is the sum of the ionic currents, S is the vector of the model variables that govern the ion currents, and g is the vector-valued function that describes the time evolution of each variable. The boundary conditions provide the LV electrical isolation.

On the intracellular level, the electrical potential arises from a very complicated interaction among ionic currents and cell organelles (organized structures in cells). The TP06 model contains the equations that describe how does the state of ion channels change with time and the kinetics of intracellular concentrations of calcium, sodium and potassium, extracellular potassium, the kinetics of calcium complexes, and calcium kinetics in the organelles. All of this processes are described by 18 phase variables. System (1)–(2) is defined at each point of the heart tissue, and, consequently, we should solve it for each node of the computational mesh.

Thus, the TP06 model is a nonlinear system of partial and ordinary differential equations (ODE) that cannot be solved analytically and, hence, must be solved on a computer using numerical techniques. This task is highly computationally intensive due to the big number of equations in the 3D domain and the stiffness of the TP06 model.

3 FEniCS and the Finite Element Method

FEM provides a powerful methodology for discretizing differential equations, however, it produces algebraic systems the solution of which is also a challenge. Linear solvers must handle sparsity and possible ill-conditioning of the algebraic systems. In addition, modern solvers should also be able to use parallel computing systems efficiently. The FEM implementation in FEniCS is intended to automate a PDE solution. In particular, FEniCS relies on the automation of discretization, discrete solution, and error control. FEniCS provides two approaches for a PDE solution: direct and iterative. Iterative solution is more efficient because it uses less memory and is easier to parallelize [6].

The FEniCS framework is a collection of software components for the formulation of variational forms (UFL [1]), the discretization of variational forms (FIAT, FFC [4]), and the assembly of the corresponding discrete operators (UFC, DOLFIN [7]). To solve a problem, FEniCS uses several highly efficient parallel algebra backends, such as PETSc and Hypre. UFL is a domain-specific language designed for convenient and understandable formulation of variational forms using the near-mathematical notation. The discretization of variational forms is done by generation of arbitrary order instances of the Lagrange elements on lines, triangles, and tetrahedra (FIAT), and compilation of efficient low-level C++ code that can be used to assemble the corresponding discrete operator (FFC). The assembly of the discrete operators (tensors) is crucial for acceleration on parallel computing systems. The idea is to split the mesh among processing units, compute the local matrix, and insert the values back into the global matrix. The FEniCS team designed the local-to-global mapping algorithm [4] to map values between the local and global matrices.

The most computationally intensive task is solving the local linear system. Hence, optimization of this step by selecting appropriate linear solver and preconditioner can provide a significant computation speedup.

4 Model Implementation in FEniCS

In order to implement the TP06 model in FEniCS, we transformed the nonlinear system (1)–(2) into a linear one, which let us use iterative solvers. The transformation was performed with the help of the first order operator splitting scheme (the Marchuk–Yanenko method) [5]. The scheme of computing \(V(t_n)\) and \(S(t_n)\) can be described as follows. Let us assume that we have already calculated the values of V(t) and S(t) for \(t<t_n\). In order to find the values of \(V(t_n)\) and \(S(t_n)\), we solve Eq. (3),

$$\begin{aligned} \begin{aligned} \frac{dV^{*}}{dt} = D \nabla V^{*}, V^{*}(t=t_{n-1}) = V(t_{n-1}), t \in [t_{n-1}, t_n], \\ \frac{dV^{**}}{dt} = I_{ions}, V^{**}(t=t_{n-1}) = V^{*}(t_{n}), \\ \frac{dS^{**}}{dt} = g(V^{*},S^{**}), S^{**}(t=t_{n-1}) = S(t_{n-1}). \end{aligned} \end{aligned}$$
(3)

First, we solve the diffusion PDE. After that we have to find the solution of the ODE system for cell ionic currents. We get the final values of \(V(t_n)\) and \(S(t_n)\) according to the rules \(V(t_n) = V^{**}(t_n)\) and \(S(t_n) = S^{**}(t_n)\). This method is also known as the method of splitting into physical processes. The disadvantage of the approach is the necessity to use a very small integration time step (0.0005 s) in order to capture the fast electrochemical processes.

The model was implemented in the Python language using UFL. The code fragment for the diffusion PDE problem formulation is presented in Listing 1.1. First, a finite element mesh is created and loaded from the file. After that the discrete function space for AP is defined. FEniCS uses the term trial function to specify the unknown function that should be approximated (the variable v contains a trial function and the v0 variable contains the initial values). The next step is to define the linear variational problem for the diffusion equation. Lastly, the PDE solver is created.

figure a

Listing 1.2 demonstrates the code fragment for solving differential equations. The first step in the for loop solves the diffusion PDE using the PDEsolver. After that, the values of the state variables and AP, which was computed on the previous step, are stored. Next, the ODE system describing the cell ion currents is solved using the ODEsolver. There is no need for explicit, manual parallelization because the parallelization is provided by FEniCS. In addition to parallel computation, FEniCS provides the parallel output, during which each process writes its part of the data to a single file.

figure b

5 Performance Evaluation

During the experiments, we simulated the electrical activity of the human heart LV using the asymmetric anatomical model that was previously developed in our group [8] (an example of LV 3D mesh is presented in Fig. 1). We used the tetrahedral mesh with the length of the tetrahedrons from 2 to 4 mm; the mesh contained 7178 points and 26156 tetrahedrons. The GMSH software [2] was used for the initial mesh generation. Next, the mesh was converted by the DOLFIN module to the HDF5 format in order to enable parallel I/O operations.

The initial simulation conditions were the activation of a small part of LV near the apex (the potential is greater than 40 millivolt). The simulation duration was 0.3 s of physical time, because after this period the electrical activity tends to the equilibrium state in absence of an external stimulus.

The experiments were carried out on the Uran supercomputer of the Krasovskii Institute of Mathematics and Mechanics with the following computational nodes configuration: 2 x Intel Xeon CPU X5675 CPU, 192 GB RAM, Infiniband DDR interconnect, CentOS 7 operating system. The FEniCS version 1.6.0 was used.

Table 1. Parallel Krylov solvers and preconditioners available in FEniCS
Full size table
Fig. 1.
figure 1

An example of 3D mesh of the left ventricle (asymmetric model)

Full size image
Table 2. The simulation time (minutes) using for various numbers of CPU cores
Full size table
Fig. 2.
figure 2

Simulation speedup depending on the number of CPU cores

Full size image

The TP06 model implementation was executed on the Uran supercomputer in parallel using various numbers of CPU cores, from 1 to 240. We used Krylov parallel linear solvers and preconditioners available in FEniCS (Table 1). The simulation time with various combinations of solvers and preconditioners is presented in Table 2, the achieved speedup is demonstrated in Fig. 2.

6 Discussion

The experiments demonstrated that the FEniCS-based TP06 model implementation provides acceptable performance and good scalability. The best result was achieved using the conjugate gradient method and the biconjugate gradient stabilized method solvers with the successive over-relaxation preconditioner: 19 min of the simulation time, 95 times speedup using the 240 CPU cores.

Choosing the appropriate combination of the solver and the preconditioner is an important task. The best combination from our experiments (Table 1) provided 30% more performance on 240 CPU cores than the worst one (the biconjugate gradient stabilized method solver with the algebraic multigrid preconditioner). To save space, we presented in the paper only the best experiment results. FEniCS includes other solvers and preconditioners not listed in Table 1. Hence, in practice, the difference in performance of the best combination and other solvers and preconditioners available in FEniCS could be more than 30%.

As the number of CPU cores increases, the preconditioner’s influence on performance becomes greater than the solver’s. When we conducted the simulation on 132 CPU cores or more, there was no tangible difference in performance between different solvers working with the same preconditioner (Table 2).

7 Conclusion

The created implementation of the TP06 model uses the near-mathematical notation provided by the FEniCS framework. As a result, computational mathematicians and biophysicists can use this implementation for experimenting with the model. They can easily modify the model parameters, the initial activation conditions, and even change the model itself. Despite the usage of the near-mathematical notation, our implementation provides an acceptable performance and scales well. The possible direction of the future work is to use the TP06 model implementation for simulation of complicated processes in LV that can cause heart diseases, such as scroll wave dynamics. Another important task is to implement the model of mechanical heart activity using FEniCS and provide the ability to simulate electro-mechanical function of the heart.