Dynamic metabolic resource allocation based on the maximum entropy principle

Abstract

Organisms have evolved a variety of mechanisms to cope with the unpredictability of environmental conditions, and yet mainstream models of metabolic regulation are typically based on strict optimality principles that do not account for uncertainty. This paper introduces a dynamic metabolic modelling framework that is a synthesis of recent ideas on resource allocation and the powerful optimal control formulation of Ramkrishna and colleagues. In particular, their work is extended based on the hypothesis that cellular resources are allocated among elementary flux modes according to the principle of maximum entropy. These concepts both generalise and unify prior approaches to dynamic metabolic modelling by establishing a smooth interpolation between dynamic flux balance analysis and dynamic metabolic models without regulation. The resulting theory is successful in describing ‘bet-hedging’ strategies employed by cell populations dealing with uncertainty in a fluctuating environment, including heterogenous resource investment, accumulation of reserves in growth-limiting conditions, and the observed behaviour of yeast growing in batch and continuous cultures. The maximum entropy principle is also shown to yield an optimal control law consistent with partitioning resources between elementary flux mode families, which has important practical implications for model reduction, selection, and simulation.

1 Introduction

Dynamic models of metabolism have been introduced as extensions to static, steady state modelling techniques such as flux balance analysis (FBA) (Varma and Palsson 1994; Orth et al. 2010) and elementary flux mode analysis (Schuster and Hilgetag 1994) in order to describe adaptation of cellular activity to changes in the environment. Established examples include dynamic FBA (DFBA) (Mahadevan et al. 2002), macroscopic bioreaction models (Provost and Bastin 2004; Provost et al. 2006), and cybernetic theory based on the optimal control framework by Young and Ramkrishna (2007) and Young et al. (2008). Both DFBA and cybernetic theory incorporate regulation of flux across a metabolic reaction network based on some optimality criteria, whereas macroscopic bioreaction models do not and are therefore considered unregulated. Young and Ramkrishna (2007) and Young et al. (2008) posit that regulatory decisions take the form of a constrained optimisation problem, which must be solved to optimally distribute limited resources among pathways in the network. More recently, various related extensions of DFBA based on resource allocation have been introduced to accommodate the limited capacity for gene expression into dynamic models of metabolism (e.g. Rügen et al. 2015; Waldherr et al. 2015; Lindhorst et al. 2018). The concept of resource allocation has also been considered in the static case (Goelzer and Fromion 2011; Wortel et al. 2014; Müller et al. 2014; Mori et al. 2016; de Groot et al. 2019), where it is suggested that resource constraints arise due to finiteness of the total cellular proteome and the fraction that corresponds to metabolic enzymes. In Wortel et al. (2014) and Müller et al. (2014), it was shown that the FBA solution to the resource allocation problem is to allocate the entirety of resource exclusively to the metabolic pathway maximising the cellular objective.

From a strategic perspective, cell populations may instead prefer to spread resource among multiple metabolic pathways in order to deal with uncertainty in a fluctuating environment, which could explain the heterogeneity in metabolic pathway use observed experimentally (Levy et al. 2012; Solopova et al. 2014; Ackermann 2015; Martins and Locke 2015; Granados et al. 2017). Such ‘bet-hedging’ arguments are akin to various economic theories (Buchen and Kelly 1996; Hansen and Sargent 2001; Sims 2003) that posit multiple investments are beneficial to individuals subjected to uncertainty, or that individuals make exclusive investments, but in receipt of slightly different information. These theories are related to the principle of maximum entropy (Jaynes 1957; Shore and Johnson 1980), because from an information-theoretic standpoint the resource distribution that best represents the current state of knowledge is the one with largest entropy: entropy uniquely satisfies the accepted axioms for an uncertainty measure (up to a constant factor) (Shannon 1948), and therefore the maximum entropy distribution consistent with known constraints is uniquely determined as the one that expresses maximum uncertainty with respect to everything else. In biology, this mathematical justification for maximum entropy as an investment strategy that best-accommodates uncertainty forms the basis of various ecological theories [see Harte and Newman 2014 for a review], and also an interpretation of stem cell multi-potency (Ridden et al. 2015). Analogously, in Kussell and Leibler (2005) it was demonstrated that phenotype-switching strategies that are adjusted to the entropy of environmental fluctuations can outperform those that are not. The maximum entropy principle has also recently been applied to static metabolic modelling in various scenarios, including: extensions of FBA to include population heterogeneity (De Martino et al. 2017; Fernandez-de-Cossio-Diaz and Mulet 2019), experimental decomposition of fluxes using elementary mode analysis (Zhao and Kurata 2009, 2010), and to put forward the suggestion that organisms evolve toward a state of maximum physical entropy (Srienc and Unrean 2010; Unrean and Srienc 2011). Hitherto, there has been no attempt to incorporate the maximum entropy principle into dynamic models of metabolism with resource allocation.

This paper builds upon the work of Young and Ramkrishna (2007) and Young et al. (2008) with the purpose of introducing a dynamic model of metabolic resource allocation based on the maximum entropy principle. Although a similar optimal control framework is employed and metabolic network decomposition is also performed using elementary flux modes (EFMs) (Schuster and Hilgetag 1994), optimality criteria for resource allocation are instead stated in terms of maximum entropy so as to accommodate environmental uncertainty, which produces an original control law. Moreover, the resulting theory is not cybernetic in the sense that there is no reliance on multiple control laws, nor are cybernetic enzymes introduced as auxiliary dynamical variables. The maximum entropy framework unifies DFBA (Mahadevan et al. 2002) and unregulated macroscopic bioreaction models (Provost and Bastin 2004; Provost et al. 2006) as two limiting extremes of the general theory. A further consequence for dynamic resource allocation is that no assumption beyond maximisation of total catalytic biomass is necessary to describe accumulation of cellular reserve compounds in growth-limiting environments (Reimers et al. 2017; Tajparast and Frigon 2018). The maximum entropy control also turns out to be consistent with model reduction using EFM families (Song and Ramkrishna 2010, 2011; Vilkhovoy et al. 2016), which is extremely useful from a modelling point of view because EFM enumeration can result in a combinatorial explosion as metabolic networks grow in size (Klamt and Stelling 2002).

The remainder of this paper is organised as follows: Sects. 2 and 3 introduce the dynamic metabolic model and maximum entropy control, which are then extended to include metabolite yields in Sect. 4. Section 5 describes the dynamic maximum entropy framework as applied to model reduction using EFM families, and Sect. 6 presents a specific application of the theory to yeast metabolism. This application is the culmination of a number of working examples found at the end of each section. Additional mathematical details expanding on some parts of the main text can be found in the “Appendix”.

2 Dynamic model of metabolism

The following dynamical system is considered as a model for metabolism in batch culture

$$\begin{aligned} \begin{aligned} \frac{d}{dt} \mathbf {m}_{ex}&= \mathbf {S}_{ex} \mathbf {v} x \\ \frac{d}{dt} \mathbf {m}_{in}&= \mathbf {S}_{in}\mathbf {v} - \mu \mathbf {m}_{in} \\ \frac{d}{dt}x&= \mu x , \quad \mu = \mathbf {c}^T \mathbf {v} . \end{aligned} \end{aligned}$$

(1)

Here $\mathbf {m}_{ex}$ ($\text{ g }\; \text{ L }^{-1}$), $\mathbf {m}_{in}$ ($\text{ g } \; \text{ L }^{-1} \; \text{ gDW }^{-1} \; \text{ L }$; $\text{ gDW }$, grams dry weight) are vectors of extra- and intracellular metabolites, respectively, and $\mathbf {S}_{ex}$, $\mathbf {S}_{in}$ the corresponding portions of the stoichiometric reaction matrix $\mathbf {S}$ (Varma and Palsson 1994; Orth et al. 2010; Schuster and Hilgetag 1994). The scalar variable x ($\text{ gDW } \; \text{ L }^{-1}$) represents the concentration of total catalytic biomass responsible for catalysing reactions involved in its own production and interconversion of metabolites, and $\mu$ ($\text{ h }^{-1}$) is the rate of its accumulation (i.e., growth rate) formed as the inner product of the non-negative, N-dimensional flux vector $\mathbf {v} = (v_1,v_2,\ldots ,v_N)^T$ ($\text{ g } \; \text{ gDW }^{-1} \; \text{ h }^{-1}$) with the constant coefficient vector $\mathbf {c} = (c_1,c_2,\ldots ,c_N)^T$ ($\text{ gDW } \; \text{ g }^{-1}$). When the reactions are irreversible (which can be assumed after splitting each reversible reaction into two irreversible ones), the reaction fluxes $v_i \ge 0$ can be decomposed as $v_i = e_i f_i(\mathbf {m})$ with $\mathbf {m} = (\mathbf {m}_{ex}, \mathbf {m}_{in})^T$, where $e_i$ is the relative concentration of the enzyme catalysing the ith reaction and $f_i(\mathbf {m})$ is the (non-negative) ‘saturation function’ of that enzyme, which includes the thermodynamic driving force, (allosteric) activation or inhibition, and other enzyme-specific effects (de Groot et al. 2019). In formulation of this model the relative enzyme concentrations $e_i$ are understood to be control variables, whose values are determined according to control laws in order to satisfy some objective, as are additional arguments of $f_i$ (omitted for notational simplicity) responsible for regulatory effects not directly attributable to the relative level of enzyme i. More precisely, while $e_i$ corresponds to relative levels of the ith enzyme, its activity is dependent on substrate availability and additional regulatory features, e.g., covalent modification, all encapsulated in the form of a single function $f_i(\mathbf {m})$. It is assumed these two types of control are enacted on distinct time scales so that the $e_i$ are considered slow control variables while the remaining control variables contained within $f_i(\mathbf {m})$ are considered fast. For the vast majority of biological models this is a realistic assumption, i.e. the process of transcription and translation of enzyme takes considerably larger than its post-translational regulation (e.g., via phosphorylation).

Following Klipp et al. (2002) and de Groot et al. (2019), the control variables corresponding to enzyme levels satisfy the constraint

$$\begin{aligned} \sum _{i=1}^N e_i \le 1, \end{aligned}$$

(2)

which in this case could correspond to a limited capacity for protein synthesis on ribosomes. In de Groot et al. (2019), expressions like (2) come with a set of weights or ‘costs’, one for each $e_i$, but here these are absorbed into the $f_i(\mathbf {m})$ (although this is only possible for the case of a single constraint). Regulation of the $e_i$ and remaining fast control variables appearing in (1) is assumed to occur such that some metabolic performance index J is maximised, which combined with constraint (2) introduces the general optimal control problem for resource allocation over the interval $[t_0,t_f]$:

$$\begin{aligned} \begin{aligned}&\text{ max } J = \Phi _{t = t_f}(\mathbf {m}_{ex}, \mathbf {m}_{in}, x) + \int _{t_0}^{t_f} L(\mathbf {m}_{ex}, \mathbf {m}_{in}, x, \mathbf {e}) dt\\&\text{ s.t. } \text{(1) } \text{ and } \text{(2) } \end{aligned} \end{aligned}$$

(3)

where $\Phi _{t = t_f}$ is a terminal objective function and L an intermediate objective function evaluated at $\mathbf {e} = (e_1,e_2,\ldots ,e_N)^T$. Initial conditions for the dynamic variables in (1) may also be given. From a modelling perspective however, it is conventionally not the case that the full system (1) is considered due to the immense number of unmeasurable parameters necessary to provide an accurate dynamical description of intracellular metabolism. Common practice is therefore to invoke the quasi-steady state assumption (QSSA) on intracellular metabolism (Varma and Palsson 1994; Schuster and Hilgetag 1994), which amounts to the assumption that metabolic transients are typically rapid compared to cellular growth rates and changes in the environment. Validity for the QSSA is obtained by comparing the time scale of metabolic processes (fast) to those of transcriptional and translational regulation (slow) (Heinrich and Schuster 1996). Assuming the dilution term $\mu \mathbf {m}_{in}$ is negligible for intracellular metabolites and invoking the QSSA reduces (1) to a lower-dimensional system of the form

$$\begin{aligned} \begin{aligned} \frac{d}{dt} \mathbf {m}_{ex}&= \mathbf {S}_{ex} \mathbf {v} x \\ 0&= \mathbf {S}_{in}\mathbf {v} \\ \frac{d}{dt}x&= \mu x , \quad \mu = \mathbf {c}^T \mathbf {v} . \end{aligned} \end{aligned}$$

(4)

There are two critical issues that should be called into question at this stage. First, the reduction of (1) to (4) based on the QSSA is formal, but it can be rigorously proven that, for fixed relative enzyme concentrations $e_i$, trajectories of the ordinary differential Eq. (4) are a good approximation for those of (1) provided the conditions of Tikhonov’s theorem are met (Khalil 2002). These conditions are almost always impossible to validate however, and so typically one needs to assume existence and stability of a quasi-steady state based on biophysical insight. See Waldherr et al. (2015) for a discussion of this point. Secondly, and this is not discussed in Waldherr et al. (2015), it is natural to approximate solutions to the optimal control problem (3) using solutions to the reduced problem

$$\begin{aligned} \begin{aligned}&\text{ max } J^{red} \\&\text{ s.t. } \text{(4) } \text{ and } \text{(2) } , \end{aligned} \end{aligned}$$

(5)

where it is understood that $J^{red}$ is the metabolic performance index evaluated on trajectories of the reduced system (4). However, to establish validity of this approximation one must appeal to the theory of singularly perturbed optimal control problems (Kokotovic et al. 1976) and prove that Pontryagin’s maximum conditions for the reduced problem (5) are equivalent to those obtained by invoking the QSSA on Pontryagin’s maximum conditions for the full control problem (3). Unfortunately, establishing equivalence of these two reduction methods remains an open problem for most nonlinear systems. That this equivalence is approximately satisfied should therefore be highlighted as an additional biological assumption for optimal control problems such as those considered here and in Waldherr et al. (2015). The assumption of this equivalence will be referred to as the quasi-reduction equivalent assumption (QREA).

Proceeding under the condition that both the QSSA and QREA are valid, a complete set of vectors $\{\mathbf {Z}^k\}_{k=1,2,\ldots ,K}$ representing EFMs (Schuster and Hilgetag 1994) or extremal rays for the flux cone $FC= \{ \mathbf {v} : \mathbf {S}_{in} \mathbf {v} = 0 , v_i \ge 0 \quad \forall i \}$ can be used to express any $\mathbf {v} \in FC$ as a conical combination

$$\begin{aligned} \mathbf {v} = \sum _{k=1}^K \lambda _k \mathbf {Z}^k , \quad \lambda _k \ge 0 \quad k =1,2,\ldots ,K. \end{aligned}$$

(6)

The $\mathbf {Z}^k$ are defined up to some multiplicative constant and the decomposition (6) represents any $\mathbf {v}$ satisfying constraints imposed by the intracellular component of the stoichiometric matrix (Schuster and Hilgetag 1994). As reviewed in Peres et al. (2018), there can also be thermodynamic constraints on $\mathbf {v}$, and restricting the set of EFMs to those that satisfy these additional constraints has recently been achieved in Peres et al. (2017). The summation in (6) may therefore be restricted to a subset of thermodynamically-feasible EFMs, because any thermodynamically-feasible $\mathbf {v}$ can be expressed solely in terms of thermodynamically-feasible EFMs (Jol et al. 2012). The converse statement however, that any $\mathbf {v}$ expressed as a linear combination of thermodynamically-feasible EFMs also satisfies the thermodynamic constraints, is not necessarily true, and so this puts a restriction on the interpretation of EFM-based dynamic modelling approaches (Provost and Bastin 2004; Provost et al. 2006; Song and Ramkrishna 2010, 2011; Vilkhovoy et al. 2016; Baroukh et al. 2014). From the decomposition $v_i = e_i f_i(\mathbf {m})$ one obtains (provided $f_i(\mathbf {m}) \ne 0$)

$$\begin{aligned} e_i = \sum _{k=1}^K \lambda _k \frac{Z^k_i}{f_i(\mathbf {m})} , \quad i =1,2,\ldots ,N \end{aligned}$$

where $Z^k_i$ is the ith element of the vector representing the kth EFM. Therefore constraint (2) becomes

$$\begin{aligned} 1 \ge \sum _{k=1}^K \lambda _k \sum _{i=1}^N \frac{Z^k_i}{f_i(\mathbf {m})} \equiv \sum _{k=1}^K u_k , \quad u_k \ge 0 \quad k =1,2,\ldots ,K \end{aligned}$$

(7)

where the new slow control variables $u_k = \lambda _k/r_k$ have been introduced along with

$$\begin{aligned} r_k(\mathbf {m}) = \left( \sum _{i=1}^N \frac{Z^k_i}{f_i(\mathbf {m})} \right) ^{-1} \end{aligned}$$

(8)

as the ‘composite’ flux through the kth EFM [compare with Young et al. (2008) in the dynamic and Wortel et al. (2014), Müller et al. (2014) in the static case].

At this point a choice needs to made for the way that the composite fluxes are to be represented in the reduced system. This is because the QSSA applied to the optimal control problem (3) using the decomposition (6) to express $\mathbf {v}$ in terms of EFMs does not follow Tikhonov’s theorem for ordinary differential equations, which is rather based on determining the slow manifold for $\mathbf {m}_{in}$ in terms of $\mathbf {m}_{ex}$. Common practice is to approximate the functional form of composite fluxes using (e.g., Michaelis–Menten) kinetic rate laws that depend on slow dynamic variables $\mathbf {m}_{ex}$ alone (Provost and Bastin 2004; Provost et al. 2006; Song and Ramkrishna 2010, 2011; Vilkhovoy et al. 2016; Baroukh et al. 2014). This choice limits the total number of parameters in the reduced system, but comes with a requirement to select a common normalisation for all EFMs because otherwise the system will not remain invariant to EFM scaling. Such an approximation is made in Sect. 5 and the application to yeast metabolism presented in Sect. 6, while for the general discussion it will be assumed that composite fluxes can be well-defined using expression (8) with the fast variables fixed at some constant value, $\mathbf {m}_{in}^*$, independent of the $\mathbf {m}_{ex}$. Substitution for $\mathbf {v}$ in the reduced system (4) yields

$$\begin{aligned} \begin{aligned} \frac{d}{dt}\mathbf {m}_{ex}&= x \sum _{k=1}^K r_k(\mathbf {m}_{ex}) \mathbf {S}_{ex} \mathbf {Z}^k u_k \\ \frac{d}{dt}x&= x \sum _{k=1}^K r_k(\mathbf {m}_{ex}) \mathbf {c}^T \mathbf {Z}^k u_k \end{aligned} \end{aligned}$$

(9)

where only explicit dependence of the $r_k$ on $\mathbf {m}_{ex}$ has been included because the $\mathbf {m}_{in}$ are now assumed constant by the QSSA as stated above. Although the vectors representing EFMs are specified only up to a multiplicative constant, the system (9) remains invariant to their re-scaling and is therefore well-defined. Under the QREA one arrives at the reduced optimal control problem

$$\begin{aligned} \begin{aligned}&\text{ max } J^{red} \\&\text{ s.t. } \text{(9) } \text{ and } \text{(7) } . \end{aligned} \end{aligned}$$

(10)

The above form of the dynamic resource allocation problem provides a natural interpretation for each control variable $u_k$ as the fraction of total catalytic biomass concentration x that is allocated to the kth EFM. The next section introduces the control law for determining the optimal fraction of this resource.

Example 1

Consider the simplified metabolic network in Fig. 1a as a model for central carbon metabolism, also chosen in Möller et al. (2018). Concentrations of extracellular metabolites glucose ($G_{ex}$), oxygen (O), product 1 ($P_1$), product 2 ($P_2$), and total catalytic biomass (x) are slow variables, while concentrations of intracellular glucose ($G_{in}$) and pyruvate (P) are fast. In this model only reactions with fluxes $v_1$ and $v_3$ are assumed to contribute directly to the growth rate, such that $c_0 = c_2 = c_4 = 0$ with $c_3>c_1 >0$. Invoking the QSSA on fast intracellular metabolite concentrations, stoichiometric matrices $\mathbf {S}_{ex}$ and $\mathbf {S}_{in}$ give rise to the reduced dynamical system

$$\begin{aligned} \frac{dG_{ex}}{dt}= & {} -v_0 x\\ \frac{dO}{dt}= & {} -v_3 x = -\frac{dP_2}{dt} \\ \frac{dP_1}{dt}= & {} (v_2 - v_4)x \\ 0= & {} v_0 - v_1 \\ 0= & {} 2v_1 - v_2 - v_3 + v_4 \\ \frac{dx}{dt}= & {} \mu x , \quad \mu = c_1v_1 + c_3 v_3. \end{aligned}$$

A complete set of three EFMs (represented graphically in Fig. 1b) is provided by the vectors

$$\begin{aligned} \mathbf {Z}^1 = \begin{pmatrix}1 \\ 1 \\ 2 \\ 0 \\ 0 \end{pmatrix}, \mathbf {Z}^2 = \begin{pmatrix}0 \\ 0 \\ 0 \\ 1 \\ 1 \end{pmatrix} , \mathbf {Z}^3 = \begin{pmatrix} 1 \\ 1 \\ 0 \\ 2 \\ 0 \end{pmatrix} , \end{aligned}$$

and these have respective composite fluxes

$$\begin{aligned} r_1(G_{ex})= & {} \left( \frac{1}{f_0(G_{ex})} + \frac{1}{f_1(G^*_{in})} + \frac{2}{f_2(P^*)} \right) ^{-1}\\ r_2(O,P_1)= & {} \left( \frac{1}{f_3(O,P^*)} + \frac{1}{f_4(P_1)} \right) ^{-1}\\ r_3(G_{ex},O)= & {} \left( \frac{1}{f_0(G_{ex})} + \frac{1}{f_1(G^*_{in})} + \frac{2}{f_3(O,P^*)} \right) ^{-1} \end{aligned}$$

where $G^*_{in}$ and $P^*$ denote the fixed steady state concentrations of intracellular glucose and pyruvate, respectively, and $f_i$ is the saturation function of the ith enzyme. Expressed in terms of EFMs and slow control variables $u_1,u_2,u_3$, the reduced dynamical system takes the form

$$\begin{aligned} \frac{dG_{ex}}{dt}= & {} -[ r_1(G_{ex})u_1 + r_3(G_{ex},O)u_3] x\\ \frac{dO}{dt}= & {} -[r_2(O,P_1)u_2 + 2 r_3(G_{ex},O)u_3] x = -\frac{dP_2}{dt} \\ \frac{dP_1}{dt}= & {} [2r_1(G_{ex})u_1 - r_2(O,P_1)u_2]x \\ \frac{dx}{dt}= & {} [c_1r_1(G_{ex})u_1 + c_3 r_2(O,P_1)u_2 + (c_1 + 2c_3) r_3(G_{ex},O) u_3]x . \end{aligned}$$

$\square$

Fig. 1

Diagrammatic representations of the simplified metabolic network and corresponding EFMs described in Example 1. Arrowheads indicate directionality. Reactions labelled $v_0,v_2,v_3,v_4$ have unit stoichiometry while that labelled $v_1$ has stoichiometry 2. $G_{ex}$, O, $P_1$, and $P_2$ are extracellular (slow) metabolites whereas $G_{in}$ and P are treated as intracellular (fast) metabolites

3 Maximum entropy control

This section considers the nature of $J^{red}$ in the reduced optimal control problem (10). Related to the separation of timescales for metabolite concentrations in the QSSA arises a similar separation of timescales for control variables. Fast regulatory control variables appearing in $f_i$ are encapsulated within the $r_k(\mathbf {m}_{ex})$, whereas the reduced system (9) has a linear dependence on slow control variables $u_k$. In what follows, it will be assumed that fast control variables are selected instantaneously (relative to the QSSA) to yield optimal values of $r_k(\mathbf {m}_{ex})$. On what basis optimality is defined for fast control variables is not of concern, but should derive from biologically reasonable principles. For example, instantaneous maximisation of the composite flux through each EFM amounts to a QSSA-based approximation of the optimal control policy in Young et al. (2008) where no separation of timescales was assumed. In this approach, a local objective for each EFM is used to determine optimal values for fast control variables that maximise the composite flux of each EFM individually, and subsequently the slower control variables are chosen to maximise a global objective combining all EFMs. Regardless of the instantaneous policy for selecting fast control variables, the optimal control problem (10) is stated so as to determine the $u_k$ assuming the $f_i$ in $r_k(\mathbf {m}_{ex})$ are given.

Combining a metabolic performance index $J^{red}$ that is linear in the $u_k$ with the constraint (7) would result in an optimal control law that allocates the entire fraction of resource exclusively to the EFM with highest return-on-investment (Wortel et al. 2014; Müller et al. 2014). Such a control is the so-called FBA or Bang–Bang policy, which for a variety of evolutionary reasons does not appear to be the most robust nor economically efficient resource allocation strategy in the face of environmental fluctuations (Young and Ramkrishna 2007; Solopova et al. 2014; Ackermann 2015; Martins and Locke 2015; Granados et al. 2017; De Martino et al. 2017; Fernandez-de-Cossio-Diaz and Mulet 2019). This motivates the revised concept (Young and Ramkrishna 2007; Lindhorst et al. 2018) that regulatory decisions for the control variables $u_k$ should be made based on the projected system response over a (short) time interval of length $\Delta t$. In this sense optimal choices for the $u_k$ are anticipatory of the effects that slower regulatory processes such as transcription and translation will have in the immediate future. Collecting dynamical and control variables into vectors $\mathbf {X}=(\mathbf {m}_{ex},x)^T$ and $\mathbf {u} = (u_1,u_2,\ldots ,u_K)^T$, respectively, and writing $\dot{\mathbf {X}} = \mathbf {F}(\mathbf {X},\mathbf {u})$, the linearisation of (9) about the state $\mathbf {X}(t)$ and a reference control input $\mathbf {u}^0$ may be assumed a good approximation to the system response at time $t + \tau$ for $\tau \in [0,\Delta t]$ (Young and Ramkrishna 2007). Linearisation yields

$$\begin{aligned} \frac{d}{d\tau } \Delta \mathbf {X} = \mathbf {F}(\mathbf {X}(t),\mathbf {u}^0) + \mathbf {A} \Delta \mathbf {X} + \mathbf {B} \Delta \mathbf {u} \end{aligned}$$

(11)

where

$$\begin{aligned} \mathbf {A} = \frac{\partial }{\partial \mathbf {X}} \mathbf {F}(\mathbf {X}(t),\mathbf {u}^0), \quad \mathbf {B} = \frac{\partial }{\partial \mathbf {u}}\mathbf {F}(\mathbf {X}(t),\mathbf {u}^0) \end{aligned}$$

and $\Delta \mathbf {X}(\tau ) = \mathbf {X}(t+\tau ) - \mathbf {X}(t)$, $\Delta \mathbf {u}(\tau ) = \mathbf {u}(t+\tau ) - \mathbf {u}^0$. When linearising and considering the change in performance index, Young and Ramkrishna (2007) augmented the accrued benefit derived across during the planning window $[t,t+\Delta t]$ by a term quadratic in the $u_k$ representing the cost or penalty associated with resource allocation. This paper takes a different approach, which is to model the change in performance over the time interval as

$$\begin{aligned} \Delta J = \mathbf {q}^T \Delta \mathbf {X}(t+\Delta t) + \sigma \int _t^{t+\Delta t} H(\mathbf {u}) d \tau , \end{aligned}$$

(12)

where

$$\begin{aligned} \Delta J = J^{red}(t+\Delta t) - J^{red}(t) , \quad \mathbf {q} = \frac{\partial \phi }{\partial \mathbf {X}} (\mathbf {X}(t)) , \end{aligned}$$

and

$$\begin{aligned} H(\mathbf {u}) = - \sum _{k=1}^K u_k \log (u_k) . \end{aligned}$$

(13)

Here the function $\phi (\mathbf {X})$ represents the metabolic objective of the system and $\sigma$ is a positive parameter that will be interpreted below.

The above choice of $H(\mathbf {u})$ is based on using the maximum entropy principle as a guide for selecting control variables $u_k$, which can be rationalised from several different perspectives: first, since the dynamical model (9) is stated in terms of total catalytic biomass of a population, maximum entropy has recently been proposed as an extension of FBA that is intended to capture heterogeneity of different allocation policies adopted by individuals within it (De Martino et al. 2017; Fernandez-de-Cossio-Diaz and Mulet 2019). Sources of this heterogeneity include stochasticity in gene expression and phenotype-switching at the single-cell level (Campbell et al. 2018), which can serve a functional purpose rather than simply reflecting noise tolerance, and provide a collective advantage to organisms living in fluctuating environments (Levy et al. 2012; Solopova et al. 2014; Ackermann 2015; Martins and Locke 2015; Granados et al. 2017). Second, by analogy with decision making problems in finance (Buchen and Kelly 1996), ecology (Harte and Newman 2014), and communication theory (Johansson and Sternad 2005), distribution of resources according to the principle of maximum entropy is the best choice for maximising expected return-on-investment in the face of uncertainty. As described in the introduction, the maximum entropy principle mathematically captures this notion of bet-hedging because it yields a unique resource allocation strategy consistent with known constraints (e.g., expected return-on-investment given current environmental conditions) while capturing maximum uncertainty in everything else (e.g., future environmental fluctuations) (Jaynes 1957; Shore and Johnson 1980). Indeed, this second point is intimately tied to the first because population heterogeneity is thought to be one way that cell populations have evolved to execute bet-hedging strategies (Levy et al. 2012; Solopova et al. 2014; Ackermann 2015; Beaumont et al. 2009), where both the entropy of the environment (Kussell and Leibler 2005) and gene expression profiles (Ridden et al. 2015) are taken into consideration. Finally, flux decomposition using maximum entropy-weighted EFMs has already been suggested for experimental flux derivation (Zhao and Kurata 2009, 2010), or where there is a direct physical interpretation for entropy as that of a chemical reaction (Srienc and Unrean 2010; Unrean and Srienc 2011). The former approach uses the maximum entropy principle as it directly applies to model inference (Jaynes 1957; Shore and Johnson 1980), where uncertainty reflects incompleteness of experimental data and the best statistical model is the one most consistent with those observed. Correspondence of the information-theoretic maximum entropy principle considered here with the physicochemical maximum entropy principle in Srienc and Unrean (2010) and Unrean and Srienc (2011) are beyond the scope of this paper, but form a deeper relationship between information theory, statistical mechanics, and thermodynamics (Jaynes 1957).

It is reasonable to assume that biological systems evolve under selection for maximal fitness by exploiting the capability to fully utilise their resource (the same assumption is made in Young and Ramkrishna (2007)). This implies the total summation constraint in (7) is satisfied as an exact equality and, because of the remaining non-negativity constraints, the vector $\mathbf {u}$ of resource fractions now can be interpreted as a discrete probability distribution across the EFMs. Applying Pontryagin’s maximum principle to the optimal control problem

$$\begin{aligned} \begin{aligned}&\text{ max } \Delta J \\&\text{ s.t. } (11) \text{ and } \sum _{k=1}^K u_k = 1, \quad u_k \ge 0 \quad k=1,2,\ldots ,K \end{aligned} \end{aligned}$$

(14)

and setting $\tau = 0$ as explained in “Appendix A”, results in the following alternative to the optimal control provided in Young and Ramkrishna (2007):

$$\begin{aligned} u_k(t) = \frac{1}{Q}\exp \left( \frac{1}{\sigma } \mathbf {q}^T \mathbf {e}^{\mathbf {A} \Delta t} \mathbf {B}^k\right) . \end{aligned}$$

(15)

Here $\mathbf {B}^k$ denotes the kth column of $\mathbf {B}$, $\mathbf {e}^{\mathbf {A} \Delta t}$ is the matrix exponential of $\mathbf {A} \Delta t$, and the normalisation factor Q is the partition function

$$\begin{aligned} Q = \sum _{k=1}^K \exp \left( \frac{1}{\sigma } \mathbf {q}^T \mathbf {e}^{\mathbf {A} \Delta t} \mathbf {B}^k \right) . \end{aligned}$$

As described by Jaynes (1957), the control (15) is the Boltzmann distribution with $\sigma$ taking the place of temperature and effective return-on-investment

$$\begin{aligned} \mathcal {R}^k_{\Delta t} = \mathbf {q}^T \mathbf {e}^{\mathbf {A} \Delta t} \mathbf {B}^k \end{aligned}$$

(16)

for the kth EFM taking the place of energy. Use of the adjective ‘effective’ will become clear shortly. In the limit $\sigma \rightarrow 0$, the control (15) collapses to the Bang–Bang/FBA policy (Wortel et al. 2014; Müller et al. 2014) where all resource is allocated to the EFM with the greatest effective return-on-investment (16) (although this does imply the $u_k$ can change rapidly, whereas formally they should be treated as slow control variables). Conversely, $u_k \rightarrow 1/K$$(\forall k = 1,2,\ldots , K)$ as $\sigma$ grows so that resource is partitioned equally among all EFMs in the limit $\sigma \rightarrow \infty$. This indifferent distribution of resource among EFMs is equivalent to the unregulated macroscopic bioreaction models of Provost and Bastin (Provost and Bastin 2004; Provost et al. 2006). Clearly neither extreme is necessarily an ideal representation of the optimal regulatory process, and therefore $\sigma > 0$ is taken to be finite so that the resource is allocated amongst EFMs according to their effective return-on-investment (larger getting more). What proportional majority of resource is awarded to the EFM with greatest effective return-on-investment is determined by the precise value of $\sigma$, which is considered to be a parameter fine-tuned over the course of evolution.

The general definition of effective return-on-investment (16) depends on a specific choice of metabolic objective that throughout the remainder of this paper is assumed to be maximisation of total catalytic biomass, i.e. $\phi (\mathbf {X}) = x$, which results in $\mathbf {q}=(\mathbf {0},1)^T$. The vector $\mathbf {B}^k$ is obtained by evaluating the derivative of $\mathbf {F}$ with respect to $u_k$ at $\mathbf {X}(t)$, and since $\mathbf {F}$ is linear in $u_k$ this choice of $\phi$ results in

$$\begin{aligned} \mathbf {B}^k = x r_k(\mathbf {m}_{ex}) \begin{pmatrix} \mathbf {S}_{ex} \\ \quad \mathbf {c}^T \end{pmatrix} \mathbf {Z}^k . \end{aligned}$$

(17)

In the first instance it is assumed that only immediate consequences of the injected control actions need to be considered when evaluating $\mathbf {u}$, and therefore $\Delta t = 0$ (the next section will consider non-zero choices of $\Delta t$ that involve additional complexity due the matrix exponential of $\mathbf {A} \Delta t$). By analogy with Young and Ramkrishna (2007), when $\Delta t = 0$, the control law (15) will be termed the greedy maximum entropy control. This simplifying assumption, that the Jacobian matrix $\mathbf {A}$ does not appear in the effective return-on-investment (16), is mathematically equivalent to the biological statement that future changes in the environment are not taken into consideration when making regulatory decisions. As described in Sect. 4, higher order corrections to the effective return-on-investment could be accounted for by a biological mechanism that has evolved to anticipate such environmental changes, but with $\Delta t = 0$ the greedy maximum entropy control serves to maximise expected return-on-investment given the current state of the environment but complete uncertainty about the future. Using the greedy maximum entropy control, the effective return-on-investment for the kth EFM reduces to

$$\begin{aligned} \mathcal {R}^k_{0}(\mathbf {m}_{ex}) = xr_k(\mathbf {m}_{ex}) \mathbf {c}^T \mathbf {Z}^k \equiv x \text{ R }_0^k(\mathbf {m}_{ex}) \end{aligned}$$

(18)

where notation $\text{ R }_0^k(\mathbf {m}_{ex})$ has been introduced for the return-on-investment evaluated at zeroth order ($\Delta t = 0$). Multiplication of $\text{ R }_0^k(\mathbf {m}_{ex})$ by x gives the greedy effective return-on-investment$\mathcal {R}^k_{0}(\mathbf {m}_{ex})$. Just as in the case of system (9), zeroth-order return-on-investment $\text{ R }_0^k(\mathbf {m}_{ex})$ and the corresponding optimal control remain invariant to re-scaling of $\mathbf {Z}^k$ because this is cancelled by the same factor appearing in the composite flux $r_k$ (8). The greedy effective return-on-investment (18) for the kth EFM is therefore proportional to a weighted harmonic mean of the $f_i(\mathbf {m}_{ex})$ multiplied by a weighted arithmetic mean of the $c_i$. The weighting for the kth EFM is provided by the N components $Z^k_i$ and the conclusion is that the greatest proportion of resource is allocated to the EFM for which the product of these two means is the largest.

In contrast to the resource allocation rules obtained by Young and Ramkrishna (2007), observe that the greedy maximum entropy control law (15) implies all EFMs, including those with with zero or negative zeroth-order return-on-investment, will be allocated a non-zero fraction of resource provided $x/\sigma$ remains finite. Spreading of resource between multiple pathways is known to be optimal for dealing with uncertainty in a non-deterministic environment (Solopova et al. 2014; Ackermann 2015; Martins and Locke 2015; Granados et al. 2017; Kussell and Leibler 2005; De Martino et al. 2017; Fernandez-de-Cossio-Diaz and Mulet 2019), and investing in each EFM is a bet-hedging strategy analogous to those in behavioural economics (Hansen and Sargent 2001; Sims 2003) that captures remaining uncertainty when it is not possible to anticipate future environmental conditions. Equipped only with knowledge about the current environment, allocating a small fraction of resource (e.g. fraction of the proteome) to EFMs not contributing directly to growth is not considered wasteful because there is always a small probability that one of these pathways will have a benefit in the future (O’Brien et al. 2016). Higher order corrections to return-on-investment will be described in Sect. 4, but without additional information the greedy maximum entropy control law spreads the remaining fraction of resource indiscriminately between EFMs with zero zeroth-order return-on-investment. The remaining resource fraction will tend to be very small when a majority of resource is heavily concentrated on EFMs having large return-on-investment (that are relatively more likely to be of benefit), combined with the appearance of total catalytic biomass x as an overall scaling factor in (18). As x increases, it plays an opposing role to $\sigma$ in the control law (15), meaning the distribution of resources amongst EFMs will become more heavily concentrated on those yielding the greatest return-on-investment (i.e., optimal resource allocation approaches the Bang–Bang/FBA policy as $x \rightarrow \infty$ with $\sigma$ fixed). This observation aligns well with the suggestion that the maximum entropy distribution represents the cumulative behaviour of individuals within a (finite) population (De Martino et al. 2017; Fernandez-de-Cossio-Diaz and Mulet 2019), since the spread of the population distribution will tend to decrease as the number of individuals within it increases.

Example 2

The greedy effective return-on-investments for the three EFMs in Example 1 are given by

$$\begin{aligned} \mathcal {R}_0^1(G_{ex})= & {} xc_1 r_1(G_{ex}) \\ \mathcal {R}_0^2(O,P_1)= & {} xc_3 r_2(O,P_1) \\ \mathcal {R}_0^3(G_{ex},O)= & {} x(c_1 + 2c_3) r_3(G_{ex},O) . \end{aligned}$$

For illustrative purposes, assume that $r_k(\mathbf {m}_{ex}) \rightarrow 1$ ($k=1,2,3$) as all extracellular metabolite concentrations become saturating. This implies that when $G_{ex}$, O, and $P_1$ are very large the greedy maximum entropy control law gives

$$\begin{aligned} u_1 \approx \frac{1}{Q}\exp (xc_1/\sigma ) , \quad u_2 \approx \frac{1}{Q}\exp (xc_3/\sigma ), \quad u_3 \approx \frac{1}{Q}\exp (x(c_1+2c_3)/\sigma ) \end{aligned}$$

where $Q = e^{xc_1/\sigma } + e^{xc_3/\sigma } + e^{x(c_1+2c_3)/\sigma }$. Since $c_3 > c_1$, in this case the EFM represented by $\mathbf {Z}^3$ receives the greatest fraction of resource, followed by that represented by $\mathbf {Z}^2$, and finally the EFM represented by $\mathbf {Z}^1$ receives the smallest fraction. On the other hand, if O becomes very small while $G_{ex}$ and $P_1$ remain saturating, i.e., oxygen concentrations become limiting, instead $\mathcal {R}_0^1>> \mathcal {R}_0^2 \approx \mathcal {R}_0^3$ and in this case the majority of resource is allocated to the EFM represented by $\mathbf {Z}^1$. Conversely, if glucose concentrations $G_{ex}$ become limiting while O and $P_1$ remain saturating, this results in $\mathcal {R}_0^2>> \mathcal {R}_0^1 \approx \mathcal {R}_0^3$ and the majority of resource is allocated to the EFM represented by $\mathbf {Z}^2$. $\square$

4 Metabolite yields and anticipatory regulation

In previous sections, $\mathbf {m}_{ex}$ was used to denote the concentrations of extracellular metabolites, assuming that all intracellular metabolites are considered fast and therefore approximately constant at any instantaneous moment in time by the QSSA. This neglected the possibility that certain intracellular metabolites may not satisfy the QSSA criteria and instead vary on the slow timescale associated with $\mathbf {m}_{ex}$ and x. Examples of slowly varying intracellular metabolites include storage compounds (see recent work Rügen et al. 2015; Reimers et al. 2017; Tajparast and Frigon 2018), which are suggested to increase growth rate across a time interval that includes several diverse environmental extremes, e.g., a 24h day-night epoch or feast-famine cycle. Such rationale may explain the regulation of storage pathways in organisms found in environments with predictable dynamics, but fails to describe the general patterns of accumulation and utilisation outside of this regime. As a relevant example, consider the case of intracellular carbohydrate reserves (Holme 1957; Preiss 1989; Lille and Pringle 1980; Françios and Parrou 2001). The observed accumulation of intracellular carbohydrates in response to nutrient limitation is not intuitively rationalised based on choosing a metabolic objective of maximising total catalytic biomass alone, because investing resources in any process not contributing directly to growth would be considered a sub-optimal control policy. For this reason, among others, authors have considered alternative metabolic objectives, such as maximising total carbon uptake, or have explicitly included intracellular reserves as an integral component of biomass (Feist and Palsson 2010; Waldherr and Lindhorst 2017; Lakshmanan et al. 2019). However, here it is demonstrated that no further assumption beyond $\phi (\mathbf {X}) = x$ (the metabolic objective of maximising total catalytic biomass described in Sect. 3) is necessary for explaining the accumulation of storage compounds in response to nutrient limitation. The discussion also involves evaluating the optimal control law (15) with $\Delta t > 0$, which by analogy with Young and Ramkrishna (2007) is called the temporal maximum entropy control.

To make the exposition more concrete, it will be useful to distinguish between two types of EFMs as suggested in Sect. 3: those that contribute directly to growth, such that $\mathbf {c}^T \mathbf {Z}^k > 0$; and those that do not, such that $\mathbf {c}^T \mathbf {Z}^k = 0$. The case $\mathbf {c}^T \mathbf {Z}^k < 0$ is excluded from consideration, but this is not a particularly restrictive assumption because in the vast majority of models all $c_i$ will be non-negative as are, necessarily, all vector components $Z^k_i$. Typical control laws based on the choice $\phi (\mathbf {X}) = x$, like the FBA/Bang–Bang policy and the greedy control law of Young and Ramkrishna (2007), preclude the allocation of resources to EFMs with $\mathbf {c}^T \mathbf {Z}^k = 0$ since then $\mathcal {R}_0^k(\mathbf {m}_{ex}) = 0$ also. These control policies therefore neglect possible benefits of allocating resources to EFMs contributing to processes other than growth directly, such as accumulation of storage compounds that may be utilised for growth should environmental conditions become unfavourable. In Sect. 3, it was shown that the greedy maximum entropy control allocates a fraction of resource to every EFM, including those with $\mathbf {c}^T \mathbf {Z}^k = 0$, which accounts for maximal uncertainty when only information about the current environment is available. Correspondingly, the fraction of resource allocated to EFMs with $\mathbf {c}^T \mathbf {Z}^k = 0$ will tend to increase as the average effective return-on-investment of EFMs with $\mathbf {c}^T \mathbf {Z}^k > 0$ decreases. On the other hand, the Jacobian matrix $\mathbf {A}$ appears in the effective return-on-investment (16) of the temporal maximum entropy control, which is equivalent to the biological statement that regulatory decisions also take into consideration effects that the control action will have on the environment in the immediate future. If, for example, a system has evolved to anticipate that formation of storage compounds provides a future opportunity to increase total catalytic biomass, this further reduction in uncertainty is accommodated into the temporal maximum entropy control. One consequence is that individual EFMs not contributing directly to growth can receive greater (or less) investment should they involve consumption or production of metabolites that make the future environment more (or less) favourable for growth. This is most clearly demonstrated by understanding the higher order corrections to return-on-investment that arise for small $\Delta t > 0$.

When $\Delta t$ is small, the matrix exponential $\mathbf {e}^{A\Delta t}$ can be approximated to first order so that the effective return-on-investment (16) becomes $\mathcal {R}^k_{\Delta t}(\mathbf {m}_{s}) \approx x[R_0^k(\mathbf {m}_s) + \Delta t R_1^k(\mathbf {m}_s)]$, where the first-order correction to return-on-investment derived in “Appendix B” is

$$\begin{aligned} R^k_1(\mathbf {m}_{s}) = \bar{R}_0(\mathbf {m}_{s}) R^k_0(\mathbf {m}_{s}) + x r_k(\mathbf {m}_{s}) \left( \frac{\partial \bar{R}_0}{\partial \mathbf {m}_{s}} (\mathbf {m}_{s}) \right) ^T \mathbf {S}_s \mathbf {Z}^k . \end{aligned}$$

(19)

Here $\mathbf {m}_{s}$ is used to denote all slow metabolite concentrations with corresponding stoichiometric matrix $\mathbf {S}_s$ and

$$\begin{aligned} \bar{R}_0(\mathbf {m}_{s}) = \sum _{k=1}^K R^k_0(\mathbf {m}_{s}) u^0_k \end{aligned}$$

is an average of the zeroth-order return-on-investment (i.e., the average contribution to growth rate) provided by components of the reference control $\mathbf {u}^0$ at time t. Observe that when the reference control is taken to be the uniform one (as suggested by Young and Ramkrishna 2007), corresponding to the $\sigma \rightarrow \infty$ limit of the maximum entropy control, then $\bar{R}_0(\mathbf {m}_{s})$ is simply the arithmetic mean of the $R^k_0(\mathbf {m}_{s})$. There are two terms in the first-order correction to return-on-investment (19): the first is the product $\bar{R}_0(\mathbf {m}_{s}) R^k_0(\mathbf {m}_{s})$, which is always non-negative and obviously large when both $\bar{R}_0(\mathbf {m}_{s})$ and $R^k_0(\mathbf {m}_{s})$ are large; the second term is

$$\begin{aligned} x r_k(\mathbf {m}_{s}) \left( \frac{\partial \bar{R}_0}{\partial \mathbf {m}_{s}} (\mathbf {m}_{s}) \right) ^T \mathbf {S}_s \mathbf {Z}^k \equiv x Y_k(\mathbf {m}_{s}) . \end{aligned}$$

(20)

To understand the newly defined quantity $Y_k(\mathbf {m}_{s})$, note that for any slow metabolite concentration m one has

$$\begin{aligned} \frac{\partial R^k_0}{\partial m}(\mathbf {m}_s) = \mathbf {c}^T\mathbf {Z}^k \frac{ \partial r_k}{\partial m}(\mathbf {m}_s) . \end{aligned}$$

(21)

Vector components of the form (21), one for each slow metabolite, provide a measure of how dependent the average contribution to growth rate is on concentration m at time t. If $r_k$ is monotonically increasing (which is true if all $f_i$ are monotonically increasing) then each component is non-negative and a large value of (21) indicates that a change in m leads to a relatively large increase of $\bar{R}_0(\mathbf {m}_{s})$, i.e., m is growth-limiting at time t; conversely, a value close to zero indicates a change in the concentration of that metabolite has a negligible effect, i.e., m is not growth-limiting at time t. In general, it will not always be true that the $f_i$ are monotonically increasing, in which case some of the components (21) can be negative indicating certain slow metabolite concentrations may be growth-prohibiting. In either case, values (21) serve to weight components of the vector $\mathbf {S}_s \mathbf {Z}^k$, which can be either positive or negative because they provide the yield of each metabolite for the kth EFM. A positive yield indicates the kth EFM will contribute to the production of a metabolite, while a negative yield means the EFM will contribute to its consumption. $Y_k(\mathbf {m}_{s})$ as defined in (20) is therefore interpreted as the total metabolite yield for the kth EFM, with weighting of each individual metabolite yield proportional to the relative ability of the corresponding metabolite to increase $\bar{R}_0(\mathbf {m}_{s})$. The relative sizes of $\bar{R}_0(\mathbf {m}_{s}) R^k_0(\mathbf {m}_{s})$ and $xY_k(\mathbf {m}_{s})$ determine whether the first-order correction $R^k_1(\mathbf {m}_{s})$ is positive or negative. A positive first-order correction to the return-on-investment on implies $\mathcal {R}^k_{\Delta t}(\mathbf {m}_{s}) > \mathcal {R}^k_0(\mathbf {m}_{s})$ whereas a negative correction means that $\mathcal {R}^k_{\Delta t}(\mathbf {m}_{s}) < \mathcal {R}^k_0(\mathbf {m}_{s})$.

Consequences of using the temporal maximum entropy control can then be summarised as follows: when no slow metabolites are growth-limiting or growth-prohibiting, the average contribution to growth rate $\bar{R}_0(\mathbf {m}_s)$ is large relative to magnitudes of the $xY_k(\mathbf {m}_s)$, and therefore resource becomes further concentrated on EFMs with $\mathbf {c}^T \mathbf {Z}^k>0$. However, when one or more slow metabolite is growth-limiting, and consequently the average contribution to growth rate $\bar{R}_0(\mathbf {m}_s)$ is low, EFMs with non-negative total metabolite yield $Y_k(\mathbf {m}_s)$ (such as those with $\mathbf {c}^T\mathbf {Z}^k = 0$) can be allocated a larger fraction of resource than in cases where $\bar{R}_0(\mathbf {m}_s)$ is high. This type of behaviour has been observed in most microbial populations (Holme 1957; Preiss 1989; Lille and Pringle 1980; Françios and Parrou 2001). For example, the storage carbohydrate glycogen is produced by yeast upon limitations in extracellular carbon or nitrogen, and in bacteria glycogen accumulates under conditions of limiting growth when carbon is in excess but other nutrients are deficient (see Wilson et al. (2010) for a review). Also in yeast, up-regulation of trehaolse synthesis is known to serve as an indicator for cell populations with lower growth rates (Levy et al. 2012), which has been rationalised using a bet-hedging argument. The greedy maximum entropy control generates such an inverse correlation between average contribution to growth rate and the levels of activation of EFMs with $\mathbf {c}^T\mathbf {Z}^k = 0$, but only the temporal maximum entropy control distinguishes between them based on total metabolite yields and their ability to shape environmental conditions. In conclusion, both maximum entropy control laws account for accumulation of intracellular reserves under growth-limiting conditions without imposing any assumption on the objective other than maximisation of total catalytic biomass. However, the temporal maximum entropy control law describes some form of anticipatory regulation, whereas the greedy maximum entropy control law accommodates maximal uncertainty if only current environmental conditions are known.

Fig. 2

Extension of the metabolic network described in Example 1 to include a storage compound C treated as an additional slow metabolite. The new reactions labelled $v_5,v_6$ both have unit stoichiometry and give rise to three additional EFMs as described in Example 3

Example 3

Consider the simplified metabolic network in Fig. 2a as an extension of the one introduced in Example 1. The storage compound with concentration C is introduced as a slow intracellular metabolite. Its consumption and production imply the addition of two reactions to the network with fluxes $v_5$ and $v_6$, respectively, which do not to contribute directly to growth (i.e., $c_5 = c_6 = 0$). The reduced system from Example 1 is extended to include an additional dynamical term for the new slow variable

$$\begin{aligned} \frac{dC}{dt} = (v_6 - v_5) - \mu C \end{aligned}$$

and the algebraic equations arising from the QSSA are modified to

$$\begin{aligned} 0= & {} v_0 -v_1 + v_5 - v_6\\ 0= & {} 2v_1 -v_2 -v_3 +v_4 . \end{aligned}$$

A complete set of six EFMs (represented graphically in Fig. 2b, c) is provided by the vectors

$$\begin{aligned}&\mathbf {Z}^1 = \begin{pmatrix} 1 \\ 1 \\ 2 \\ 0 \\ 0 \\ 0 \\ 0 \end{pmatrix} , \mathbf {Z}^2 = \begin{pmatrix} 0 \\ 0 \\ 0 \\ 1 \\ 1 \\ 0 \\ 0 \end{pmatrix} , \mathbf {Z}^3 = \begin{pmatrix} 1 \\ 1 \\ 0 \\ 2 \\ 0 \\ 0 \\ 0 \end{pmatrix} , \mathbf {Z}^4 = \begin{pmatrix} 0 \\ 1 \\ 2 \\ 0 \\ 0 \\ 1 \\ 0 \end{pmatrix} ,\nonumber \\&\quad \mathbf {Z}^5 = \begin{pmatrix} 0 \\ 1 \\ 0 \\ 2 \\ 0 \\ 1 \\ 0 \end{pmatrix} , \mathbf {Z}^6 = \begin{pmatrix}1 \\ 0 \\ 0 \\ 0 \\ 0 \\ 0 \\ 1 \end{pmatrix} . \end{aligned}$$

The composite EFM fluxes $r_1,r_2,r_3$ and greedy effective return-on-investments $\mathcal {R}_0^1,\mathcal {R}_0^2,\mathcal {R}_0^3$ are those given in Examples 1 and 2, respectively, while

$$\begin{aligned} r_4(C)= & {} \left( \frac{1}{f_1(G^*_{in})} + \frac{2}{f_2(P^*)} + \frac{1}{f_5(C)} \right) ^{-1}\\ r_5(O,C)= & {} \left( \frac{1}{f_1(G^*_{in})} + \frac{2}{f_3(O, P^*)} + \frac{1}{f_5(C)} \right) ^{-1}\\ r_6(G_{ex})= & {} \left( \frac{1}{f_0(G_{ex})} + \frac{1}{f_6(G^*_{in})} \right) ^{-1} \end{aligned}$$

and

$$\begin{aligned} \mathcal {R}_0^4(C) = x c_1 r_4(C), \quad \mathcal {R}_0^5(O,C) = x(c_1 + 2c_3) r_5(O,C) , \quad \mathcal {R}_0^6 = 0 . \end{aligned}$$

Omitting explicit dependencies of the $r_k$ on slow metabolites to ease notation, the reduced system expressed in terms of EFMs and control variables $u_k$ is

$$\begin{aligned} \frac{dG_{ex}}{dt}= & {} -[ r_1u_1 + r_3u_3 + r_6u_6] x\\ \frac{dO}{dt}= & {} -[r_2u_2 + 2 r_3u_3 + 2 r_5u_5] x = - \frac{dP_2}{dt} \\ \frac{dP_1}{dt}= & {} [2r_1u_1 - r_2u_2 + 2r_4u_4]x \\ \frac{dC}{dt}= & {} r_6u_6 - r_4u_4 - r_5u_5 - \mu C \\ \frac{dx}{dt}= & {} [c_1(r_1u_1 + r_4u_4) + c_3 r_2u_2 + (c_1 + 2c_3) ( r_3u_3 + r_5u_5)]x. \end{aligned}$$

The metabolite yields for each EFM are supplied by the vectors

$$\begin{aligned} \mathbf {S}_s \mathbf {Z}^1= & {} \begin{pmatrix}-1 \\ 0 \\ 2 \\ 0 \end{pmatrix} , \quad \mathbf {S}_s \mathbf {Z}^2 = \begin{pmatrix}0 \\ -1 \\ -1 \\ 0 \end{pmatrix} , \quad \mathbf {S}_s \mathbf {Z}^3 = \begin{pmatrix} -1 \\ -2 \\ 0 \\ 0 \end{pmatrix} ,\\ \mathbf {S}_s \mathbf {Z}^4= & {} \begin{pmatrix}0 \\ 0 \\ 2 \\ -1 \end{pmatrix} , \quad \mathbf {S}_s \mathbf {Z}^5 = \begin{pmatrix}0 \\ -2 \\ 0 \\ -1 \end{pmatrix} , \quad \mathbf {S}_s \mathbf {Z}^6 = \begin{pmatrix}-1 \\ 0 \\ 0 \\ 1 \end{pmatrix} , \end{aligned}$$

and, assuming for simplicity that the oxygen concentration O is saturating at time t so that $\partial r_k(t) /\partial O =0$ ($k = 1,2,\ldots ,6$), the total metabolite yields evaluated using the uniform reference control $u^0_k = 1/6$ ($k = 1,2,\ldots ,6$) are

$$\begin{aligned} Y_1(t)= & {} - \frac{r_1}{6}\left[ c_1 \frac{\partial (r_1 + r_3)}{\partial G_{ex}} + 2c_3 \left( \frac{\partial r_3}{\partial G_{ex}} - \frac{\partial r_2}{\partial P_1} \right) \right] _t \\ Y_2(t)= & {} - \frac{r_2}{6}\left[ c_3 \frac{\partial r_2}{\partial P_1} \right] _t \\ Y_3(t)= & {} - \frac{r_3}{6}\left[ c_1 \frac{\partial (r_1 + r_3)}{\partial G_{ex}} + 2c_3 \frac{\partial r_3}{\partial G_{ex}} \right] _t \\ Y_4(t)= & {} - \frac{r_4}{6}\left[ c_1 \frac{\partial (r_4 + r_5) }{\partial C} + 2c_3 \left( \frac{\partial r_5}{\partial C} - \frac{\partial r_2}{\partial P_1} \right) \right] _t \\ Y_5(t)= & {} - \frac{r_5}{6}\left[ c_1 \frac{\partial (r_4+r_5)}{\partial C} + 2c_3 \frac{\partial r_5}{\partial C} \right] _t \\ Y_6(t)= & {} \frac{r_6}{6}\left[ c_1 \left( \frac{\partial r_4}{\partial C} - \frac{\partial r_1 }{\partial G_{ex}} \right) + (c_1 + 2c_3) \left( \frac{\partial r_5 }{\partial C} - \frac{\partial r_3}{\partial G_{ex}} \right) \right] _t \end{aligned}$$

where $[\cdot ]_t$ indicates the expression inside square parentheses is to be evaluated at t. Observe that in a regime where metabolite concentrations are such that

$$\begin{aligned} \frac{\partial r_2(t)}{\partial P_1}< \frac{\partial r_3(t)}{\partial G_{ex}}< \frac{\partial r_5(t)}{\partial C} \quad \text{ and } \quad \frac{\partial r_1(t)}{\partial G_{ex}} < \frac{\partial r_4(t)}{\partial C} , \end{aligned}$$

then $Y_k(t) < 0$ for $k = 1,2,\ldots ,5$ whereas $Y_6(t)>0$. In fact, when oxygen is not saturating it can be shown that the contributions from non-zero derivatives $\partial r_k(t) /\partial O$ ($k=2,3,5$) decrease $Y_2,Y_3,Y_5$ further while leaving $Y_1,Y_4,Y_6$ unchanged. $\square$

5 Model reduction using EFM families

This section explores the practical aspects of model design and simulation. Enumeration of EFMs for large stoichiometry matrices $\mathbf {S}$ can lead to a combinatorial explosion as their number grows with increasing network size and connectivity (Klamt and Stelling 2002). This necessitates inclusion of many undetermined parameters and control variables in the reduced system (9), which are difficult to model accurately should sufficient experimental data not be available. Consequently, previous attempts to reduce the complexity of dynamic models like (9) have introduced rules for selecting a subset of relevant EFMs, or grouping EFMs into families to be considered together (e.g. Song and Ramkrishna 2010, 2011; Vilkhovoy et al. 2016). An additional simplification is to approximate the composite EFM fluxes $r_k(\mathbf {m}_s)$ by Michaelis-Menten kinetics, such that

$$\begin{aligned} r_k(\mathbf {m}_s) = V^{max}_k \prod _a \frac{m_a}{\kappa _{a,k} + m_a} \end{aligned}$$

where $V^{max}_k$, $\kappa _{a,k}$ are constants and the product includes all slow metabolite concentrations $m_a$ whose uptake fluxes are in the support of the kth EFM. In what follows it is assumed that such an approximation (although not necessarily the Michaelis-Menten one) has been provided for the functional form of the $r_k(\mathbf {m}_s)$ and that vectors representing EFMs have therefore been normalised to a common scale, such as total uptake carbon content. As described in Sect. 2, choosing a common normalisation for the EFM representative vectors is essential when the $r_k$ are approximated in this way because then (9) is no longer invariant to $\mathbf {Z}^k$ re-scaling. The focus of this section is to understand the effect of further model reduction, by grouping EFMs into families, on resource allocation from the perspective of the maximum entropy control. Rules for composing EFM families are not the object of consideration here, but could involve, for example (Song and Ramkrishna 2010, 2011; Vilkhovoy et al. 2016), grouping together all EFMs whose support contain the same uptake flux.

Partitioning of EFMs into M families means partitioning indices $k=1,2,\ldots ,K$ into M mutually disjoint subsets $F_J$ (of size $N_J$) $J=1,2,\ldots ,M$, and partitioning total resource into M fractions $U_J$ such that

$$\begin{aligned} \sum _{j \in F_J} u_j = U_J , \forall J =1,2,\ldots ,M : \quad \sum _{J=1}^M U_J = 1 . \end{aligned}$$

Consequently, the EFM with index $j \in F_J$ is allocated a fraction $\tilde{u}_j = u_j/U_J$ of the resource $U_J$ available to the Jth family. These values are collected in vectors $\mathbf {U} = (U_1,U_2,\ldots ,U_M)^T$ and $\tilde{\mathbf {u}}_J = (\tilde{u}_{F_{J,1}},\tilde{u}_{F_{J,2}},\ldots ,\tilde{u}_{F_{J,N_J}})^T$, where $F_{J,i}$ denotes the ith element of $F_J$. Representative vectors $\tilde{\mathbf {Z}}^J$ ($J=1,2,\ldots ,M$) are formed as weighted combinations of EFMs and used to express the dynamical system (9) in terms of EFM families. Several different weightings have previously been considered (Song and Ramkrishna 2010, 2011; Vilkhovoy et al. 2016), but the entropy constraint identifies a particularly natural one to be

$$\begin{aligned} \tilde{r}_J(\mathbf {m}_{s})\tilde{\mathbf {Z}}^J = \frac{\sum _{j \in F_J} r_j(\mathbf {m}_{s}) \mathbf {Z}^ju_j }{\sum _{j \in F_J} u_j } = \sum _{j \in F_J} r_j(\mathbf {m}_{s}) \mathbf {Z}^j\tilde{u}_{F_{J,j}} \end{aligned}$$

(22)

where $\tilde{\mathbf {Z}}^J$ is defined given $\tilde{r}_J(\mathbf {m}_{s})$, $\tilde{\mathbf {u}}_J$, and the $r_j(\mathbf {m}_{s})$ ($j \in F_J$). To understand resource allocation in terms of this partitioning, observe the optimal control (15) is obtained by maximisation of the objective functional

$$\begin{aligned} \mathcal {F}(\mathbf {u}) = \sum _{k=1}^K \mathcal {R}^k_{\Delta t}(\mathbf {m}_{s}) u_k + \sigma H(\mathbf {u}) \end{aligned}$$

(23)

where the effective return-on-investment $\mathcal {R}^k_{\Delta t}(\mathbf {m}_{s})$ is defined in (16). A classical result (Jaynes 1957; Shore and Johnson 1980) is that entropy satisfies the composition property

$$\begin{aligned} H(\mathbf {u}) = H(\mathbf {U}) + \sum _{J =1}^M U_J H(\tilde{\mathbf {u}}_J) \end{aligned}$$

(24)

where H is defined as in (13) on components of each respective vector. As shown in “Appendix C”, combined with the weighting (22) this implies $\mathcal {F}(\mathbf {u})$ can be expressed as

$$\begin{aligned} \mathcal {F}(\mathbf {u}) = \sum _{J=1}^M \mathcal {F}_J(\tilde{\mathbf {u}}_J) U_J + \sigma H(\mathbf {U}) , \end{aligned}$$

(25)

where $\mathcal {F}_J$ is the restriction of $\mathcal {F}$ to EFMs in the Jth family.

The full resource allocation problem can be viewed as a two-stage process involving an initial distribution of resource across EFM families, followed by further partitioning of the $U_J$ among their constituent EFMs (Song and Ramkrishna 2010, 2011). This procedure is best captured by maximising $\mathcal {F}(\mathbf {u})$ in two steps: first, the $\mathcal {F}_J(\tilde{\mathbf {u}}_J)$ are maximised for each family, providing maximum entropy controls for EFMs in terms of $\tilde{\mathbf {u}}_J$. Next, the resulting $\tilde{\mathbf {u}}_J$ are used for the EFM weightings (22) and $\mathcal {F}(\mathbf {u})$ is maximised with respect to the $U_J$, yielding a maximum entropy control for EFM families. From a modelling perspective it is informative to consider the related objective functional

$$\begin{aligned} \mathcal {F}(\mathbf {U}) = \sum _{J=1}^M \mathcal {\tilde{R}}^J_{\Delta t}(\mathbf {m}_{s})U_J + \sigma H(\mathbf {U}) , \end{aligned}$$

(26)

where $\mathcal {\tilde{R}}^J_{\Delta t}(\mathbf {m}_{s})$ is the Jth family’s effective return-on-investment derived, as for individual EFMs in Sect. 3, directly from system (9) expressed in terms of EFM families. Maximisation of $\mathcal {F}(\mathbf {U})$ provides the maximal entropy control to be used if the constituents of EFM families are not known, and in this case $\mathcal {\tilde{R}}^J_{\Delta t}(\mathbf {m}_{s})$ should be a suitable approximation of $\mathcal {F}_J(\tilde{\mathbf {u}}_J)$. For reasons described previously however, even when an enumeration of EFMs exists one might want to simplify the calculation of optimal controls by approximating $\mathcal {F}_J(\tilde{\mathbf {u}}_J)$ using EFMs in a manner consistent with maximisation. One way to do this is to set $r_k=x=1$ in the greedy effective return-on-investment (18) and use the resulting (fixed) greedy maximum entropy controls

$$\begin{aligned} \tilde{u}_j = \frac{e^{\eta _j/\sigma }}{\sum _{l \in F_J} e^{\eta _l/\sigma }} \quad \forall j \in F_J , \quad J = 1,2,\ldots ,M \end{aligned}$$

(27)

with $\eta _j \equiv \mathbf {c}^T\mathbf {Z}^j$ to express the reduced dynamical system (9) in terms of EFM family vectors $\tilde{\mathbf {Z}}^J = \sum _{j \in F_j} \tilde{u}_j \mathbf {Z}^j$. The effective return-on-investment $\mathcal {\tilde{R}}^J_{\Delta t}(\mathbf {m}_{s})$ derived for the Jth family then takes the same form as (16), but with $B^k$ replaced by

$$\begin{aligned} \tilde{B}^J = x \tilde{r}_J(\mathbf {m}_{ex}) \begin{pmatrix} \mathbf {S}_{ex} \\ \mathbf {c}^T \end{pmatrix} \tilde{\mathbf {Z}}^J . \end{aligned}$$

(28)

Maximisation of the objective functional $\mathcal {F}(\mathbf {U})$ in (26) therefore provides the maximum entropy control for optimal allocation of resource among EFM families represented by these $\tilde{\mathbf {Z}}^J$. This approach is analogous to the EFM lumping proposed by Song and Ramkrishna (2010, 2011), where it is of relevance to note that their choice of EFM weighting also depends on a parameter $n_v$ that modulates spread across EFMs within a family, thus playing the equivalent of $\sigma$ in the maximal entropy control. In fact, their choice of fixed weighting $u_j = \eta _j^{n_v}$ is related to the fixed maximum entropy weighting $u_j \propto e^{\eta _j/\sigma }$ in the same way that maximum entropy relates to fuzzy clustering (Karayiannis 1994). An alternative method for approximating $\tilde{\mathbf {Z}}^J$, related to Vilkhovoy et al. (2016) and the maximum entropy control in the $\sigma \rightarrow 0$ limit, is to select the vector $\mathbf {Z}^j$ ($j\in F_J$) representing the EFM with the largest return-on-investment (i.e., the FBA/Bang–Bang solution) in the Jth family.

Although EFM families and various EFM weightings have been described previously (Song and Ramkrishna 2010, 2011; Vilkhovoy et al. 2016), the above discussion implicates their unification under the maximal entropy control framework. Stated in terms of a modelling endeavour to best approximate $\mathcal {F}_J(\tilde{\mathbf {u}}_J)$, choices for individual EFM return-on-investments and EFM family representatives can be derived from first principles. These approximations may involve any form of dynamic model reduction or simplification that respects the basic maximum entropy control laws for partitioning resources among EFMs within a family. Moreover, subsequent partitioning of resource among EFM families has always previously involved the original cybernetic control laws of Young and Ramkrishna (2007), whereas for consistency the maximum entropy control should once more be used to determine the $U_J$. Doing so makes it possible to recursively apply this form of model reduction when establishing the appropriate number of EFM families; the recursive nature of the maximum entropy control framework is succinctly captured by the objective functional (25).

Example 4

Consider the metabolic network (Fig. 2a) from Example 3, but now with C interpreted as the concentration of a fast intracellular metabolite. The QSSA on C imposes the additional algebraic constraint $v_5 - v_6 = 0$, which results in the collapse of one EFM (represented by vector $\mathbf {Z}^6$ in Example 3). A complete set of five EFMs for this network is therefore provided by the representative vectors

$$\begin{aligned} \mathbf {Z}^1 = \frac{1}{6} \begin{pmatrix} 1 \\ 1 \\ 2 \\ 0 \\ 0 \\ 0 \\ 0 \end{pmatrix} , \quad \mathbf {Z}^2 = \frac{1}{3} \begin{pmatrix}0 \\ 0 \\ 0 \\ 1 \\ 1 \\ 0 \\ 0 \end{pmatrix} , \quad \mathbf {Z}^3 = \frac{1}{6} \begin{pmatrix} 1 \\ 1 \\ 0 \\ 2 \\ 0 \\ 0 \\ 0 \end{pmatrix} , \quad \mathbf {Z}^4 = \frac{1}{6} \begin{pmatrix} 1 \\ 1 \\ 2 \\ 0 \\ 0 \\ 1 \\ 1 \end{pmatrix} , \quad \mathbf {Z}^5 = \frac{1}{6} \begin{pmatrix} 1 \\ 1 \\ 0 \\ 2 \\ 0 \\ 1 \\ 1 \end{pmatrix} , \end{aligned}$$

each normalised by their uptake carbon content. Partitioning of these EFMs into families on the basis of their shared products and substrates means $F_1 = \{1,4\}$, $F_2 = \{2\}$, and $F_3 = \{3,5\}$, which gives rise to the three EFM families represented in Fig. 3.

Fig. 3

Partitioning of the five EFMs from the metabolic network in Example 3 with C interpreted as a fast metabolite. The five EFMs are represented by vectors $\mathbf {Z}^k$ (dashed lines) and each of the three EFM families is represented by an individual vector $\tilde{\mathbf {Z}}^J$ (bold lines). In this case $\tilde{\mathbf {Z}}^2 = \mathbf {Z}^2$, while $\tilde{\mathbf {Z}}^1$ and $\tilde{\mathbf {Z}}^3$ are formed from linear combinations of the pairs $(\mathbf {Z}^1,\mathbf {Z}^4)$ and $(\mathbf {Z}^3,\mathbf {Z}^5)$, respectively, as described in Example 4

Setting $r_k=x=1$ ($k =1,2,3,4,5$) in (18) yields the fixed return-on-investments

$$\begin{aligned} \eta _1 = \eta _4 = \frac{1}{6}c_1, \quad \eta _2 = \frac{1}{3}c_3, \quad \eta _3 = \eta _5 = \frac{1}{6}(c_1 + 2c_3) \end{aligned}$$

and in this case the fixed greedy maximum entropy controls (27) for each family are independent of $\sigma$:

$$\begin{aligned} \tilde{u}_1 = \tilde{u}_4 = \frac{1}{2}, \quad \tilde{u}_2 = 1, \quad \tilde{u}_3 = \tilde{u}_5 = \frac{1}{2} . \end{aligned}$$

This results in the three EFM family representative vectors

$$\begin{aligned} \tilde{\mathbf {Z}}^1 = \frac{1}{2}( \mathbf {Z}^1 + \mathbf {Z}^4), \quad \tilde{\mathbf {Z}}^2 = \mathbf {Z}^2, \quad \tilde{\mathbf {Z}}^3 = \frac{1}{2} ( \mathbf {Z}^3 + \mathbf {Z}^5) \end{aligned}$$

and using a Michaelis–Menten approximation for the $\tilde{r}_J$ one has

$$\begin{aligned} \tilde{r}_1(G_{ex})= & {} \tilde{V}^{max}_1 \frac{G_{ex}}{\tilde{\kappa }_{G,1} + G_{ex}}\\ \tilde{r}_2(O,P_1)= & {} \tilde{V}^{max}_2 \frac{O}{\tilde{\kappa }_{O,2} + O} \frac{P_1}{\tilde{\kappa }_{P,2} + P_1} \\ \tilde{r}_3(G_{ex},O)= & {} \tilde{V}^{max}_3 \frac{G_{ex}}{\tilde{\kappa }_{G,3} + G_{ex}} \frac{O}{\tilde{\kappa }_{O,3} + O} . \end{aligned}$$

Substituting for these expressions in (28) provides $\tilde{B}^{J}$ to be used for calculating the effective return-on-investment of the Jth EFM family and maximum entropy controls $U_1,U_2,U_3$. Note that in this example, independence of $\sigma$ in the fixed greedy maximum entropy controls $\tilde{\mathbf {u}}_1$, $\tilde{\mathbf {u}}_2$, $\tilde{\mathbf {u}}_3$ implies they are equivalent to a set of FBA/Bang–Bang policies. Alternatively, one could assume some cost is associated with one of the reactions $v_5$ or $v_6$ (e.g., $c_5 < 0$), in which case $\eta _1 > \eta _4$ and $\eta _1 > \eta _5$ so that this equivalence would only hold in the $\sigma \rightarrow 0$ limit where $\tilde{\mathbf {Z}}^1 = \mathbf {Z}^1$ and $\tilde{\mathbf {Z}}^3 = \mathbf {Z}^3$. In both cases the reduced dynamic model can be mapped on to that described in Example 1. $\square$

6 Application to yeast metabolism

Here a dynamic model of resource allocation is introduced for central carbon metabolism in a single-celled eukaryotic organism, yeast, which builds upon the concepts and examples presented in previous sections. The metabolic network in Fig. 2a describes the participating reactions, and product 1 is ethanol (concentration E), with C the lumped concentration of storage carbohydrates glycogen and trehalose modelled in this case (contrast with Example 3) as an extracellular term that combines with total catalytic biomass x to represent total biomass $x +C$ in the system. Note that without storage carbohydrates, the simplified network (Fig. 1a) may be viewed as the reduction of a much larger network composed of glycolysis, the pentose-phosphate pathway, citric acid cycle, glyoxylate shunt, and oxidative phosphorylation, using EFM families exactly as was suggested by Song and Ramkrishna (2010). Introduction of the two reactions with fluxes $v_5,v_6$ follows this same principle by grouping all reserve carbohydrate pathways into a single representative family. Precise details of EFM family reduction have been overlooked but, because of its recursive nature as described in the previous section, the relevant EFM families can be represented by the six vectors $\mathbf {Z}^k$ from Example 2 normalised by their uptake carbon content. The full dynamical system expressed in terms of these (with the subscript dropped from $G_{ex}$) is

$$\begin{aligned} \frac{dG}{dt}= & {} D(G_0 - G) -\frac{1}{2}[ r_1u_1 + r_3u_3 + r_6u_6] x\\ \frac{dO}{dt}= & {} kLa(O^* - O) -[r_2u_2 + r_3u_3 + r_5u_5] x \\ \frac{dE}{dt}= & {} -DE+[r_1u_1 - r_2u_2 + r_4u_4]x \\ \frac{dC}{dt}= & {} -DC+ \frac{1}{2}[r_6u_6 - r_4u_4 - r_5u_5]x \\ \frac{dx}{dt}= & {} \left[ \frac{1}{2}c_1(r_1u_1 + r_4u_4) + c_3 r_2u_2 + \frac{1}{2}(c_1 + 2c_3) ( r_3u_3 + r_5u_5) - D\right] x \end{aligned}$$

where a dilution rate D and volumetric mass transfer coefficient $k_La$ have been introduced for simulation of cultures with inflow glucose concentration $G_0$ and dissolved oxygen solubility limit $O^*$.

The $r_k$ are approximated by Michaelis-Menten kinetics according to the metabolites whose uptake fluxes are in the support of each $\mathbf {Z}^k$, such that

$$\begin{aligned} r_1(G)= & {} V^{max}_1 \frac{G}{K_1 + G}, \quad r_2(O,E) = V^{max}_2 \frac{E}{K_2 + E}\frac{O}{K_{O,2} + O}\\ r_3(G,O)= & {} V^{max}_3 \frac{G}{K_3 + G}\frac{O}{K_{O,3} + O}, \quad r_4(C/x) = V^{max}_4 \frac{C/x}{K_4 + C/x}\\ r_5(O,C/x)= & {} V^{max}_5 \frac{C/x}{K_5 + C/x}\frac{O}{K_{O,5} + O}, \quad r_6(G) = V^{max}_6 \frac{G}{K_6 + G} . \end{aligned}$$

Observe that composite fluxes $r_4,r_5$ depend on the fractional concentration C/x of storage carbohydrate. The zeroth-order return-on-investments are

$$\begin{aligned} R^1_0(G)= & {} \frac{1}{2}c_1 r_1(G), \quad R^2_0(O,E) = c_3 r_2(O,E) ,\\ R^3_0(G,O)= & {} \frac{1}{2}(c_1 + 2c_3) r_3(G,O) , \quad R^4_0(C/x) = \frac{1}{2}c_1 r_4(C/x) ,\\ R^5_0(O,C/x)= & {} \frac{1}{2}(c_1 + 2c_3) r_5(O,C/x) , \quad R^6_0 = 0 \end{aligned}$$

and the greedy maximum entropy control law for determining each $u_k$ is then

$$\begin{aligned} u_k = \frac{\exp (xR^k_0(\mathbf {m},x)/\sigma )}{\sum _{l=1}^6 \exp (x R^l_0(\mathbf {m},x)/\sigma )} \end{aligned}$$

with $\mathbf {m} = (G,O,E,C)^T$ and $\sigma > 0$. Numerical simulations of this system were performed using custom-built software based on the SUNDIALS solvers (Hindmarsh et al. 2005) and the fixed parameter values listed in Table 1. Note that parameter values have been chosen generically in the sense that, although biologically realistic values have been used, there has been no attempt to fit these to experimental data or perform bifurcation analysis. The results of simulations should therefore be treated as qualitative and absolute concentration measurements used for quantitative predictions only after using biological knowledge or data to refine parameter values, which will almost certainly increase overall predictive power of the model. In particular, it is known that modifying parameters $V_1^{max}, V_2^{max}, V_3^{max}$ or introducing a cost for each pathway [i.e., weights for $u_1,u_2,u_3$ in the summation constraint from (14)] generates a static model of resource allocation for the network in Example 1 where either pure oxidation, pure fermentation, or a mixture of oxidation and fermentation, are optimal strategies for maximal ATP production [see Möller et al. (2018) and references therein]. This result is based on the proposal that overflow metabolism is a consequence of lower yield- higher rate pathways being preferred when certain environmental conditions are combined with the constraints of resource allocation (Basan et al. 2015). By contrast, simulations based on the parameter values in Table 1 are only intended to capture isolated effects of the maximum entropy control law without imposing any additional biological information.

Table 1 Values for parameters used in all simulations

Fig. 4

Results of numerical simulation for various values of $\sigma$ using the parameters reported in Table 1 with: $D=0.0$, $k_La = 30.0\,\text{ h }^{-1}$, $G_0 = 0\,\text{ g } \; \text{ L }^{-1}$ and initial conditions $x(0)=0.1$, $G(0) = 10.0$, $O(0) = 0.0001$, $E(0) = 0.0\,\text{ g } \; \text{ L }^{-1}$ and $C(0) = 0.0\,\text{ g } \; \text{ g }^{-1}\; \text{ L }^{-1}$ for batch culture; and $D = 0.1 - 0.135\,\text{ h }^{-1}$, $G_0 = 10.0\,\text{ g } \; \text{ L }^{-1}$, $k_La = 150.0\,\text{ h }^{-1}$ and initial conditions $x(0)=0.1$, $G(0) = 0.04$, $O(0) = 0.001$, $E(0) = 0.0\,\text{ g } \; \text{ L }^{-1}$ and $C(0) = 0.0\,\text{ g } \; \text{ g }^{-1}\; \text{ L }^{-1}$ for continuous culture (at steady state)

Figure 4a–c show batch culture simulations for increasing values of $\sigma$. When $\sigma$ is small (Fig. 4a), there is relatively little accumulation of ethanol or storage carbohydrate over the course of simulation because, for the given parameter values at the provided oxygen concentrations, the glucose oxidation pathway (EFM represented by $\mathbf {Z}^3$) continues to receive the majority of resource until glucose is depleted and growth halted. The fraction of resource allocated to glucose fermentation (EFM represented by $\mathbf {Z}^1$) only becomes comparable towards the end of the simulation. When $\sigma$ is increased to an intermediate value however (Fig. 4b), there is a large accumulation of extracellular ethanol, which in turn feeds in to the ethanol oxidation pathway (EFM represented by $\mathbf {Z}^2$) that receives the majority of resource during the latter half of the simulation. This transition from glucose metabolism to ethanol oxidation qualitatively captures the Crabtree effect observed in batch cultures of baker’s or brewer’s yeast Saccharomyces cerevisiae (Pfeiffer and Morley 2014), but note that this effect could have been reproduced at lower values of $\sigma$ should $V_1^{max}>> V_3^{max}$, $K_1>> K_3$, or the cost of the oxidative pathways (weights for $u_2,u_3$ in (14)) be increased in analogy with the model of overflow metabolism presented in Möller et al. (2018). Instead, for the parameter values reported here, ethanol accumulation and the subsequent transition between substrates is solely attributable to the maximum entropy control law, which allocates non-zero fractions of resource to EFMs with lower return-on-investment. This behaviour is indicative of a bet-hedging component of the Crabtree effect, possibly related to that observed experimentally in the case of the diauxic shift (Solopova et al. 2014). When $\sigma$ is increased further (Fig. 4c), there is a considerable accumulation of storage carbohydrate that leads to the pathways involved in its consumption (EFMs represented by $\mathbf {Z}^4$ and $\mathbf {Z}^5$) receiving the majority of resource for a brief window of time at the point of transition from glucose to ethanol. Accumulation of storage carbohydrate occurs because a greater fraction of resource is allocated to the EFM represented by $\mathbf {Z}^6$, even though this pathway does not contribute directly to growth. Closer inspection reveals a sharp increase in the rate of storage carbohydrate accumulation immediately prior to the onset of the transition from glucose to ethanol and its subsequent utilisation, which agrees very well with the observed dynamics of glycogen and trehalose metabolism in S. cerevisiae batch culture: these metabolites rapidly accumulate near the end of the growth phase on glucose and are quickly consumed when consumption of ethanol begins (Lille and Pringle 1980; Françios and Parrou 2001; Panek 1962, 1963; Parrou et al. 1999). Figure 4d plots steady state concentrations of extracellular ethanol against dilution rate for simulations of the model in continuous culture with $\sigma = 1.0$. As a general trend, increasing dilution rate leads to an increase in extracellular ethanol concentrations before reaching a maximum value and decreasing prior to wash out (when dilution rate exceeds growth rate). In the future, these continuous culture simulations could be compared with data reported in Van Hoek et al. (1998) for further refinement of parameter values.

For comparison with other models of yeast metabolism at this level of complexity, two closely related examples are the model derived using EFM lumping in Song and Ramkrishna (2010) and the model by Jones and Kompala (1999). Both are based on cybernetic laws originating from the greedy control of Young and Ramkrishna (2007), but unlike the maximum entropy model presented here have been enlarged to include auxiliary variables (called ‘cybernetic enzymes’) responsible for conferring the regulatory effects of control. Cybernetic enzymes may confer additional robustness to dynamic models, but also imply the existence of extra parameter values that are especially difficult to determine experimentally because they do not correspond to biological reality. One could argue that the EFM control variables $u_k$ do not correspond directly to biological entities either, but in this case they come without the overhead of additional uninterpretable parameters. Although based on EFM families, the model of Song and Ramkrishna (2010) does not include the dynamics of storage carbohydrates. In any case, their cybernetic control law would negate the allocation of resource to any EFMs involved in storage carbohydrate formation unless additional assumptions were imposed upon the model. Parameters used in their simulations were only partially informed by experimental data. On the other hand, the model of Jones and Kompala (1999) does include a term for storage carbohydrates, but the coupling of C to other variables in the model (structurally equivalent to that of Song and Ramkrishna) is based entirely on empirical observations (Lille and Pringle 1980; Françios and Parrou 2001; Panek 1962, 1963; Parrou et al. 1999) for the dynamics of carbohydrate accumulation and utilisation. Conversely, the description of this phenomenon presented here relies on nothing more than the maximum entropy principle. As a final note, parameter values in the Jones and Kompala model were obtained by direct fit to experimental data. Such an approach could be expected to improve the quantitate behaviour of the maximum entropy model and the lumped cybernetic model in Song and Ramkrishna (2010), both to be viewed as qualitatively predictive in nature.

7 Conclusion

Using the maximum entropy principle to extend the optimal control framework of Young and Ramkrishna (2007) provides a dynamic theory of metabolic resource allocation in the face of environmental uncertainty. This concept both generalises and unifies prior approaches by establishing a smooth interpolation between DFBA (Mahadevan et al. 2002) at one extreme, and dynamic metabolic models without regulation (Provost and Bastin 2004; Provost et al. 2006) at the other. In contrast to alternative optimal control laws, no assumption other than instantaneous maximisation of total catalytic biomass based on the maximum entropy principle is required to explain activation of pathways not contributing directly to growth rate, such as the formation of reserve compounds that have previously been explicitly included as an integral component of total biomass. From an evolutionary perspective this is a particularly appealing explanation for the observed accumulation of reserve compounds in growth-limiting conditions, because selection for maximal rates of self-replication extends beyond cellular biology to the RNA world (Orgel 2004). However, it likely that this form of bet-hedging constitutes just one component of the biological mechanism that governs dynamic regulation of metabolism in fluctuating environmental conditions.

Application of the dynamic maximum entropy framework to a simplified model of yeast metabolism has shown that the theory successfully reproduces some observed behaviour of cell populations in batch and continuous culture. This reduced model can almost surely be improved by including additional biological knowledge on the nature of overflow metabolism in yeast (Möller et al. 2018; Basan et al. 2015; Pfeiffer and Morley 2014), but also serves to illustrate contributions that come from bet-hedging alone. When considering cell populations, there are at least two (not mutually exclusive) theoretical interpretations for such bet-hedging mechanisms (Hansen and Sargent 2001; Sims 2003): that the maximum entropy distribution of resources is a result of heterogeneity in the regulatory FBA/Bang–Bang strategies of individuals (Ackermann 2015; Martins and Locke 2015; De Martino et al. 2017; Fernandez-de-Cossio-Diaz and Mulet 2019), or is chosen by each cell as the optimal strategy for dealing with uncertainty in the environment (Solopova et al. 2014; Granados et al. 2017). It is important to note that, if thermodynamic constraints are imposed, then only the FBA/Bang–Bang policy will be guaranteed to yield a thermodynamically-consistent, single-cell resource allocation strategy provided the set of EFMs is restricted to those that are thermodynamically-feasible (Jol et al. 2012). This observation therefore promotes the interpretation that each individual in the population adopts an FBA/Bang–Bang policy (Giordano et al. 2016), and that the relative fraction of total resource allocated to an EFM reflects the fraction of individuals within the population investing exclusively in that metabolic pathway. Due to the overall scaling effect of total catalytic biomass concentration x in the effective return-on-investment, the concentration of resource on the EFM with largest return-on-investment will increase as x does also, and therefore the maximum entropy resource allocation strategy of the entire population also approaches the FBA/Bang–Bang policy as the population grows in size. This leads to further concentration of resource on the pathway with greatest contribution to growth, whereas, when the population is growth-limited, resource allocation based on the temporal maximum entropy control is increased to pathways with higher total metabolite yield.

Entropy is the uniquely-defined continuous function of a discrete probability distribution, monotonically increasing in the number of states when each occurs with equal probability, which respects the composition law of fractional partitioning (Jaynes 1957; Shore and Johnson 1980). For applications to dynamic resource allocation this naturally identifies the maximum entropy distribution as the appropriate control law for performing model reduction based on EFM families, where several alternative EFM weightings have been introduced previously (Song and Ramkrishna 2010, 2011; Vilkhovoy et al. 2016). The suggested EFM weighting based on the maximum entropy control law generalises those considered in prior work. Using EFM families becomes particularly important for larger metabolic networks where an explosion in the number of EFMs (Klamt and Stelling 2002) makes direct model parameterisation infeasible; however, the maximum entropy framework provides a consistent methodology for recursive model reduction. It is of worth pointing out that DeVilbiss and Ramkrishna have recently proposed an information theory-based model selection scheme (DeVilbiss and Ramkrishna 2017), using as a test case metabolic models expressed in terms of EFMs and the control laws of Young and Ramkrishna. Models based on EFM families were shown to provide the most succinct description of steady state fluxes as measured by information-theoretic criteria, and it would therefore be intriguing to explore how these criteria synergise with the information-theoretic concept of the maximum entropy control.

Appendices

A Derivation of maximum entropy control

To solve the optimal control problem (14) one introduces the Hamiltonian

$$\begin{aligned} \mathcal {H}(\Delta \mathbf {X},\mathbf {u},\varvec{\lambda }) = \varvec{\lambda }^T[ \mathbf {F}(\mathbf {X}(t),\mathbf {u}^0) + \mathbf {A} \Delta \mathbf {X} + \mathbf {B} \Delta \mathbf {u} ] + \sigma H(\mathbf {u}) , \end{aligned}$$

where $\varvec{\lambda }$ is a co-state vector the same dimension as $\mathbf {X}$. Applying Pontryagin’s maximum principle implies maximisation of $\mathcal {H}$, or equivalently the functional

$$\begin{aligned} \mathcal {F}(\mathbf {u}) = \varvec{\lambda }^T \mathbf {B} \mathbf {u} + \sigma H(\mathbf {u}) , \end{aligned}$$

with respect to $\mathbf {u}$ subject to the constraint

$$\begin{aligned} \sum _{k=1}^K u_k = 1 . \end{aligned}$$

This results in the K first-order conditions

$$\begin{aligned} \varvec{\lambda }^T \mathbf {B}^k -\sigma ( 1+ \log u_k) - \alpha = 0 \end{aligned}$$

where $\alpha \ge 0$ is a Lagrange multiplier and $\mathbf {B}^k$ is the kth column of $\mathbf {B}$. The general solution of these equations takes the form

$$\begin{aligned} u_k = \frac{1}{Q}\exp (\varvec{\lambda }^T \mathbf {B}^k/\sigma ) \end{aligned}$$

and Q is determined from the above constraint such that

$$\begin{aligned} Q = \sum _{k=1}^K \exp (\varvec{\lambda }^T \mathbf {B}^k/\sigma ) . \end{aligned}$$

The value of the co-state vector $\varvec{\lambda }$ is obtained as in Young and Ramkrishna (2007) by solving the boundary value problem

$$\begin{aligned} - \frac{d}{d \tau } \varvec{\lambda } = \frac{\partial \mathcal {H}}{\partial \Delta \mathbf {X}} = \mathbf {A}^T \varvec{\lambda } , \quad \varvec{\lambda }(t + \Delta t ) = \mathbf {q} \end{aligned}$$

whose solution is

$$\begin{aligned} \varvec{\lambda }(t + \tau ) = \mathbf {e}^{\mathbf {A}^T(\Delta t - \tau )} \mathbf {q} , \quad 0 \le \tau \le \Delta t. \end{aligned}$$

This expression for $\varvec{\lambda }$ is substituted into the above expression for $u_k$ and because, ultimately, only the optimal control input at the current time t is of interest, one can set $\tau = 0$ as in Young and Ramkrishna (2007), which yields the maximum entropy control (15).

B First-order correction to return-on-investment

Expressed in terms of the average zeroth-order return-on-investment

$$\begin{aligned} \bar{R}_0(\mathbf {m}_s) = \sum _{k=1}^K r_k(\mathbf {m}_s)\mathbf {c}^T \mathbf {Z}^k u^0_k , \end{aligned}$$

the vector $\mathbf {q}^T\mathbf {A}$ is

$$\begin{aligned} \mathbf {q}^T\mathbf {A} = (\mathbf {0},1) \frac{\partial }{\partial \mathbf {X}} \mathbf {F}(\mathbf {X}(t),\mathbf {u}^0)= \left( x \left( \frac{\partial \bar{R}_0}{\partial \mathbf {m}_{s}} (\mathbf {m}_{s}) \right) ^T, \bar{R}_0(\mathbf {m}_s) \right) . \end{aligned}$$

The first-order expansion of $\mathbf {e}^{\mathbf {A} \Delta t}$ takes the form $\mathbf {I} + \Delta t \mathbf {A}$, where $\mathbf {I}$ is the identity matrix, and therefore the effective return-on-investment $\mathcal {R}^k_{\Delta t}(\mathbf {m}_s)$ in (16) is approximated to first-order by

$$\begin{aligned} x r_k(\mathbf {m}_s)\mathbf {q}^T(\mathbf {I} + \Delta t \mathbf {A}) \begin{pmatrix} \mathbf {S}_s \\ \mathbf {c}^T \end{pmatrix} \mathbf {Z}^k . \end{aligned}$$

Taking the coefficient of $x \Delta t$ and substituting for $\mathbf {q}^T\mathbf {A}$ gives the first-order correction to return-on-investment

$$\begin{aligned} R^k_1(\mathbf {m}_s) = r_k(\mathbf {m}_s) \left[ \bar{R}_0(\mathbf {m}_s) \mathbf {c}^T + x \left( \frac{\partial \bar{R}_0}{\partial \mathbf {m}_{s}} (\mathbf {m}_{s}) \right) ^T \mathbf {S}_s \right] \mathbf {Z}^k , \end{aligned}$$

which can be written as

$$\begin{aligned} R^k_1(\mathbf {m}_s) = \bar{R}_0(\mathbf {m}_s) R^k_0(\mathbf {m}_s) + x Y_k(\mathbf {m}_s) \end{aligned}$$

using the definition of $Y_k(\mathbf {m}_s)$ provided in (20).

C Expressing $\mathcal {F}(\mathbf {u})$ in terms of EFM families

Using (17) to substitute for $\mathbf {B}^k$ in the effective return-on-investment (16) means that $\mathcal {F}(\mathbf {u})$ takes the form

$$\begin{aligned} \mathcal {F}(\mathbf {u}) = x \mathbf {q}^T \mathbf {e}^{\mathbf {A} \Delta t} \begin{pmatrix} \mathbf {S}_{ex} \\ \mathbf {c}^T \end{pmatrix} \sum _{k=1}^K r_k(\mathbf {m}_s) \mathbf {Z}^k u_k + \sigma H(\mathbf {u}) . \end{aligned}$$

The sum over k in the first term can be written as

$$\begin{aligned} \sum _{k=1}^K r_k(\mathbf {m}_s) \mathbf {Z}^k u_k = \sum _{J=1}^M \sum _{j \in F_J} r_j(\mathbf {m}_s) \mathbf {Z}^j u_j = \sum _{J=1}^M \left( \sum _{j \in F_J} r_j(\mathbf {m}_s) \mathbf {Z}^j \tilde{u}_j \right) U_J, \end{aligned}$$

where the second equality follows from the identity $\tilde{u}_j = u_j /U_J$ for $j \in F_J$, which implies

$$\begin{aligned} \mathcal {F}(\mathbf {u}) = \sum _{J=1}^M \left( \sum _{j \in F_J} \mathcal {R}^j_{\Delta t}(\mathbf {m}_s) \tilde{u}_j \right) U_J + \sigma H(\mathbf {u}) . \end{aligned}$$

Using the composition property (24) of entropy $H(\mathbf {u})$, one has

$$\begin{aligned} \mathcal {F}(\mathbf {u}) = \sum _{J=1}^M \left( \sum _{j \in F_J} \mathcal {R}^j_{\Delta t}(\mathbf {m}_s) \tilde{u}_j + \sigma H(\tilde{\mathbf {u}}_J)\right) U_J + \sigma H(\mathbf {U}) \end{aligned}$$

which is (25) with

$$\begin{aligned} \mathcal {F}_J(\tilde{\mathbf {u}}_J) = \sum _{j \in F_J} \mathcal {R}^j_{\Delta t}(\mathbf {m}_s) \tilde{u}_j + \sigma H(\tilde{\mathbf {u}}_J) \end{aligned}$$

the objective functional $\mathcal {F}$ restricted to the Jth family. Notice that the weighting (22) defines the effective return-on-investment for the Jth family derived directly from system (9) to be

$$\begin{aligned} \mathcal {\tilde{R}}^J_{\Delta t} (\mathbf {m}_s) = \sum _{j \in F_J} \mathcal {R}^j_{\Delta t}(\mathbf {m}_s) \tilde{u}_j \end{aligned}$$

and therefore $\mathcal {F}_J(\tilde{\mathbf {u}}_J)$ includes an entropic correction that is only zero when $\tilde{u}_J$ describes an FBA/Bang–Bang policy.