Abstract
Let \(\{N(t),\;t \geq 0\}\) be a nonnegative-integer-valued stochastic process that counts the occurrences of a given event. That is, N(t) is the number of events in the time interval \([0,t)\). For example, N(t) can be the number of bulb replacements in a lamp that is continuously on, and the dead bulbs are immediately replaced
Access provided by Autonomous University of Puebla. Download chapter PDF
Similar content being viewed by others
Keywords
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.
1 Basic Theory of Renewal Processes
Let \(\{N(t),\;t \geq 0\}\) be a nonnegative-integer-valued stochastic process that counts the occurrences of a given event. That is, N(t) is the number of events in the time interval \([0,t)\). For example, N(t) can be the number of bulb replacements in a lamp that is continuously on, and the dead bulbs are immediately replaced (Fig. 4.1).
Renewal process
Let \(0 \leq {t}_{1} \leq {t}_{2} \leq \ldots \) be the times of the occurrences of consecutive events and t 0 = 0 and \({T}_{i} = {t}_{i} - {t}_{i-1}\), i = 1, 2, 3, … be the time intervals between consecutive events.
Definition 4.1.
\({t}_{1} \leq {t}_{2} \leq \ldots \) is a renewal process if the time intervals between consecutive events \({T}_{i} = {t}_{i} - {t}_{i-1}\), i = 2, 3, …, are independent and identically distributed (i.i.d.) random variables with CDF
The nth event time, t n , n = 1, 2, …, is referred to as the nth renewal point or renewal time. According to the definition, the first time interval might have a different distribution.
We assume that F(0) = 0 and \(F(+0) = \mathbf{P}\left ({T}_{k} = 0\right ) < 1\). In this case
Remark 4.2.
{N(t), t ≥ 0} and {t n , n ≥ 1} mutually and univocally determine each other because for arbitrary t ≥ 0 and k ≥ 1 we have
Definition 4.3.
When \(\mathbf{P}\left ({T}_{k} \leq x\right ) = F(x),\;\;k = 2,3,\ldots \), but \({F}_{1}(x) = \mathbf{P}\left ({T}_{1} \leq x\right )\not\equiv F(x)\), the process is referred to as a delayed renewal process.
Remark 4.4.
\({T}_{1},{T}_{2},\ldots \) are i.i.d. random variables and from \({t}_{n} = {T}_{1} + \ldots + {T}_{n}\), and we can compute the distribution of the time of the nth event \({F}^{(n)}(x) = \mathbf{P}\left ({t}_{n} \leq x\right )\) using the convolution formula
Starting from \({F}^{(1)}(x) = {F}_{1}(x)\) the same formula applies in the delayed case.
Definition 4.5.
The function \(H(t) = \mathbf{E}\left (N(t)\right ),\;\;t \geq 0\), is referred to as a renewal function.
One of the main goals of renewal theory is the analysis of the renewal function H(t) and the description of its asymptotic behavior. Below we discuss the related results for regular renewal processes. The properties of delayed renewal processes are similar, and we do not provide details on them here. We will show that the law of large numbers and the central limit theorem hold for the renewal process (see also Ch. 5. in [48]).
Theorem 4.6.
If \(\{{T}_{n},\;n = 1,2,\ldots \}\) is a series of nonnegative i.i.d. random variables and \(\mathbf{P}\left ({T}_{1} = 0\right ) < 1\) , then there exists ρ 0 > 0 such that for all 0 < ρ < ρ 0 and t ≥ 0
holds.
Proof (Proof 1 of Theorem 4.6).
From the Markov inequality (Theorem 1.35) we have
where \(\rho < {\rho }_{0} = \kappa \), \(\kappa =\log \frac{1} {h}\), and \(h = \mathbf{E}\left ({\mathrm{e}}^{-{T}_{1}}\right )\). Additionally, h < 1 because F(0) = 0 and \(\mathbf{P}\left ({T}_{1} = 0\right ) < 1\). □
Proof (Proof 2 of Theorem 4.6).
According to the condition of the theorem, there exist ε and δ positive numbers such that \(\mathbf{P}\left ({T}_{k} \geq \delta \right ) > \epsilon \). Introducing \(\{{T}_{k}^{{\prime}} = \delta {\mathcal{I}}_{\left \{{T}_{k}\geq \delta \right \}},\;k = 1,2,\ldots \}\) (where \({T}_{n}^{{\prime}},\;n = 1,\) 2, …, is a series of i.i.d. random variables) and the related \(\{{N}^{{\prime}}(t),\;t \geq 0\}\) renewal process we have that \(\mathbf{P}\left ({T}_{k}^{{\prime}}\leq {T}_{k}\right ) = 1,\;k \geq 1\), and consequently \(\mathbf{P}\left ({N}^{{\prime}}(t) \geq N(t)\right ) = 1,\;t \geq 0\). The distribution of N ′(t) is negative binomial with the parameter \(p = \mathbf{P}\left ({T}_{k}^{{\prime}}\geq \delta \right )\) and order \(r = \lfloor t/\delta \rfloor \),
from which the statement of the theorem follows. □
Corollary 4.7.
All moments of N(t) (t ≥ 0) are finite, and the renewal function H(t) is also finite for all t ≥ 0.
Proof.
The corollary comes from Theorem 4.6 and the inequality \({x}^{n} \leq n!{\mathrm{e}}^{x},\) \(n \geq 1,\;x \geq 0\). □
Before conducting an analysis of the renewal function we recall some properties of convolution.
Let A(t) and B(t) be monotonically nondecreasing right-continuous functions such that \(A(0) = B(0) = 0\).
Definition 4.8.
The convolution of A(t) and B(t) [denoted by A ∗ B(t)] is
Lemma 4.9.
A ∗ B(t) = B ∗ A(t).
Proof.
From B(0) = 0 we have \(B(t - y) =\int\limits_{0}^{t-y}\mathrm{d}B(s)\), and consequently
□
Remark 4.10.
The definition of the renewal function H(t)
immediately determines the relation between the renewal function and the order k of the convolutions of the event time distribution
Theorem 4.11.
If {T n , n = 1,2,…} is a series of i.i.d. random variables and \(\mathbf{P}\left ({T}_{1} < 0\right ) = 0,\;\;\mathbf{P}\left ({T}_{1} = 0\right ) < 1\) , then H(t) satisfies the renewal equation
Proof.
According to Remarks 4.4 and 4.10, the renewal function can be written as
where the order of the summation and the integration are interchanged based on Corollary 4.7. □
In the case of a delayed renewal process, the renewal function is denoted by H 1(t), and the same composition holds as for the regular renewal process (Remark 4.10)
but in this case \({F}_{1}\not\equiv F\).
Theorem 4.12.
The renewal function can be written in the following forms:
1.1 Renewal Equations
1.1.1 Renewal Equations
Definition 4.13.
An integral equation of the type
where a(t) and F(t) are known functions and A(t) is unknown, is referred to as a renewal equation (see also Theorem 4.1 of Ch. 5. in [48]).
Theorem 4.14.
If a(t), t ≥ 0, is a bounded real function that is Riemann–Stieltjes integrable according to H(t) over any finite interval, then there uniquely exists the function A(t), t ≥ 0, which is finite over any finite interval and satisfies the renewal equation
and furthermore it satisfies
where \(H(t) =\sum \limits_{k=1}^{\infty }{F}^{(k)}(t),\;t \geq 0\) , is the renewal function.
Proof.
First we show that the function A(t), t ≥ 0, defined by equation (ii), is (a) bounded on the \([0,T]\) interval for all T > 0 and (b) satisfies (i). Next we prove that (c) all solutions of (i) that are bounded on \([0,T]\) can be given in form (ii), i.e., the solution is unique.
-
(a)
Since a(t) is bounded and H(t) is monotonically nondecreasing, we have
$$ \begin{array}{rcl} {\sup }_{0\leq t\leq T}\vert A(t)\vert &\leq &{\sup }_{0\leq t\leq T}\vert a(t)\vert +\int\limits_{0}^{T}{[\sup }_{ 0\leq y\leq T}\vert a(y)\vert \mathrm{d}H(x) \\ & \leq &{\sup }_{0\leq t\leq T}\vert a(t)\vert (1 + H(T)) < \infty.\end{array}$$ -
(b)
Furthermore, we have
$$\begin{array}{rcl} A(t)& =& a(t) + H {\ast} a(t) = a(t) + \left (\sum \limits_{k=1}^{\infty }{F}^{(k)}\right ) {\ast} a(t) \\ & =& a(t) + F {\ast} a(t) + \left (\sum \limits_{k=2}^{\infty }{F}^{(k)}\right ) {\ast} a(t) \\ & =& a(t) + F {\ast} [a(t) + \left (\sum \limits_{k=1}^{\infty }{F}^{(k)}\right ) {\ast} a(t)] \\ & =& a(t) + F {\ast} A(t).\end{array}$$ -
(c)
We prove this by successive approximation. According to equation (i), \(A = a + F {\ast} A\). Substituting this into (i) we have
$$\begin{array}{rcl} A = a(t) + F {\ast} (a + F {\ast} A)& =& a + F {\ast} a + F {\ast} (F {\ast} A) \\ & =& a + F {\ast} a + {F}^{(2)} {\ast} A.\end{array}$$Continuously substituting equation (i) we obtain for n ≥ 1 that
$$A = a + F {\ast}a + {F}^{(2)} {\ast} (a + F {\ast}A) = \ldots = a +\sum \limits_{k=1}^{n-1}({F}^{(k)} {\ast}a) + {F}^{(n)} {\ast}A.$$Since A(t) is bounded on every finite interval according to (a), \({F}^{(n)}(0-) = 0,\) F (n)(y) is monotonically nondecreasing, and \({F}^{(n)}(t) \rightarrow 0,\;n \rightarrow \infty \), for all fixed t, we have that for a fixed t
$$\vert {F}^{(n)} {\ast}A(t)\vert = \left \vert \int\limits_{0}^{t}A(t - y)\mathrm{d}{F}^{(n)}(y)\right \vert {\leq \sup }_{ 0\leq y\leq t}\vert A(t-y)\vert {F}^{(n)}(t) \rightarrow 0,\;\;n \rightarrow \infty.$$From the fact that a(t) is bounded it follows that
$${\lim }_{n\rightarrow \infty }\left (\sum \limits_{k=1}^{n-1}{F}^{(k)}\right ) {\ast} a(t) = \left (\sum \limits_{k=1}^{\infty }{F}^{(k)}\right ) {\ast} a(t) = H {\ast} a(t),$$and consequently
$$A(t) = a(t) {+\lim }_{n\rightarrow \infty }\left [\left (\sum \limits_{k=1}^{n-1}{F}^{(k)}\right ) {\ast} a(t) + {F}^{(n)}A(t)\right ] = a(t) + H {\ast} a(t).$$This means that if A is a bounded solution of (i), then it is identical with (ii).
□
1.1.2 Analysis of the Renewal Function
One of the main goals of the renewal theorem is the analysis of the renewal function. According to Theorem 4.12, in the case of delayed renewal processes the renewal function H 1(t) can be obtained from F 1(t) and H(t). In the rest of this section we focus on the analysis of the renewal function of an ordinary renewal process, H(t), that is, F k = F, k ≥ 1. During the subsequent analysis we assume that F(t) is such that \(F(0-) = 0\) and F(0 + ) < 1.
Theorem 4.15 (Elementary renewal theorem).
There exists the limit
and it is 0 if \(\mathbf{E}\left ({T}_{1}\right ) = \infty \) .
Definition 4.16.
The random variable X has a lattice distribution if there exists d > 0 and r ∈ R such that the random variable \(\frac{1} {d}(X - r)\) is distributed on the integer numbers, that is, \(\mathbf{P}\left (\frac{1} {d}(X - r) \in \mathbf{Z}\right ) = 1\). The largest d with that property is referred to as the step size of the distribution.
Remark 4.17.
If X has a lattice distribution with step size d, then
where \(\psi (u) = \mathbf{E}\left ({\mathrm{e}}^{iuX}\right ),\;u \in R\), denotes the characteristic function of X. In this case, | ψ(u) | < 1 if 0 < | u | < 2π ∕ d. If the distribution of X is not lattice, then | ψ(u) | < 1 if u≠0.
Theorem 4.18 (Blackwell’s theorem).
If F(t) is a lattice distribution with step size d, then
where \({q}_{n} = H(nd) - H((n - 1)d)\) . If F(t) is not a lattice distribution, then for all h > 0
holds.
The following theorems require the introduction of direct Riemann integrability, which is more strict than Riemann integrability.
Let g be a nonnegative function on the interval \([0,\infty )\) and
Definition 4.19.
The function g is directly Riemann integrable if s(δ) and S(δ) are finite for all δ > 0 and
Remark 4.20.
If the function g is directly Riemann integrable, then g is bounded, and the limit of s(δ) and S(δ) at \(\delta \rightarrow 0\) is equal to the infinite Riemann integral, that is,
Sufficient and necessary conditions for direct Riemann integrability:
-
(a)
There exists δ > 0 such that \(S(\delta ) < \infty \).
-
(b)
g is almost everywhere continuous along the real axes according to the Lebesgue measure (that is, equivalent to Riemann integrability on every finite interval).
Sufficient conditions for direct Riemann integrability:
g is bounded and has a countable number of discontinuities, and at least either condition (a) or (b) holds:
-
(a)
g equals 0 apart from a finite interval.
-
(b)
g is monotonically decreasing and \(\int\limits_{0}^{\infty }g(x)\mathrm{d}x < \infty \).
Theorem 4.21 (Smith’s renewal theorem).
If g(x) ≥ 0, x ≥ 0, is a nonincreasing directly Riemann integrable function on the interval \([0,\infty )\) , then for \(t \rightarrow \infty \) one of the following identities holds:
-
(a)
If F is a nonlattice distribution, then
$${\lim }_{t\rightarrow \infty }H {\ast} g(t) {=\lim }_{t\rightarrow \infty }\int\limits_{0}^{t}g(t - u)\mathrm{d}H(u) = \frac{1} {\mathbf{E}\left ({T}_{1}\right )}\int\limits_{0}^{\infty }g(u)\mathrm{d}u.$$ -
(b)
If F is a lattice distribution with step size d, then
$${\lim }_{n\rightarrow \infty }H{\ast}g(x+nd) {=\lim }_{n\rightarrow \infty }\int\limits_{0}^{x+nd}g(x+nd-u)\mathrm{d}H(u) = \frac{d} {\mathbf{E}\left ({T}_{1}\right )}\sum \limits_{k=0}^{\infty }g(x+kd).$$
Remark 4.22.
Blackwell’s theorem (Theorem 4.18) follows from Smith’s renewal theorem (Theorem 4.21) assuming that \(g(u) = {\mathcal{I}}_{\left \{0<u\leq h\right \}}\). The reverse direction is an implicit consequence of the proof of Blackwell’s theorem provided by Feller in [31].
Before proving Theorem 4.21 we collect some simple properties of the renewal function H(t).
Lemma 4.23.
H is monotonically nondecreasing and continuous from the right.
Proof.
F (k)(t) is monotonically nondecreasing and continuous from the right for all k ≥ 1, and the series \(\sum\limits_{k=1}^{\infty }{F}^{(k)}(t)\) is uniformly convergent on every finite interval, from which the lemma follows. □
Lemma 4.24.
The function H is subadditive, that is,
for t,h ≥ 0.
Proof.
Since H(0) = 0, it is enough to consider the case where t, h > 0. Let \(n(t) =\inf \{ n :\; {t}_{n} \geq t,\;n \geq 0\}\). If t n ≤ t for all n ≥ 0, then let \(n(t) = \infty \). This case can occur only on a set with measure 0.
Due to the fact that \(\mathbf{P}\left ({T}_{1} = 0\right )\) might be positive, the relation of n(t) and N(t) is not deterministic. It holds that n(t) ≥ N(t) + 1 and the right continuity of N(t) implies \(N({t}_{n(t)}) = N(t),\;t \geq 0\). Using that we have
and using the total probability theorem, we obtain
Since t k is a renewal point, the conditional expected value in the last summation does not depend on the condition
and in this way we have
from which the lemma follows. □
Lemma 4.25.
For the renewal function H the following inequality holds:
Proof.
From the previous statement and the monotonicity of H
□
Remark 4.26.
The nonnegative subadditive functions can be estimated from the preceding expression by a linear function.
Lemma 4.27.
For arbitrary λ > 0 the Laplace–Stieltjes transform \({H}^{\sim }(\lambda ) =\int\limits_{0}^{\infty }{\mathrm{e}}^{-\lambda t}\mathrm{d}H(t)\) , λ ≥ 0, of the function H can be represented in the Laplace–Stieltjes transform as
where \({\varphi }^{\sim }(\lambda ) = \mathbf{E}\left ({\mathrm{e}}^{-\lambda {T}_{1}}\right )\) is the Laplace–Stieltjes transform of the distribution function F.
Proof.
For λ > 0 there obviously exists H ∼ (λ) since, according to Eqs. (1.3) and (4.2),
It is clear that
Using this equality we obtain
where 0 < φ(λ) < 1 if λ > 0. □
Proof of Elementary Renewal Theorem. First we prove that the limit exists. If t ≥ 1, then we have that \(0 \leq \frac{H(t)} {t} \leq \frac{1+t} {t} H(1) \leq 2H(1)\) is bounded. Let \(c {=\inf }_{t\geq 1}\frac{H(t)} {t}\). Then for arbitrary ε > 0 there exists a number t 0 > 0 such that
Moreover, for all integers k ≥ 1 and τ ≥ 0
and consequently
and
follows. We have proved the existence of the limit.
Using the preceding expression for the Laplace–Stieltjes transform h(λ),
and we obtain
By means of the relation for the derivative of the Laplace–Stieltjes transform
On the other hand, in the case 0 < λ ≤ 1, we can give a uniform upper estimation for the integrand in Eq. (4.3):
furthermore,
so from the Lebesgue majorated convergence theorem
Summing up the previous results we obtain
□
1.2 Limit Theorems for Renewal Processes
Theorem 4.28.
Let \(0 < \mathbf{E}\left ({T}_{1}\right ) = \mu < \infty \) ; then the following stochastic convergence holds:
Proof.
The proof of Theorem 4.28 is based on the relation
from Comment 4.2. Let us estimate the probability \(\mathbf{P}\left (\vert N(t)/t - 1/\mu \vert > \epsilon \right )\) for arbitrary ε > 0. Let \(n = n(t) = \lfloor t/\mu + \epsilon t\rfloor \); then
which by Bernoulli’s law of large numbers tends to 0 for the sequence t n , n = 1, 2, …, as \(t \rightarrow \infty \). The probability \(\mathbf{P}\left (N(t)/t - 1/\mu < -\epsilon \right )\) is estimated in a similar way. □
Remark 4.29.
By the strong law of large numbers, \(\frac{{t}_{k}} {k} \rightarrow \mu \), \(k \rightarrow \infty \), with probability 1. Using this fact one can prove that with probability 1
The convergence with probability 1 remains valid for delayed renewal processes if the first time interval is finite with probability 1.
Theorem 4.30.
If \(\mathbf{E}\left ({T}_{1}\right ) = \mu > 0,\ {\mathbf{D}}^{2}\left ({T}_{1}\right ) = {\sigma }^{2} < \infty \) , then as \(t \rightarrow \infty \)
Proof.
Let x be a real number and denote
Note that r(t) ≥ 1 if \(\sqrt{t} + x\sigma /\sqrt{\mu } - \mu /\sqrt{t} \geq 0\). Since \(r(t) \rightarrow \infty \) as \(t \rightarrow \infty \), then from the central limit theorem it follows that for all \(x \in \mathbb{R}\)
Using the relation \(\{N(t) \leq r(t)\} =\{ {t}_{r(t)} > t\}\) we have
It can be easily checked that
and the continuity of the standard normal distribution function implies the convergence
The equation \(1 - \Phi (-x) = \Phi (x)\) follows from the symmetry of the standard normal distribution. □
The following results (without proof) concerning the mean value and variance of the renewal process N(t) are a generalization of previous results and are valid for the renewal processes with delay, too.
Theorem 4.31.
If \({\mu }_{2} = \mathbf{E}\left ({T}_{1}^{2}\right ) < \infty \) and T 1 has a nonlattice distribution, then as \(t \rightarrow \infty \) [31, XIII-12§]
If, additionally, \({\mu }_{3} = \mathbf{E}\left ({T}_{1}^{3}\right ) < \infty \) , then [31]
2 Regenerative Processes
Many queueing systems can be described by means of regenerative processes. This property makes it possible to prove the limit and stability theorems in order to use the method of simulation.
Definition 4.32.
Let T be a nonnegative random variable and \(Z(t),\;t \in [0,T)\) be a stochastic process. The pair (T, Z(t)), taking on values in the measurable space \((\mathcal{Z},\mathcal{B})\), is called a cycle of length T.
Definition 4.33.
The stochastic process Z(t), t ≥ 0, taking on values in the measurable space \((\mathcal{Z},\mathcal{B})\), is called a regenerative process with moments of regeneration \({t}_{0} = 0 < {t}_{1} < {t}_{2} < \ldots \) if there exists a sequence of independent cycles \(({T}_{k},{Z}_{k}(t)),\;\;k \geq 1\), such that
-
(1)
\({T}_{k} = {t}_{k} - {t}_{k-1},\;\;k \geq 1\);
-
(2)
\(\mathbf{P}\left ({T}_{k} > 0\right ) = 1,\;\;\mathbf{P}\left ({T}_{k} < \infty \right ) = 1\);
-
(3)
All cycles are stochastically equivalent.
-
(4)
\(Z(t) = {Z}_{k}(t - {t}_{k-1})\;\) if \(t \in [{t}_{k-1},{t}_{k}),\;k \geq 1\).
Definition 4.34.
If property (3) is fulfilled only starting with the second cycle (analogously to the renewal processes), then we have a delayed regenerative process.
Remark 4.35.
t k , k ≥ 1, is a renewal process.
In the case of regenerative processes, an important task is to find conditions assuring the existence and possibility of determining the limit
It is also important to estimate the rate of convergence (especially upon examination of the stability problems of queueing systems and simulation procedures).
Let {Z(t), t ≥ 0} be a regenerative process taking on values in the measurable space \((\mathcal{Z},\mathcal{B})\) with regeneration points \({t}_{0} = 0 < {t}_{1} < {t}_{2} < \ldots \), \({T}_{n} = {t}_{n} - {t}_{n-1},\;\;n = 1,2,\ldots \). Assume that Z(t) is right continuous and there exists a limit from the left. Then the cycles \(\{{T}_{n},\ \{Z({t}_{n-1} + u) :\; 0 \leq u < {T}_{n}\}\},\;n = 1,2,\ldots \), are independent and stochastically equivalent; \(\{{t}_{n},\;n \geq 1\}\); and the corresponding counting process {N(t), t ≥ 0} is a renewal process. Let F denote the common distribution of random variables {T n , n ≥ 1}.
The most important application of Smith’s theorem is the determination of limit values \({\lim }_{t\rightarrow \infty }\mathbf{E}\left (W(t)\right )\) for the renewal and regenerative processes, where \(W(t) = \Psi (t,N,Z)\) is the function of t, the renewal process N, and the regenerative process Z. The determination of the limit value is based on a more general theorem.
Theorem 4.36.
Let {V (t), t ≥ 0} be a real-valued stochastic process on the same probability space as the process {N(t), t ≥ 0}, and for which the mean value \(f(t) = \mathbf{E}\left (V (t)\right )\) is bounded on each finite interval. Let
Assume that the positive and negative parts of g are directly Riemann integrable. If F is a nonlattice distribution, then
A similar result is valid if F is a lattice distribution.
Remark 4.37.
In the theorem, the property of direct Riemann integrability was required separately for the positive and negative parts of the function g. The reason is that the property is defined only for nonnegative functions.
Proof.
It is clear that
Let us add and subtract F ∗ f(t); then we get the renewal equation
The solution of the equation is \(f(t) = g + H {\ast} g(t)\), which because of the convergence \(g(t) \rightarrow 0,\;t \rightarrow \infty \), and the elementary renewal theorem as a simple consequence of direct Riemann integrability tends to \(\frac{1} {\mu }\int\limits_{0}^{\infty }g(x)\mathrm{d}x\) as \(t \rightarrow \infty \). □
Remark 4.38.
From the proof it is clear that under the condition of Theorem 4.36 for an arbitrary process V (t) there exists the representation \(\mathbf{E}\left (V (t)\right ) = H {\ast} g(t)\) and for the existence of the limit the direct Riemann integrability is required. This representation is interesting if V (t) depends on Z(t).
Special Case Let \(h\, :\; \mathcal{Z}\rightarrow R\) be a measurable function for which, for all t, E | h(Z(t)) | < ∞. Z(t) is a regenerative process, and the part starting with the second cycle is independent of the first cycle of length T 1, so for arbitrary 0 < s < t
Using the previous notation
Theorem 4.39.
If g + and g − are directly Riemann integrable, then
For arbitrary \(A \in \mathcal{B}\) the following equality holds:
Proof.
The first relation follows from the previous theorem, and for the second one it is necessary to mention that, since the trajectories of Z are right continuous and have left limits, the (integrable, bounded) function \(\mathbf{P}\left (Z(s) \in A,\;{T}_{1} > s\right )\) has a countable number of discontinuities and, consequently, is directly Riemann integrable. □
We give one more limit theorem (without proof) that is often useful in practice.
Theorem 4.40.
Let F be a nonlattice distribution, and let at least one of the following conditions be fulfilled:
-
(a)
\(\mathbf{P}\left (Z(t) \in A\right )\) is Riemann integrable on an arbitrary finite interval, and \(\mu =\int\limits_{0}^{\infty }x\,\mathrm{d}F(x) <\ \infty \) holds.
-
(b)
Starting with a certain integer n ≥ 1 the distribution functions defined by \({F}^{(1)} =\ F,\;\;{F}^{(n+1)} = {F}^{(n)} {\ast} F\) , are absolute continuous and \(\mu =\int\limits_{0}^{\infty }x\,\mathrm{d}F(x) < \infty \).
Then the following relation holds:
Example 4.41.
Let us consider the renewal process {N(t), t ≥ 0}; the renewal moments are
and, furthermore, \(\mathbf{P}\left ({T}_{k} \leq x\right ) = F(x),\;k \geq 1\) , \(\mu =\int\limits_{0}^{\infty }x\,\mathrm{d}F(x)\). For arbitrary t > 0 we define
(For example, at instant t, δ(t) indicates how much time passed without a car arriving at the station, and γ(t) indicates how long it was necessary to wait till the arrival of the next car, on the condition that the interarrival times are i.i.d. random variables with the common distribution function F.)
Theorem 4.42.
{δ(t), t ≥ 0} and {γ(t), t ≥ 0} are regenerative processes, and in the case of the nonlattice distribution F,
Proof.
Both processes are obviously regenerative with common regeneration points t n , n ≥ 1. By our previous theorem,
furthermore,
so
using the identity \(\mu =\int\limits_{0}^{\infty }(1 - F(s))\textrm{ d}s\) (Exercise 1.5). Similarly, for the process {γ(t), t ≥ 0} we obtain
The statement for {γ(t), t ≥ 0} can be obtained analogously. □
Similarly to the renewal processes, the law of large numbers and the central limit theorem can be proved for the regenerative processes, too. Here we will not deal with these questions.
2.1 Estimation of Convergence Rate for Regenerative Processes
For a wide class of regenerative processes (e.g., stochastic processes describing queueing systems) one can estimate the rate of convergence of distributions of certain parameters to a stationary distribution by means of the so-called coupling method [65].
Lemma 4.43 (Coupling lemma).
For the arbitrary random variables X and Y and an arbitrary Borel set A of the real line the following statements hold:
-
(i)
\(\vert \mathbf{P}\left (X \in A\right ) -\mathbf{P}\left (Y \in A\right )\vert \leq \mathbf{P}\left (X\neq Y \right ).\)
-
(ii)
If \(X = {X}_{1} + \ldots + {X}_{n}\) and \(Y = {Y }_{1} + \ldots + {Y }_{n}\) , then \(\vert \mathbf{P}\left (X \in A\right ) -\mathbf{P}\left (Y \in A\right )\vert \leq \sum \limits_{k=1}^{n}\mathbf{P}\left ({X}_{k}\neq {Y }_{k}\right ).\)
Proof.
If \(\mathbf{P}\left (X \in A\right ) = \mathbf{P}\left (Y \in A\right )\), then (i) is obviously true.
Suppose that \(\mathbf{P}\left (X \in A\right ) > \mathbf{P}\left (Y \in A\right )\) (if one changes the notation, then this can always be done if the two probabilities differ). Then
Proof of relation (ii). Since \(\{X\neq Y \} \subset {\bigcup }_{k=1}^{n}\{{X}_{k}\neq {Y }_{k}\}\), we have
□
Application of Coupling Lemma Let Z = { Z(j), j ≥ 1} be the discrete-time, real-valued regenerative process under consideration. Assume that there exists the weak stationary limit of the process \(\tilde{Z} =\{ Z(j + n),\;j \geq 1\}\) as \(n \rightarrow \infty \) (its finite-dimensional distributions weakly converge to the finite-dimensional distributions of a stationary process), which is also regenerative, and let \(Y =\{ Y (j),\;j \geq 1\}\) be its realization, not necessarily different from Z on the same probability space. Let τ denote the first instant when the processes Z and Y are regenerated at the same time (in many concrete cases the distribution of τ can be easily estimated). Then the convergence rate of the distribution of Z(j) can be estimated by means of the distribution of τ as follows: if after the regeneration point τ the process Z is replaced by the next part of process Y following the common regeneration point τ, then the finite-dimensional distributions of process Z do not change. It is clear that \(\{\tau < j\} \subseteq \{ Z(j) = Y (j)\}\), i.e., \(\{Z(j)\neq Y (j)\} \subseteq \{ \tau \geq j\}\), from which, using the coupling lemma for the arbitrary Borel set A of the real line, the estimation
holds.
3 Analysis Methods Based on Markov Property
Definition 4.44.
A discrete-state, continuous-time stochastic process, X(t), possesses the Markov propety at time t n if for all n, m ≥ 1, \(0 \leq {t}_{0} < {t}_{1} < \ldots < {t}_{n} < {t}_{n+1} < \ldots < {t}_{n+m}\), and \({x}_{0},{x}_{1},\ldots ,{x}_{n},{x}_{n+1},\ldots ,{x}_{n+m} \in S\) we have
In this case t n is referred to as a regenerative point.
A commonly applied interpretation of the Markov property is as follows. Assuming that the current time is t n (present), which is a regenerative point, and we know the current state of the process X(t n ), then the future of the stochastic process X(t) for t n ≤ t is independent of the past history of the process X(t) for 0 ≤ t < t n , and it only depends on the current state of the process X(t n ). That is, if one knows the present state, the future is independent of the past.
In the case of discrete-time processes, it is enough to check if the one-step state transitions are independent of the past, i.e., it is enough to check the condition for m = 1.
Usually, we restrict our attention to stochastic processes with nonnegative parameters (positive half of the time axes), and in these cases we assume that t = 0 is a regenerative point.
3.1 Time-Homogeneous Behavior
Definition 4.45.
The stochastic process X(t) is time homogeneous if the stochastic behavior of X(t) is invariant for time shifting, that is, the stochastic behavior of X(t) and \(X^\prime(t) = X(t + s)\) are identical in distribution \(X(t)\mathop{ =}\limits^{ d}X^\prime(t)\).
Corollary 4.46.
If the time-homogeneous stochastic process X(t) possesses the Markov property at time T and X(T) = i, then \(X(t)\mathop{ =}\limits^{ d}X(t - T)\) if X(0) = i.
The corollary states that starting from two different Markov points with the same state results in stochastically identical processes.
4 Analysis of Continuous-Time Markov Chains
Definition 4.47.
The discrete-state, continuous-time stochastic process X(t) is a continuous-time Markov chain (CTMC) if it possesses the Markov property for all t ≥ 0.
Based on this definition and assuming time-homogeneous behavior we obtain the following properties.
Corollary 4.48.
An arbitrary finite-dimensional joint distribution of a CTMC is composed of the product of transition probabilities multiplied by an initial probability.
Corollary 4.49.
For the time points t < u < v the following Chapman–Kolmogorov equation holds:
where \(\hat{{p}}_{ij}(t,u) = \mathbf{P}\left (X(u) = j\mid X(t) = i\right )\) for all i, j ∈ S,0 ≤ t ≤ u, \(\hat{\Pi }(t,u) = \left [\hat{{p}}_{ij}(t,u)\right ]\) . In the case of time-homogeneous processes the time shifts \(u - t = {\tau }_{1}\) and \(v - u = {\tau }_{2}\) play a role:
where \(\Pi (\tau ) = \left [{p}_{ij}(\tau )\right ],\) \({p}_{ij}(\tau ) = \mathbf{P}\left (X(\tau ) = j\mid X(0) = i\right ),\) for all \(i,\ j\,\in \,S,0 \leq \ \tau \) .
Definition 4.50.
The stochastic evolution of a CTMC is commonly characterized by an infinitesimal generator matrix (commonly denoted by \(\mathbf{Q}\)) that can be obtained from the derivative of the state-transition probabilities as follows:
Corollary 4.51.
The sojourn time of a CTMC in a given state i is exponentially distributed with the parameter \({q}_{i} = -{q}_{ii}\) . The probability that after state i the next visited state will be state j is \({q}_{ij}/{q}_{i}\) , and it is independent of the sojourn time in state i.
Remark 4.52.
Based on Corollary 4.51 and the properties of the exponential distribution, the state transitions of a CTMC can also be interpreted in the following way. When the CTMC moves to state i, several exponentially distributed activities start, exactly one for each nonzero transition rate. The time of the activity associated with the state transition from state i to state j is exponentially distributed with the parameter q ij . The CTMC leaves state i and moves to the next state when the first one of these activities completes. The next visited state is the state whose associated activity finishes first.
Corollary 4.53 (Short-term behavior of CTMCs).
During a short time period \(\Delta \) , the behavior of a CTMC is characterized by the following transition probabilities:
-
(a)
\(\mathbf{P}\left (X(t + \Delta ) = i\vert X(t) = i\right ) = 1 - {q}_{i}\Delta + o\left (\Delta \right )\);
-
(b)
\(\mathbf{P}\left (X(t + \Delta ) = j\vert X(t) = i\right ) = {q}_{ij}\Delta + o\left (\Delta \right )\) for i≠j;
-
(c)
\(\mathbf{P}\left (X(t + \Delta ) = j,X(u) = k\vert X(t) = i\right ) = o\left (\Delta \right )\) for i≠k, j≠k, and \(t < u < t + \Delta \) ,
where \(o\left (x\right )\) denotes the set of functions with the property \({\lim }_{x\rightarrow 0}o\left (x\right )/x = 0\) .
According to the corollary, two main events can happen with significant probability during a short time period:
-
The CTMC stays in the initial state during the whole period [(a)].
-
It moves from state i to j [(b)].
The event that more than one state transition happens during a short time period [(c)] has a negligible probability as \(\Delta \rightarrow 0\).
Corollaries 4.51 and 4.53 allow different analytical approaches for the description of the transient behavior of CTMCs.
4.1 Analysis Based on Short-Term Behavior
Let X(t) be a CTMC with state space S, and let us consider the change in state probability \({P}_{i}(t + \Delta ) = \mathbf{P}\left (X(t + \Delta ) = i\right )\) (i ∈ S) considering the possible events during the interval \((t,t + \Delta )\). The following cases must be considered:
-
There is no state transition during the interval \((t,t + \Delta )\). In this case \({P}_{i}(t + \Delta ) = {P}_{i}(t)\), and the probability of this event is \(1 - {q}_{i}\Delta + o\left (\Delta \right )\).
-
There is one state transition during the \((t,t + \Delta )\) interval from state k to state i. In this case \({P}_{i}(t + \Delta ) = {P}_{k}(t)\), and the probability of this event is \({q}_{ki}\Delta + o\left (\Delta \right )\).
-
The process stays in state i at time \(t + \Delta \) such that there is more than one state transition during the interval \((t,t + \Delta )\). The probability of this event is \(o\left (\Delta \right )\).
Considering these cases we can compute \({P}_{i}(t + \Delta )\) from P k (t), k ∈ S, as follows:
from which
Finally, setting the limit \(\Delta \rightarrow 0\) we obtain that
Introducing the row vector of state probabilities P(t) = { P i (t)}, i ∈ S, we obtain the vector-matrix form of the previous equation:
A differential equation describes the evolution of a transient state probability vector. To define the state probabilities, we additionally need to have an initial condition. In practical applications, the initial condition is most often the state probability distribution at time 0, i.e., P(0). The solution of Eq. (4.9) with initial condition P(0) is [55]
Transform Domain Description The Laplace transform of the two sides of Eq. (4.9) gives
from which we can express P ∗ (s) in the following form:
Comparing the time and transform domain expressions we have that eQt and \({[s\mathbf{I} -\mathbf{Q}]}^{-1}\) are Laplace transform pairs of each other.
Stationary Behavior If \({\lim }_{t\rightarrow \infty }{P}_{i}(t)\) exists, then we say that \({\lim }_{t\rightarrow \infty }{P}_{i}(t) = {P}_{i}\) is the stationary probability of state i. In this case, \({\lim }_{t\rightarrow \infty }\mathrm{d}{P}_{i}(t)/\mathrm{d}t = 0\), and the stationary probability satisfies the system of linear equations \(\sum\limits_{k\in S}{q}_{ki}{P}_{k}(t) = 0\) for all k ∈ S.
4.2 Analysis Based on First State Transition
Let X(t) be a CTMC with state space S, and let \({T}_{1},{T}_{2},{T}_{3},\ldots \) denote the time of the first, second, etc. state transitions of the CTMC. We assume that T 0 = 0, and \({\tau }_{1},{\tau }_{2},{\tau }_{3},\ldots \) are the sojourn times spent in the consecutively visited states (\({\tau }_{i} = {T}_{i} - {T}_{i-1}\)). We compute the state-transition probability \({\pi }_{ij}(t) = \mathbf{P}\left (X(t) = j\ \vert \ X(0) = i\right )\) assuming that T 1 = h, i.e., we are interested in
We have
where δ ij is the Kronecker delta ( δ ij = 1 if i = j and δ ij = 0 if i≠j), and \(\frac{{q}_{ik}} {-{q}_{ii}}\) is the probability that after visiting state i the Markov chain moves to state k. In the case of general stochastic processes, this probability might depend on the sojourn time in state i, but in the case of CTMCs, it is independent.
Equation (4.10) has two cases:
-
If the time point of interest, t, is before the first state transition of the CTMC, h ≥ t, then the conditional state-transition probability is either 1 (if the initial and final states are identical i = j) or 0 (if i≠j).
-
If the time point of interest, t, is after the first state transition of the CTMC, T 1 < t, then we can analyze the evolution of the process from T 1 to t using the fact that the process possesses the Markov property at time T 1. In this case we need to consider all possible states that might be visited at time T 1, k ∈ S, k≠i, with the associated probability \(\frac{{q}_{ik}} {-{q}_{ii}}\). The state-transition probabilities from T 1 to t are identical with the state-transition probabilities of the original process from 0 to T 1 − t, assuming that the original process starts from state k.
The distribution of T 1 is known. It is exponentially distributed with the parameter − q ii . Its cumulated and probability density functions are \({F}_{{T}_{1}}(x) = 1 -{\mathrm{e}}^{{q}_{ii}x}\) and \({f}_{{T}_{1}}(x) = -{q}_{ii}{\mathrm{e}}^{{q}_{ii}x}\), respectively. With that we can apply the total probability theorem to compute the (unconditional) state-transition probability π ij (t):
The obtained integral equation is commonly referred to as a Volterra integral equation. Its only unknown is the state-transition probability function π ij (t). The numerical methods developed for the numerical analysis of Volterra integral equations can be used to compute the state-transition probabilities of a CTMC.
Relation of Analysis Methods We can rewrite Eq. (4.11) in the following form:
The derivation of the two sides of Eq. (4.12) according to t is as follows:
where we used Eq. (4.11) for the substitution of the integral expression. The obtained differential equation is similar to that provided by the analysis of the short-term behavior.
Transform Domain Description To relate the two transient descriptions of the CTMC, one with a differential equation and one with an integral equation, we transform these descriptions into a Laplace transform domain. It is easy to take the Laplace transform from the last line of Eq. (4.11) because the second term of the right-hand side is a convolution integral. That is,
Multiplying by the denominator and using that \(-{q}_{ii} =\sum\limits_{k\in S,k\neq i}{q}_{ik}\) we obtain
which can be written in the matrix form
Finally, we have
which is identical to the Laplace transform expression obtained from the differential equation.
Embedded Markov Chain at State Transitions Let X i ∈ S, i = 0, 1, …, denote the ith visited state of the Markov chain X(t), which is the state of the Markov chain in the interval \(({T}_{i},{T}_{i+1})\) (Fig. 4.2). The \({X}_{0},{X}_{1},\ldots \) series of random variables is a discrete-time Markov chain (DTMC) due to the Markov property of X(t). This DTMC is commonly referred to as a Markov chain embedded at the state transitions or simply an embedded Markov chain (EMC). The state-transition probability matrix of the EMC is
Semi-Markov process that possesses the Markov property at the indicated time points
Stationary Analysis Based on the EMC The stationary distribution of the EMC \(\hat{P}\) (which is the solution of \(\hat{P} =\hat{ P}\Pi ,\sum\limits_{i}\hat{{P}}_{i} = 1\)) defines the relative frequency of the visits to the state of the Markov chain. The higher the stationary probability is, the more frequently the state is visited. The stationary behavior of the CTMC X(t) is characterized by two main factors: how often the state is visited (represented by \(\hat{{P}}_{i}\)) and how long a visit lasts. If state i is visited twice as frequently as state j but the mean time of a visit to state i is half the mean time of a visit to j, then the stationary probabilities of states i and j are identical. This intuitive behavior is summarized in the following general rule of renewal theory [58]:
where \(\hat{{\tau }}_{j}\) is the mean time spent in state j, which is known from the diagonal element of the infinitesimal generator, \(\hat{{\tau }}_{j} = -1/{q}_{jj}\).
Discrete-Event Simulation of CTMCs There are at least two possible approaches.
-
When the CTMC is in state i, first draw an exponentially distributed random sample with parameter − q ii for the sojourn time in state i, then draw a discrete random sample for deciding the next visited state with distribution \({\Pi }_{ij}\), j ∈ S.
-
When the CTMC is in state i, draw an exponentially distributed random sample with parameter q ij , say τ ij , for all positive transition rates of row i of the infinitesimal generator matrix. Find the minimum of these samples, \({\min }_{j}{\tau }_{ij}\). The sojourn time in state i is this minimum, and the next state is the one whose associated random sample is minimal.
5 Semi-Markov Process
Definition 4.54.
The discrete-state, continuous-time random process X(t) is a semi-Markov process if it is time homogeneous and it possesses the Markov property at the state-transition instances (Fig. 4.2).
The name semi-Markov process comes from the fact that such processes do not always possess the Markov property (during its sojourn in a state), but there are particular instances (state-transition instances) when they do.
Corollary 4.55.
The sojourn time in state i can be any general real-valued positive random variable. During a sojourn in state i, both the remaining time in both that state and the next visited state depend on the elapsed time since the process entered state i.
Example 4.56.
A two-state (up/down) system fails at a rate λ (the up time of the system is exponentially distributed with parameter λ) and gets repaired at a rate μ. To avoid long down periods, the repair process is stopped and a replacement process is initialized after a deterministic time limit d. The time of the replacement is a random variable with a distribution G(t). Define a system model and check if it is a semi-Markov process.
Because a CTMC always possesses the Markov property, it follows that the sojourn time in a state is exponentially distributed and that the distribution of the next state is independent of the sojourn time. For example, considering the first state transition and the sojourn time in the first state we have
This property does not hold for semi-Markov processes in general. The most important consequences of the definition of semi-Markov processes are the following ones. The sojourn time in a state can have any positive distribution, and the distribution of the next state and the time spent in a state are not independent in general. Consequently, to define a semi-Markov process, this joint distribution must be given. This is usually done by defining the kernel matrix of a process whose i, j element is
Utilizing the time homogeneity of the process we further have for T i that
The analysis of semi-Markov processes is based on the results of renewal theory and the analysis of an EMC (of state-transition instances). The definition of a semi-Markov process requires knowledge of the kernel matrix Q(t) = { Q ij (t)} (for t ≥ 0) and an initial distribution. It is commonly assumed that X(t) possesses the Markov property at time t = 0.
5.1 Analysis Based on State Transitions
Let X(t) ∈ S be a continuous-time semi-Markov process, \({T}_{1},{T}_{2},{T}_{3},\ldots \) the state-transition instances, and \({\tau }_{1},{\tau }_{2},{\tau }_{3},\ldots \) the consecutive sojourn times (\({\tau }_{i} = {T}_{i} - {T}_{i-1}\)). We assume T 0 = 0. We intend to compute the state-transition probability \({\pi }_{ij}(t) = \mathbf{P}\left (X(t) = j\ \vert \ X(0) = i\right )\) assuming that the sojourn in the first state finishes at time h (T 1 = h), that is,
In this case
where \(\mathbf{P}\left (X({T}_{1}) = j\ \vert \ X(0) = i,{T}_{1} = h\right )\) is the probability that the process will start from state i at time 0 and is in state j right after the state transition at time T 1 assuming T 1 = h. In contrast with CTMCs, this probability depends on the sojourn time in state i:
where Q i (h) denotes the distribution of time spent in state i,
It is commonly assumed that state transitions are real, which means that after staying in state i a state transition moves the process to a different state. This means that \({Q}_{ii}(t) = 0,\ \forall i \in S\). It is also possible to consider virtual state transitions from state i to state i, but this does not expand the set of semi-Markov processes and we do not consider it here. Note that the meaning of a diagonal element of a semi-Markov kernel matrix is completely different from that of a diagonal element of an infinitesimal generator of a CTMC. One of the technical consequences of this difference is the fact that we do not need to exclude the diagonal element from the summations over the set of states.
Two cases are considered in Eq. (4.13):
-
If the time point of interest, t, is before the first state transition of the process (h ≥ t), then the conditional state-transition probability is either 0 or 1 depending on the initial and final states. If the initial state i is identical with the final state j, then the transition probability is 1 because there is no state transition up to time t, otherwise it is 0.
-
If the time point of interest, t, is after the first state transition of the process (h < t), then we need to evaluate the distribution of the next state k, assuming that the state transition occurs at time h, and after that the state-transition probability from the new state k to the final state j during time t − h, using the Markov property of the process at time h. The probability that the process moves to state k assuming it occurs at time h is \(\frac{d{Q}_{ij}(h)} {d{Q}_{i}(h)}\), and the probability of its moving from state k to state j during an interval of length t − h is π ij (t − h).
The distribution of the condition of Eq. (4.13) is known. The distribution of the sojourn time in state i is Q i (h). Using the law of total probability we obtain
Similar to the case of CTMCs, analysis based on the first state transition resulted in a Volterra integral equation also in the case of semi-Markov processes. The transient behavior of semi-Markov processes can be computed using the same numerical procedures.
Transform Domain Description We take the Laplace transform of both sides of the Volterra integral Eq. (4.15). The only nontrivial term is a convolution integral on the right-hand side:
where \({q}_{ik}(t) = \mathrm{d}{Q}_{ik}(t)/\mathrm{d}t\) and the transform domain functions are defined as \({f}^{{\ast}}(s) =\int\limits_{0}^{\infty }f(t){\mathrm{e}}^{-st}\mathrm{d}t\).
Introducing the diagonal matrix D ∗ (s) composed of the elements \(1 - {Q}_{i}^{{\ast}}(s)\), that is , \({D}^{{\ast}}(s) =\mathrm{ diag}\langle 1 - {Q}_{i}^{{\ast}}(s)\rangle\), the Laplace transforms of the state transition probabilities are obtained in matrix form,
from which
Stationary Behavior The stationary analysis of a semi-Markov process is very similar to the stationary analysis of a CTMC based on an EMC. Let the transition probability matrix of the EMC be \(\Pi \). It is obtained from the kernel matrix through the following relation:
The stationary distribution of the EMC \(\hat{P}\) is the solution of the linear system \(\hat{P} =\hat{ P}\Pi ,\sum\limits_{i}\hat{{P}}_{i} = 1\). The stationary distribution of the semi-Markov process is
where \(\hat{{\tau }}_{i}\) is the mean time spent in state i. It can be computed from a kernel matrix using \(\hat{{\tau }}_{i} =\int\limits_{0}^{\infty }(1 - {Q}_{i}(t))\mathrm{d}t\).
Discrete-Event Simulation of Semi-Markov Processes The initial distribution and the Q ij (t) kernel completely define the stochastic behavior of a semi-Markov process. As a consequence, it is possible to simulate the process behavior based on them.
The key step of the simulation is to draw dependent samples for the sojourn time and the next visited state. This can be done based on the marginal distribution of one of the two random variables and a conditional distribution of the other one. Depending on which random variable is sampled first, there are two ways to simulate a semi-Markov process:
-
When the process is in state i, first draw a Q i (t) distributed sample for the sojourn time, denoted by τ, then draw a sample for the next state assuming that the sojourn is τ based on the discrete probability distribution \(\mathbf{P}\left (X({T}_{1}) = j\,\vert \,X(0) = i,{T}_{1} = \tau \right )\) (\(\forall j \in S\)) given in Eq. (4.14).
-
When the process is in state i, first draw a sample for the next visited state based on the discrete probability distribution \({\Pi }_{ij} = \mathbf{P}\left (X({T}_{1}) = j\,\vert \,X(0) = i\right )\) (\(\forall j \in S\)), then draw a sample for the sojourn time given in the next state with a distribution
$$\mathbf{P}\left ({T}_{1} \leq t\,\vert \,Z(0) = i,Z({T}_{1}) = j\right ) = \frac{{Q}_{ij}(t)} {{\Pi }_{ij}}\ \cdot $$(4.17)
5.2 Transient Analysis Using the Method of Supplementary Variables
A semi-Markov process does not possess the Markov property during its sojourn in a state. For example, the distribution of the time till the next state transition may depend on the amount of time that has passed since the last state transition. It is possible to extend the analysis of semi-Markov processes so that all information that makes the future evolution of the process conditionally independent of its past history is involved in the process description for \(\forall t \geq 0\). It is indeed the Markov property for \(\forall t \geq 0\). In the case of semi-Markov processes, this means that the discrete state of the process X(t) and the time passed since the last state transition \(Y (t) = t -\max ({T}_{i} \leq t)\) need to be considered together because the vector-valued stochastic process \(\{X(t),Y (t)\}\) is already such that the future behavior of this vector process is conditionally independent of its past given the current value of the vector. That is, the \(\{X(t),Y (t)\}\) process possesses the Markov property for \(\forall t \geq 0\). The behavior of the \(\{X(t),Y (t)\}\) process is depicted in Fig. 4.3.
Analysis of semi-Markov process with supplementary variable
This extension of a random process with an additional variable such that the obtained vector-valued process possesses the Markov property is referred to as the method of supplementary variables [24].
With X(t) and Y (t) and the kernel matrix of the process we can compute the distribution of time till the next state transition at any time instant; this is commonly referred to as the remaining sojourn time in the given state. If at time t the process stays in state i for a period of τ [X(t) = i, Y (t) = τ] and the distribution of the total sojourn time in state i is Q i (t), then the distribution of the remaining sojourn time in state i, denoted by γ, is
where γ t denotes the total time spent in state i during this visit in state i.
To analyze the \(\{X(t),Y (t)\}\) process, we need to characterize the joint distribution of the following two quantities:
It is possible to obtain h i (t, x) based on the analysis of the short-term behavior of CTMCs:
where h i (t + Δ, x | { condition}) denotes \(\frac{\mathbf{P}\left (X(t) = i,x \leq Y (t) < x + \Delta \,\vert \,\text{ condition}\right )} {\Delta }\). The probability of the state transition can be computed based on the distribution of the remaining sojourn time:
from which
Immediately following a state transition Y (t) is reset to zero. Consequently, the probability that \(Y (t + \Delta ) = x\) for a fixed x > 0 is zero when \(\Delta \) is sufficiently small. That is,
It follows that
Analysis of the process \(\{X(t),Y (t)\}\) is made much simpler by the use of the transition rate of α i instead of its distribution Q i (t). The transition rate is defined by
It is also referred to as the hazard rate in probability theory. The probability of a state transition can be written in the following form:
from which
Based on all of these expressions, h i (t, x) satisfies
From this difference equation we can go through the usual steps to obtain the partial differential equation for h i (t, x). First we move \({h}_{i}(t,x - \Delta )\) to the other side,
then we add and subtract h i (t, x),
and reorder the terms,
Finally, the \(\Delta \rightarrow 0\) transition results in
This partial differential equation describes h i (t, x) for x > 0. The case of x = 0 requires a different treatment:
The probability that in the interval \((t,t + \Delta )\) the process moves from state k to state i is
where the second term is already known from Eq. (4.14). We can also introduce the intensity of transition from k to i:
The transition probability can be written in the form
Using this we can write
from which a multiplication with \(\Delta \) and the \(\Delta \rightarrow 0\) transition result in
In summary, the method of supplementary variable allows for the analysis of the process \(\{X(t),Y (t)\}\) through the function h i (t, x), which is given by a partial differential equation (4.18) for x > 0 and a boundary equation (4.19) for x = 0. Based on these equations and the initial distributions of h i (0, x) for \(\forall i \in S\) numerical partial differential solutions methods can be applied to compute the transient behavior of a semi-Markov process.
Stationary Behavior If the limit \({\lim }_{t\rightarrow \infty }{h}_{i}(t,x) = {h}_{i}(x)\) exists for all states i ∈ S, then we can evaluate the limit \(t \rightarrow \infty \) of Eqs. (4.18) and (4.19)
The solution of ordinary differential Eq. (4.20) is
where the unknown quantity is h i (0). It can be obtained from Eq. (4.21) as follows:
where
That is, we are looking for the solution of the linear system
with the normalizing condition
where the normalizing condition is the sum of the stationary-state probabilities. From
and Eq. (4.16) we have that the required solution is
6 Markov Regenerative Process
Definition 4.57.
The X(t) discrete-state, continuous-time, time-homogeneous stochastic process is a Markov regenerative process if there exists a random time series \({T}_{0},{T}_{1},{T}_{2},\ldots \) (T 0 = 0) such that the X(t) process possesses the Markov property at time \({T}_{0},{T}_{1},{T}_{2},\ldots \) [23, 58] (Fig. 4.4).
Compared to the properties of semi-Markov processes, where the process possesses the Markov property at all state-transition points, the definition of Markov regenerative processes is less restrictive. It allows that at some state-transition point the process does not possess the Markov property, but the analysis of Markov regenerative processes is still based on the occurrence of time points where the process possesses the Markov property.
Markov regenerative process; circles denote points with Markov property
Since Definition 4.57 does not address the behavior of the process between the consecutive time points \({T}_{0},{T}_{1},{T}_{2},\ldots \), Markov regenerative processes can be fairly general stochastic processes. In practice, the use of a renewal theorem for the analysis of these processes is meaningful only when the stochastic behavior between the consecutive time points \({T}_{0},{T}_{1},{T}_{2},\ldots \) is easy to analyze.
A common method for analyzing Markov regenerative processes is based on the next time point with the Markov property (T 1).
Definition 4.58.
The series of random variables \(\{{Y }_{n},{T}_{n};n \geq 0\}\) is a time-homogeneous Markov renewal series if
for all n ≥ 0, y ∈ S, and t ≥ 0.
It can be seen from the definition of Markov renewal series that the series \({Y }_{0},{Y }_{1},\ldots \) is a DTMC. According to Definition 4.57, the sequence of states X(T i ) of a Markov regenerative process at the time sequence T i instants with the Markov property and the time sequence T i instants with the Markov property form a Markov renewal sequence \(\{X({T}_{i}),{T}_{i}\}\) (i = 0, 1, …).
Analysis of Markov regenerative processes is based on this embedded Markov renewal series. To this end the joint distribution of the next time point and the state in that time point must be known. In contrast with the similar kernel of semi-Markov processes, in the case of Markov regenerative processes, the kernel is denoted by
and the matrix \(K(t) =\{ {K}_{ij}(t)\}\) is referred to as the global kernel of a Markov regenerative process. The global kernel of a Markov regenerative process completely defines the stochastic properties of the Markov regenerative process at time points with the Markov property. The description of the process between those time points is complex, but for a transient analysis of the process (more precisely for computing transient-state probabilities) it is enough to know the transient-state probabilities between consecutive time points with the Markov property. This is given by the local kernel matrix of the Markov regenerative process E(t) = { E ij (t)} whose elements are
where E ij (t) is the probability that the process will start in state i, the first point with the Markov property will be later than t, and the process will stay in state j at time t.
6.1 Transient Analysis Based on Embedded Markov Renewal Series
Let the transient-state transition probability matrix be \(\Pi (t)\) whose elements are
Assuming that T 1 = h, we can compute the conditional state-transition probability as follows:
Similar to the transient analysis of semi-Markov processes, Eq. (4.22) describes two exclusive cases: h ≤ t and h > t. In the case of semi-Markov processes, the h > t case results in 0 or 1; in the case of a Markov regenerative process, the conditional probability for h > t can be different from 0 or 1 because the process can have state transitions also before T 1.
Using the distribution of T 1 and the formula of total probability we obtain
Let us consider the first term on the right-hand side:
from which
Assuming that \(\mathbf{K}(t)\) is derivable and \(dK(t)/dt = k(t)\) we have
Similar to the transient analysis of CTMCs and semi-Markov processes we obtain a Volterra equation for the transient analysis of Markov regenerative processes.
Transform Domain Description The Laplace transform of Eq. (4.25) is
which can be written in matrix form:
The solution of Eq. (4.27) is
Based on Eq. (4.28), numerical inverse Laplace methods can also be used for the transient analysis of Markov regenerative processes.
Stationary Behavior Despite the differences between semi-Markov and Markov regenerative processes, their stationary analysis follows the same steps. The state-transition probability of the DTMC embedded in time points with the Markov property is
The stationary distribution of the EMC is the solution of \(\hat{P} =\hat{ P}\Pi ,\sum\limits_{i}\hat{{P}}_{i} = 1\). Now we need to compute the mean time spent in the different states during the interval \(({T}_{0},{T}_{1})\). Fortunately, the local kernel carries the necessary information. Let τ ij be the mean time the process spends in state j during the interval \(({T}_{0},{T}_{1})\) assuming that it starts from state i (X(T 0) = i). Then
where \({\mathcal{I}}_{\left \{\bullet \right \}}\) is the indicator of event \(\bullet \). The mean length of the interval \(({T}_{0},{T}_{1})\) is
Finally, the stationary distribution of the process can be computed as
7 Exercises
Exercise 4.1.
Applying Theorem 4.42, find the limit (stationary) distributions of age, residual lifetime, and total lifetime [\(\delta (t) = t - {t}_{N(t)},\) \(\gamma (t) = {t}_{N(t)+1} - t,\) \(\beta (t) = {t}_{N(t)+1} - {t}_{N(t)}\)] if the interarrival times are independent random variables having a joint exponential distribution with the parameter λ. Show the expected values for the limit distributions.
Exercise 4.2 (Ergodic property of semi-Markov processes).
Consider a system with the finite state space \(\mathcal{X} =\{ 1,\ldots ,N\}\). The system begins to work at the moment T 0 = 0 in a state \({X}_{0} \in \mathcal{X}\) and changes states at the random moments \(0 < {T}_{1} < {T}_{2} < \ldots \). Denote by \({X}_{1},{X}_{2},\ldots \) the sequence of consecutive states of the system, and suppose that it constitutes a homogeneous, irreducible, and aperiodic Markov chain with initial distribution \(({p}_{i} = \mathbf{P}\left ({X}_{0} = i\right ),\ 1 \leq i \leq N)\) and probability transition matrix \(\Pi ={ \left ({p}_{ij}\right )}_{i,j=1}^{n}\). Define the process \(X(t) = {X}_{n-1},\ {T}_{n-1} \leq t < {T}_{n},\ n = 1,2,\ldots \), assume that the sequence of holding times \({Y }_{k} = {T}_{k} - {T}_{k-1},\ k = 1,2,\ldots \), depends only conditionally on the states \({X}_{k-1} = i\) and X k = j, and denote \({F}_{ij}(x) = \mathbf{P}\left ({Y }_{k} \leq x\ \vert \ {X}_{k-1} = i,\ {X}_{k} = j\right )\) if p ij > 0, where \({\nu }_{ij} =\int\limits_{0}^{\infty }x\mathrm{d}{F}_{ij}(x) < \infty \).
Find the limits for
-
(a)
The average number of transitions/time;
-
(b)
The relative frequencies of states i in the sequence \({X}_{0},{X}_{1},\ldots \);
-
(c)
The limit distribution \(\mathbf{P}\left ({X}_{t} = i\right ),\ i \in X\);
-
(d)
The average time spent in a state i ∈ X.
References
802.11. IEEE standard for information technology-telecommunications and information exchange between systems-local and metropolitan area networks-specific requirements - part 11: Wireless LAN medium access control (mac) and physical layer (phy) specifications. http://ieeexplore.ieee.org/servlet/opac?punumber=4248376, 2007.
N. Abramson. The aloha system: another alternative for computer communications. In: Proceedings Fall Joint Computer Conference. AFIPS Press, 1970.
D. Aldous, L. Shepp. The least variable phase type distribution is Erlang. Stoch. Models, 3:467–473, 1987.
T. Apostol. Calculus I. Wiley, New York, 1967.
T. Apostol. Calculus II. Wiley, New York, 1969.
J. R. Artalejo, A. Gómez-Corral. Retrial Queueing Systems: A Computational Approach. Springer, Berlin Heidelberg New York, 2008.
S. Asmussen. Applied Probability and Queues. Springer, Berlin Heidelberg New York, 2003.
F. Baccelli, P. Brémaud. Elements of Queueing Theory, Applications of Mathematics. Springer, Berlin Heidelberg New York, 2002.
F. Baskett, K. Mani Chandy, R. R. Muntz, F. G. Palacios. Open, closed and mixed networks of queues with different classes of customers. J. ACM, 22:248–260, 1975.
S. N. Bernstein. Theory of Probabilities. Moskva, Leningrad, 1946. (in Russian).
G. Bianchi. Performance analysis of the IEEE 802.11 distributed coordination function. IEEE J. Select. Areas Commun., 18:535–547, 2000.
D. Bini, G. Latouche, B. Meini. Numerical methods for structured Markov chains. Oxford University Press, Oxford, 2005.
A. Bobbio, M. Telek. A benchmark for PH estimation algorithms: results for Acyclic-PH. Stoch. Models, 10:661–677, 1994.
A. A. Borovkov. Stochastic processes in queueing theory. Applications of Mathematics. Springer, Berlin Heidelberg New York, 1976.
A. A. Borovkov. Asymptotic Methods in Queueing Theory. Wiley, New York, 1984.
L. Breuer, D. Baum. An Introduction to Queueing Theory and Matrix-Analytic Methods. Springer, Berlin Heidelberg New York, 2005.
P. J. Burke. The output of a queuing system. Oper. Res., 4:699–704, 1956.
J. Buzen. Computational algorithms for closed queueing networks with exponential servers. Commun. ACM, 16:527–531, 1973.
V. Ceric, L. Lakatos. Measurement and analysis of input data for queueing system models used in system design. Syst. Anal. Modell. Simul., 11:227–233, 1993.
Hong Chen, David D. Yao. Fundamentals of Queueing Networks: Performance, Asymptotics, and Optimization. Springer, Berlin Heidelberg New York, 2001.
Y. Chow, H. Teicher. Probability Theory. Springer, Berlin Heidelberg New York, 1978.
K. Chung. Markov chains with stationary transition probabilities. Springer, Berlin Heidelberg New York, 1960.
E. Cinlar. Introduction to Stochastic Processes. Prentice-Hall, Englewood Cliffs, NJ, 1975.
D. R. Cox. The analysis of non-Markovian stochastic processes by the inclusion of supplementary variables. Proc. Cambridge Philos. Soc., 51:433–440, 1955.
A. Cumani. On the canonical representation of homogeneous Markov processes modelling failure-time distributions. Microelectron. Reliab., 22:583–602, 1982.
D.J. Daley, D. Vere-Jones. An Introduction to the Theory of Point Process. Springer, Berlin Heidelberg New York, 2008. 2nd edn.
Gy. Dallos, Cs. Szabó. Random access methods of communication channels. Akadémiai Kiadó, Budapest, 1984 (in Hungarian).
M. De Prycker. Asynchronous Transfer Mode, Solutions for Broadband ISDN. Prentice Hall, Englewood Cliffs, NJ, 1993.
P. Erdős, W. Feller, H. Pollard. A theorem on power series. Bull. Am. Math. Soc., 55:201–203, 1949.
G. I. Falin, J. G. C. Templeton. Retrial queues. Chapman and Hall, London, 1997.
W. Feller. An Introduction to Probability Theory and its Applications, vol. I. Wiley, New York, 1968.
Chuan Heng Foh, M. Zukerman. Performance analysis of the IEEE 802.11 MAC protocol. In Proceedings of European wireless conference, Florence, February 2002.
F. G. Foster. On the stochastic matrices associated with certain queuing processes. Ann. Math. Stat., 24:355–360, 1953.
G. Giambene. Queuing Theory and Telecommunications: Networks and Applications. Springer, Berlin Heidelberg New York, 2005.
I.I. Gihman, A.V. Skorohod. The Theory of Stochastic Processes, vol. I. Springer, Berlin Heidelberg New York, 1974.
I. I. Gihman, A. V. Skorohod. The Theory of Stochastic Processes, vol. II. Springer, Berlin Heidelberg New York, 1975.
B. Gnedenko, E. Danielyan, B. Dimitrov, G. Klimov, V. Matveev. Priority Queues. Moscow State University, Moscow, 1973 (in Russian).
B. V. Gnedenko. Theory of Probability. Gordon and Breach, Amsterdam, 1997. 6th edn.
B. V. Gnedenko, I. N. Kovalenko. Introduction to Queueing Theory, 2nd edn. Birkhauser, Boston 1989.
W. J. Gordon, G. F. Newell. Closed queueing systems with exponential servers. Oper. Res., 15:254–265, 1967.
D. Gross, J. F. Shortle, J. M. Thompson, C. M. Harris. Fundamentals of Queueing Theory, 4th edn. Wiley, New York, 2008.
W. Henderson. Alternative approaches to the analysis of the M/G/1 and G/M/1 queues. J. Oper. Res. Soc. Jpn., 15:92–101, 1972.
A. Horváth, M. Telek. PhFit: A general purpose phase type fitting tool. In Tools 2002, pages 82–91, London, April 2002. Lecture Notes in Computer Science, vol. 2324. Springer, Berlin Heidelberg New York.
J. R. Jackson. Jobshop-like queueing systems. Manage. Sci., 10:131–142, 1963.
N. K. Jaiswal. Priority Queues. Academic, New York, 1968.
N. L. Johnson, S. Kotz. Distributions in Statistics: Continuous Multivariate Distributions. Applied Probability and Statistics. Wiley, New York, 1972.
V.V. Kalashnikov. Mathematical Methods in Queueing Theory. Kluwer, Dordrecht, 1994.
S. Karlin, H. M. Taylor. A First Course in Stochastic Processes. Academic, New York, 1975.
S. Karlin, H. M. Taylor. A Second Course in Stochastic Processes. Academic, New York, 1981.
J. Kaufman. Blocking in a shared resource environment. IEEE Trans. Commun., 29: 1474–1481, 1981.
F. P. Kelly. Reversibility and Stochastic Networks. Wiley, New York, 1979.
D. G. Kendall. Stochastic processes occurring in the theory of queues and their analysis by the method of the imbedded Markov chain. Ann. Math. Stat., 24:338–354, 1953.
A. Khinchin. Mathematisches über die Erwartung vor einem öffentlichen Schalter. Rec. Math., 39:72–84, 1932 (in Russian with German summary).
J. F. C. Kingman. Poisson Processes. Clarendon, Oxford, 1993.
L. Kleinrock. Queuing Systems. Volume 1: Theory. Wiley-Interscience, New York, 1975.
G. P. Klimov. Extremal Problems in Queueing Theory. Energia, Moskva, 1964 (in Russian).
E. V. Koba. On a retrial queueing system with a FIFO queueing discipline. Theory Stoch. Proc., 8:201–207, 2002.
V. G. Kulkarni. Modeling and Analysis of Stochastic Systems. Chapman & Hall, London, 1995.
L. Lakatos. On a simple continuous cyclic waiting problem. Annal. Univ. Sci. Budapest Sect. Comp., 14:105–113, 1994.
L. Lakatos. A note on the Pollaczek-Khinchin formula. Annal. Univ. Sci. Budapest Sect. Comp., 29:83–91, 2008.
L. Lakatos. Cyclic waiting systems. Cybern. Syst. Anal., 46:477–484, 2010.
G. Latouche, V. Ramaswami. Introduction to matrix analytic methods in stochastic modeling. SIAM, 1999.
A. Lewandowski. Statistical tables. http://www.alewand.de. Nov. 13., 2012.
D. V. Lindley. The theory of queues with a single server. Math. Proc. Cambridge Philos. Soc., 48:277–289, 1952.
T. Lindwall. Lectures on the Coupling Method. Wiley, New York, 1992.
J. D. C. Little. A proof of the queuing formula: L =AW. Oper. Res., 9:383–387, 1961.
A. A. Markov. Rasprostranenie zakona bol’shih chisel na velichiny, zavisyaschie drug ot druga. Izvestiya Fiziko-matematicheskogo obschestva pri Kazanskom universitete, 15:135–156, 1906 (in Russian).
L. Massoulie, J. Roberts. Bandwidth sharing: Objectives and Algorithms. In Infocom, 1999.
V. F. Matveev, V. G. Ushakov. Queueing systems. Moscow State University, Moskva, 1984 (in Russian).
P. Medgyessy, L. Takács. Probability Theory. Tankönyvkiadó, Budapest, 1973 (in Hungarian).
S. Meyn, R. Tweedie. Markov chains and stochastic stability. Springer, Berlin Heidelberg New York, 1993.
NIST: National Institute of Standards and Technology. Digital library of mathematical functions. http://dlmf.nist.gov. Nov. 13., 2012.
M. Neuts. Probability distributions of phase type. In Liber Amicorum Prof. Emeritus H. Florin, pp. 173–206. University of Louvain, Louvain, Belgium, 1975.
M.F. Neuts. Matrix Geometric Solutions in Stochastic Models. Johns Hopkins University Press, Baltimore, 1981.
C. Palm. Methods of judging the annoyance caused by congestion. Telegrafstyrelsen, 4:189–208, 1953.
A. P. Prudnikov, Y. A. Brychkov, O. I. Marichev. Integrals and series, vol. 2. Gordon and Breach, New York, 1986. Special functions.
S. Rácz, M. Telek, G. Fodor. Call level performance analysis of 3rd generation mobile core network. In IEEE International Conference on Communications, ICC 2001, 2:456–461, Helsinki, Finland, June 2001.
S. Rácz, M. Telek, G. Fodor. Link capacity sharing between guaranteed- and best effort services on an atm transmission link under GoS constraints. Telecommun. Syst., 17(1–2):93–114, 2001.
M. Reiser, S. S. Lavenberg. Mean value analysis of closed multi-chain queueing networks. J. ACM, 27:313–322, 1980.
J. Roberts. A service system with heterogeneous user requirements - application to multi-service telecommunications systems. In Proceedings of Performance of Data Communications Systems and Their Applications, pp. 423–431, Paris, 1981.
K. W. Ross. Multiservice Loss Models for Broadband Telecommunication Networks. Springer, Berlin Heidelberg New York, 1995.
T. Saaty. Elements of Queueing Theory. McGraw-Hill, New York, 1961.
R. Serfozo. Introduction to Stochastic Networks. Springer, Berlin Heidelberg New York, 1999.
A. N. Shiryaev. Probability. Springer, Berlin Heidelberg New York, 1994.
D. L. Snyder. Random Point Processes. Wiley, New York, 1975.
L. Szeidl. Estimation of the moment of the regeneration period in a closed central-server queueing network. Theory Probab. Appl., 31:309–313, 1986.
L. Szeidl. On the estimation of moment of regenerative cycles in a general closed central-server queueing network. Lect. Notes Math., 1233:182–189, 1987.
L. Takács. Investigation of waiting time problems by reduction to Markov processes. Acta Math. Acad. Sci. Hung., 6:101–129, 1955.
L. Takács. The distribution of the virtual waiting time for a single-server queue with Poisson input and general service times. Oper. Res., 11:261–264, 1963.
L. Takács. Combinatorial Methods in the Theory of Stochastic Processes. Wiley, New York, 1967.
H. Takagi. Queueing Analysis. North Holland, Amsterdam, 1991.
M. Telek. Minimal coefficient of variation of discrete phase type distributions. In G. Latouche, P. Taylor, eds., Advances in algorithmic methods for stochastic models, MAM3, pp. 391–400. Notable Publications, 2000.
A. Thümmler, P. Buchholz, M. Telek. A novel approach for fitting probability distributions to trace data with the em algorithm. IEEE Trans. Depend. Secure Comput., 3(3):245–258, 2006. Extended version of DSN 2005 paper.
H. Tijms. Stochastic Models: An Algorithmic Approach. Wiley, New York, 1994.
W. Whitt. A review of l = λw and extensions. Queue. Syst., 9:235–268, 1991.
V. M. Zolotarev. Modern Theory of Summation of Random Variables. VSP, Utrecht, 1997.
Author information
Authors and Affiliations
Rights and permissions
Copyright information
© 2013 Springer Science+Business Media New York
About this chapter
Cite this chapter
Lakatos, L., Szeidl, L., Telek, M. (2013). Renewal and Regenerative Processes. In: Introduction to Queueing Systems with Telecommunication Applications. Springer, Boston, MA. https://doi.org/10.1007/978-1-4614-5317-8_4
Download citation
DOI: https://doi.org/10.1007/978-1-4614-5317-8_4
Published:
Publisher Name: Springer, Boston, MA
Print ISBN: 978-1-4614-5316-1
Online ISBN: 978-1-4614-5317-8
eBook Packages: Mathematics and StatisticsMathematics and Statistics (R0)