1 Introduction

Optimal insurance policies have been extensively studied in the literature. The commonly used optimality criteria are risk minimization, expected utility (EU) maximization, or some combination of them. Borch [10] pioneered the study of optimal insurance (reinsurance) design that minimizes risk. He proved that the excess-of-loss insurance policy can minimize the variance of the insured’s total loss. Other risk measures are considered more recently. For example, Cai et al. [12] and Bernard and Tian [7] proposed to study the optimal insurance policy that minimizes the insured’s value-at-risk (VaR) or tail value-at-risk (TVaR). The results for optimal insurance policies that minimize the distortion risk measures can be found in, for example, Assa [5], Zhuang et al. [36], Lo [28], Cheung et al. [15] and the references therein.

Arrow [1] applied the EU theory in determining the optimal reinsurance contract. His model was extended along many directions in the past few decades. To name a few, Raviv [30] derived the optimal policy that maximizes the insured’s EU under the participation constraint of the insurer. Recently, Bernard et al. [9] and Xu et al. [33] considered an optimal insurance design problem for an individual whose preference is dictated by the rank-dependent expected utility. Other results related to the optimal insurance design, such as those considering heterogeneous beliefs and higher-order risk attitudes, can be found in Ghossoub [22], Chi and Wei [18], Chi [16] and the references therein.

In practice, an EU-maximizing insured may be subject to risk constraints. To the authors’ best knowledge, the literature for optimal policy that maximizes EU under risk constraints are rather thin. We list some references below. Huang [25] studied the optimal reinsurance contract by maximizing the insured’s EU subject to its VaR constraint. Zhou and Wu [35] revisited the problem of Huang [25] by considering the counter-party’s VaR constraint. Bernard and Tian [8] studied the optimal contract from both the insured’s and insurer’s perspectives under the insurer’s VaR constraint. However, as shown in Huang [25] and Bernard and Tian [8], the optimal indemnity functions are usually discontinuous if there are no restrictions on their forms. This would give rise to ex post moral hazard issues. Particularly, an upward jump in indemnity function gives insured incentive to over-report losses, whereas a downward jump incentivizes insured to under-report losses.

The problem of ex post moral hazard in insurance policy design has attracted much attention in the economics literature. For example, Huberman et al. [26] suggested “the search for an optimal indemnity schedule can be confined to those schedules under which the insured has no incentive to misrepresent the damage”. In particular, it was pointed out that ex post moral hazard problem can be excluded if the both ceded and retained loss functions are non-decreasing. This condition is called incentive compatibility and has been widely adopted in optimal insurance design. See, for example, See, for example, Bernard et al. [9], Asimit and Boonen [4], Xu et al. [33], Chi and Zhuang [20], Tan et al. [31] and references therein. As pointed out in Tan et al. [31], the incentive compatibility condition also rules out indemnity functions that have slope greater than one or less than zero, which could also lead to ex post moral hazard. We remark here that the incentive compatibility condition is also referred to as the no-sabotage condition as per Carlier and Dana [13].

In this paper, we study the optimal insurance policy that maximizes the EU of a DM (could be either the insured or the insurer) under the distortion-risk-measure-based risk constraints. Our model is different from those in the literature on the following aspects: first, we impose the incentive compatibility condition on admissible indemnity functions, which mitigates the ex post moral hazard problems; second, we adopt the general distortion-risk-measure-based risk constraints for both parties in the transaction. Our main results are summarized in the following. First, we provide implicit characterization of the optimal indemnity function when the risk constraints are defined through general distortion risk measures. Second, we derive explicit forms of the optimal indemnity function when the risk constraints are defined through VaR and TVaR. Third, we demonstrate the effect of the incentive compatibility condition on optimal insurance policies in great detail. We find that with the VaR risk constraints, the DM chooses to ignore the potential large losses that do not contribute to its VaR value and only manages the risk that contributes to its VaR. This problem is alleviated under the TVaR risk constraints. Fourth, we explore the optimal policy when the risk constraints are defined in terms of other distortion risk measures, such as the proportional hazard (PH) transform introduced by Wang [32].

The remainder of this paper is structured as follows. Section 2 reviews some preliminaries and sets up the problem. Section 3 provides a general solution to the main problem. Section 4 derives the closed-form optimal indemnity functions when the risk constraints are defined by specific risk measures. Section 5 provides numerical examples and demonstrates the implications of our results. Section 6 concludes.

2 Model setup

Suppose that a risk-averse insured is endowed with initial wealth \(w_d\) and an increasing and strictly concave utility function u. The insured faces a ground-up loss X whose support is [0, M] with \(M\le \infty \). The probability density function (PDF), cumulative distribution function (CDF) and survival function of X are denoted by \(f_X(\cdot )\), \(F_X(\cdot )\) and \(S_X(\cdot )\) respectively.

The insured is negotiating with an insurer, who is endowed with initial wealth \(w_r\) and an increasing and strictly concave utility function v, for an insurance policy, which will pay the insured I(X) for a premium \(\pi (I)\). To ensure that the indemnity functions satisfy the incentive compatibility condition of Huberman et al. [26], we follow the literature (e.g., Xu et al. [33] and Chi and Zhuang [20]) by assuming that the set of admissible indemnity functions is given by

figure a

For any function \(I\in \mathcal {C}_0\), the functions I(x) and \(R(x)=x-I(x)\) are nondecreasing and therefore the ceded loss I(X) and retained loss R(X) are comonotonic. Further, any function \(I\in \mathcal {C}_0\) is 1-Lipschitz continuous and therefore differentiable almost everywhere [17]. As shown in Zhuang et al. [36], the function \(I\in \mathcal {C}_0\) admits the integral representation

$$\begin{aligned} I(x)=\int _{0}^{x}\eta (t)dt, \end{aligned}$$
(2.1)

where \(\eta (\cdot )\) is called the marginal indemnity function (MIF) and belongs to the set

$$\begin{aligned} \tilde{\mathcal {C}}_0:=\left\{ \eta :[0,M]\rightarrow [0,1]\ \Big |\ 0\le \eta (x)\le 1\right\} . \end{aligned}$$

To simplify discussions, we assume that the insurance premium is determined by the expectation principle

$$\begin{aligned} \pi (I)=(1+\theta )\mathbf {E}[I(X)] \end{aligned}$$

where \(\theta \ge 0\) is the safety loading. However, from the analysis presented in the next section, it can be seen that our methodology applies when premium is determined by arbitrary actuarial premium principle \(\pi (I)=\phi (\mathbf {E}[I(X)])\), where \(\phi (x)\) is a general strictly increasing function (as in Bernard and Tian [8]), or by general distortion principle (as in Escobar and Pflug [21]).

In the following, we study the optimal insurance policy under the distortion risk constraints from both the insured’s and insurer’s perspectives. Before introducing the main problem, we give a very brief introduction to distortion risk measures.

2.1 Distortion risk measures

Distortion risk measures have been extensively studied and widely applied in actuarial and risk management literature. The distortion risk measure of a non-negative random variable X is defined by

$$\begin{aligned} \rho (X)=\int _{0}^{M}g\left( S_X(x)\right) dx, \end{aligned}$$

where \(g:[0,1]\rightarrow [0,1]\) is called the distortion function, which is non-decreasing and satisfies \(g(0)=0\) and \(g(1)=1\).

It is well known that VaR and TVaR are special cases of distortion risk measures.

Definition 2.1

The VaR of a random variable X at confidence level \(\alpha \in (0,1)\) is given by

$$\begin{aligned} \text {VaR}_{\alpha }(X)=\inf \left\{ x: F_X(x)\ge \alpha \right\} . \end{aligned}$$

The distortion function for VaR is given by

$$\begin{aligned} g_V(x)=\mathbbm {1}_{[1-\alpha ,1]}(x). \end{aligned}$$
(2.2)

where \(\mathbbm {1}_S(x)\) is the indicator function which equals to 1 if \(x\in S\) and 0 otherwise.

Definition 2.2

The TVaR of a random variable X at confidence level \(\alpha \in (0,1)\) is given by

$$\begin{aligned} \text {TVaR}_{\alpha }(X)=\frac{1}{1-\alpha }\int _{\alpha }^{1}\text {VaR}_s(X)ds. \end{aligned}$$

The distortion function for TVaR is

$$\begin{aligned} g_T(x)=\frac{x}{1-\alpha }\cdot \mathbbm {1}_{[0,1-\alpha )}(x)+\mathbbm {1}_{[1-\alpha ,1]}(x). \end{aligned}$$
(2.3)

As proved rigorously in Zhuang et al. [36] and Cheung and Lo [14], the distortion risk measure of I(X) admits the following representation

$$\begin{aligned} \rho (I(X))=\int _{0}^{M}g\left( S_X(t)\right) dI(t)=\int _{0}^{M}g\left( S_X(t)\right) \eta (t)dt. \end{aligned}$$

Now we are ready to present this paper’s main problems.

2.2 Main problems

First, with an insurance policy characterized by the indemnity function I(x), the insured’s utility is given by

$$\begin{aligned} \mathcal {J}_1(I)=u\left( w_d-X+I(X)-\pi (I)\right) . \end{aligned}$$

and the insurer’s utility is

$$\begin{aligned} \mathcal {J}_2(I)=v\left( w_r-I(X)+\pi (I)\right) . \end{aligned}$$

The insured’s goal is to determine the optimal indemnity function that maximizes its expected utility subject to the risk constraints that are defined through distortion risk measures \(\rho _d\) and \(\rho _r\) respectively for the insured and insurer. The distortion functions corresponding to \(\rho _d\) and \(\rho _r\) are \(g_d\) and \(g_r\) respectively. For some predetermined risk tolerance levels \(A_d\) and \(A_r\), the insured’s decision problem is given by

Problem 1

(The insured’s decision problem)

$$\begin{aligned} \max _{I\in \mathcal {C}_0}&\ \mathbf {E}\left[ \mathcal {J}_1(I)\right] , \\ \text {s.t.}&\ \rho _d\left( X-I(X)+\pi (I)\right) \le A_d, \\&\ \rho _r\left( I(X)-\pi (I)\right) \le A_r, \\&\ \pi (I)\le \pi _0, \end{aligned}$$

where \(\pi _0\) is the insured’s maximal budget.

From the insurer’s perspective, the decision problem is

Problem 2

(The insurer’s decision problem)

$$\begin{aligned} \max _{I\in \mathcal {C}_0}& \, \mathbf {E}\left[ \mathcal {J}_2(I)\right] \\ \text {s.t.}&\ \rho _d\left( X-I(X)+\pi (I)\right) \le A_d, \\&\ \rho _r\left( I(X)-\pi (I)\right) \le A_r, \\&\ \pi (I)\le \pi _0. \end{aligned}$$

3 The optimal indemnity function

In this section, we solve both the insured’s and insurer’s decision problems analytically.

3.1 The insured’s decision problem

To solve Problem 1, we adopt a two-step procedure. We first fix the premium at some level \(B\in (0,\pi _0]\) and solve

Problem 1a

(Problem 1 with fixed premium)

$$\begin{aligned} \max _{I\in \mathcal {C}_0}&\ \mathbf {E}\left[ \mathcal {J}_1(I)\right] , \end{aligned}$$
(3.1)
$$\begin{aligned} \text {s.t.}&\ \rho _d\left( X-I(X)+\pi (I)\right) \le A_d, \end{aligned}$$
(3.2)
$$\begin{aligned}&\ \rho _r\left( I(X)-\pi (I)\right) \le A_r, \end{aligned}$$
(3.3)
$$\begin{aligned}&\ \pi (I)=B, \end{aligned}$$
(3.4)

where \(B\in [0,\pi _0]\).

Then we search for the optimal premium level \(B^*\) numerically within \([0,\pi _0]\). Such a two-step procedure is widely applied in the literature [3, 17].

Let

$$\begin{aligned} \mathcal {C}_1&:=\left\{ I: I\in \mathcal {C}_0, \ \pi (I)=B, \text { and }\rho _d\left( X-I(X)+B\right) \le A_d\right\} , \\ \mathcal {C}_2&:=\left\{ I: I\in \mathcal {C}_0,\, \pi (I)=B, \text {and }\rho _r\left( I(X)-B\right) \le A_r\right\} . \end{aligned}$$

To ensure that the set of admissible indemnity functions is non-empty, we assume that \(\mathcal {C}_1\cap \mathcal {C}_2\ne \emptyset \) for at least some B in the remaining parts of this paper.

In practice, some necessary conditions for the compatibility of these constraints could be easily checked. For example, in order for \(\mathcal {C}_1\cap \mathcal {C}_2\ne \emptyset \), we must have

$$\begin{aligned} \rho _d\left( X-I(X)\right) +\rho _r\left( I(X)\right) \le A_d+A_r, \end{aligned}$$

which means that the sum of risks borne by the two parties must be smaller than the total tolerance level. Otherwise, the problem has no solution.

Problem 1a generalizes the models in Huang [25] and Bernard and Tian [8]. Without the incentive compatibility condition, Problem 1a could be solved using a number of different approaches, such as the calculus of variations, point-wise optimization, stochastic ordering, etc. These methods have been frequently used in the optimal (re)insurance literature. For example, the use of calculus of variations can be found in Golubin [23], Golubin [24], Lo [28], Chi and Zhuang [20]. The point-wise maximization approach were applied in, for example, Bernard and Ludkovski [6], Ghossoub [22], Zhang et al. [34] and Jiang et al. [27]. Examples of using stochastic ordering approach can be found in Cai and Wei [11], Lu et al. [29], Chi and Wei [18].

With the incentive compatibility condition, the previously mentioned methods have to be modified accordingly. In this paper, similar to the methodology applied by Chi and Wei [19] and Chi and Zhuang [20] in different contexts, we apply the calculus of variations with some modification. To this end, suppose that \(I^*\) is the solution to Problem 1a. Then, for \(I\in \mathcal {C}_1\cap \mathcal {C}_2\) and \(\epsilon \in [0,1]\), the convex combination \(\epsilon I^*+(1-\epsilon )I\) belongs to \(\mathcal {C}_1\cap \mathcal {C}_2\).

Define

$$\begin{aligned} \mathcal {H}(\epsilon )=\mathbf {E}[\mathcal {J}_1(\epsilon I^*(X)+(1-\epsilon )I(X))]. \end{aligned}$$

It is easy to verify that \(\mathcal {H}''(\epsilon )<0\) due to the strict concavity of \(u(\cdot )\). Therefore, a sufficient and necessary condition for \(I^*\) to be the optimal solution of Problem 1a is that

$$\begin{aligned} \begin{aligned} \mathcal {H}'(\epsilon )\big |_{\epsilon =1}&=\mathbf {E}[u'(w-X+I^*(X)-B)(I^*(X)-I(X))]\ge 0 \\&\Longrightarrow \mathbf {E}[u'(w-X+I^*(X)-B)I^*(X)]\ge \mathbf {E}[u'(w-X+I^*(X)-B)I(X)]. \end{aligned} \end{aligned}$$

In other words, the optimal indemnity function \(I^*\) solves

$$\begin{aligned} \max _{I\in \mathcal {C}_1\cap \mathcal {C}_2}\ \mathbf {E}[u'(w-X+I^*(X)-B)I(X)], \end{aligned}$$

where the objective function can be rewritten as (see also equation (3.2) of Chi and Zhuang [20])

$$\begin{aligned} \begin{aligned}&\mathbf {E}[u'(w-X+I^*(X)-B)I(X)] \\&\quad =\int _{0}^{M}u'(w-x+I^*(x)-B)\left\{ \int _{0}^{x}\eta (t)dt\right\} dF_X(x) \\&\quad =\int _{0}^{M}\left\{ \int _{t}^{M}u'(w-x+I^*(x)-B)dF_X(x) \right\} \eta (t)dt, \end{aligned} \end{aligned}$$

where the second equation is due to the Fubini’s Theorem.

Applying the comonotonic additivity and translation invariance properties of distortion risk measures, the risk constraints (3.2) and (3.3) boil down to

$$\begin{aligned} \rho _d(I(X))= \int _{0}^{M}g_d\left( S_X(t)\right) \eta (t)dt\ge C_d, \end{aligned}$$

where \(C_d=\rho _d(X)+B-A_d\), and

$$\begin{aligned} \rho _r(I(X))=\int _{0}^{M}g_r\left( S_X(t)\right) \eta (t)dt\le C_r, \end{aligned}$$

where \(C_r=A_r+B\).

The budget constraint (3.4) becomes

$$\begin{aligned} \int _{0}^{M}S_X(t)\eta (t)dt=\frac{B}{1+\theta }. \end{aligned}$$

To sum up, solving Problem 1a is equivalent to solving Problem 1b (Another form of Problem 1a)

$$\begin{aligned}&\max _{\eta \in \tilde{\mathcal {C}}_0} \int _{0}^{M}\left\{ \int _{t}^{M}u'(w-x+I^*(x)-B)dF_X(x) \right\} \eta (t)dt \\&\text {s.t.}\ \int _{0}^{M}g_d\left( S_X(t)\right) \eta (t)dt\ge C_d, \\&\ \quad \int _{0}^{M}g_r\left( S_X(t)\right) \eta (t)dt\le C_r, \\& \quad \int _{0}^{M}S_X(t)\eta (t)dt=\frac{B}{1+\theta }. \end{aligned}$$

By adopting the Lagrangian dual approach [36], along with the point-wise maximization, we obtain the following result. Its proof is given in the appendix.

Theorem 3.1

Assume that the set of admissible indemnity functions for Problem 1b is non-empty. Let

$$\begin{aligned} L(t; I^*, \lambda _1,\lambda _2,\lambda _3)=\int _{t}^{M}u'(w-x+I^*(x)-B)dF_X(x)+\lambda _1g_d\left( S_X(t)\right) -\lambda _2 g_r\left( S_X(t)\right) +\lambda _3S_X(t), \end{aligned}$$
(3.5)

where \(\lambda _1, \lambda _2\in \mathbf {R}^+\) and \(\lambda _3\in \mathbf {R}.\) Then a function \(\eta ^*(t)\) solves the Problem 1b if and only if

$$\begin{aligned} \eta ^*(t)=\mathbbm {1}_{D}(t)+\xi (t)\cdot \mathbbm {1}_{E}(t), \end{aligned}$$
(3.6)

where

$$\begin{aligned} D= & {} \left\{ t: L(t; I^*, \lambda _1,\lambda _2,\lambda _3)>0\right\} ,\\ E= & {} \left\{ t: L(t; I^*, \lambda _1,\lambda _2,\lambda _3)=0 \right\} , \end{aligned}$$

\(\mathbbm {1}_{A}(t)\) is an indicator function and \(\xi (t)\in [0,1]\) is any function such that \(\eta ^*\in \tilde{\mathcal {C}}_0\). The parameters \(\lambda _1\), \(\lambda _2\), and \(\lambda _3\) are determined by the slackness conditions

$$\begin{aligned}&\lambda _1\left( \int _{0}^{M}g_d\left( S_X(t)\right) \eta ^*(t)dt- C_d\right) =0, \\&\lambda _2\left( \int _{0}^{M}g_r\left( S_X(t)\right) \eta ^*(t)dt- C_r\right) =0, \end{aligned}$$

and

$$\begin{aligned} \lambda _3\left( \int _{0}^{M}S_X(t)\eta ^*(t)dt-\frac{B}{1+\theta }\right) =0. \end{aligned}$$

Note that the indemnity function characterized by Theorem 3.1 is implicit since the sets D and E both contain \(I^*\). Nevertheless, it sheds light on the form of the optimal indemnity function. For example, if the insured’s risk constraint is binding, then \(\lambda _1>0\), which increases the value of function L defined in (3.5). This leads to a larger set D and consequently more insurance coverage (\(\eta ^*(t)=1\)) than that without the risk constraint. On the other hand, if the insurer’s risk constraint is binding, then \(\lambda _2>0\), which decreases the value of function L and leads to less insurance coverage. We will study in detail this observation in the next section where the risk measures are VaR and TVaR.

Remark 3.1

For the uniqueness of solution to Problem 1a (or 1b), we refer the interested readers to Chi and Zhuang [20] (Lemma 2.1) for rigorous and detailed discussions of a similar problem.

3.2 The insurer’s decision problem

We now solve Problem 2. The methodology is rather close to that in the last section, so we only provide an outline.

Similar to Sect. 3.1, we first fix the premium level, i.e. \(\pi (I)=B\), and rewrite Problem 2 as

Problem 2a (Problem 2 with fixed premium)

$$\begin{aligned}&\min _{\eta \in \tilde{\mathcal {C}}_0} \int _{0}^{M}\left\{ \int _{t}^{M}v'(w_r+B-I^*(x))dF_X(x) \right\} \eta (t)dt \\&\text {s.t.}\ \int _{0}^{M}g_d\left( S_X(t)\right) \eta (t)dt\ge C_d, \\&\ \quad \int _{0}^{M}g_r\left( S_X(t)\right) \eta (t)dt\le C_r, \\& \quad \int _{0}^{M}S_X(t)\eta (t)dt=\frac{B}{1+\theta }. \end{aligned}$$

Further, for \(\lambda _1, \lambda _2\in \mathbf {R}^+\) and \(\lambda _3\in \mathbf {R}\), define

$$\begin{aligned} \tilde{L}(t; I^*, \lambda _1,\lambda _2,\lambda _3)= & {} \int _{t}^{M}u'(w_r-I^*(x)+B)dF_X(x)-\lambda _1g_d\left( S_X(t)\right) \nonumber \\&+\lambda _2 g_r\left( S_X(t)\right) +\lambda _3 S_X(t),\nonumber \\ \tilde{D}= & {} \left\{ t: \tilde{L}(t; I^*, \lambda _1,\lambda _2, \lambda _3)<0\right\} , \end{aligned}$$
(3.7)

and

$$\begin{aligned} \tilde{E}=\left\{ t: \tilde{L}(t; I^*, \lambda _1,\lambda _2, \lambda _3)=0 \right\} . \end{aligned}$$

Then we have

Corollary 3.1

The solution to Problem 2a can be obtained through Theorem 3.1 by replacing sets D with \(\tilde{D}\) and E with \(\tilde{E}\).

Remark 3.2

We observe that when the insurer’s EU is to be maximized, a binding risk constraint on the insurer’s side leads to a larger value of function \(\tilde{L}\) and thus less insurance coverage. A binding risk constraint on the insured’s side leads to a smaller value of function \(\tilde{L}\) and thus more insurance coverage.

4 Specific cases

As VaR and TVaR are commonly used to determine regulatory capital requirement, in this section, we first focus on these two risk measures and derive explicit formulas for the optimal indemnity functions. We also provide a closed-form optimal indemnity function for the insured’s decision problem where the insured’s risk constraint is defined by the PH transform and binding.

For the VaR/TVaR case, the following assumption is made.

Assumption 1

  1. (i)

    The probability levels adopted by the insured and insurer are \(\alpha \) and \(\beta \) respectively where \(0<\alpha<\beta <1\).

  2. (ii)

    \(A_d<\rho _d(X)<A_d+A_r\).

The quantities \(A_d\) and \(A_r\) can be regarded as the capital available to the insured and insurer respectively. Then Assumption 1(ii) states that \(A_d\) is not enough to cover the risk X, however \(A_d+A_r\) can.

The following notations are used in the remainder of this paper: \(x\wedge y=\min \{x,y\}\) and \((x)_+=\max (x,0)\).

4.1 Optimal insurance policy that maximizes the insured’s EU under the VaR constraints

Let \(a=\text {VaR}_{\alpha }(X)\) and \(b=\text {VaR}_{\beta }(X)\) so that \(0\le a<b<M\). Then, applying the distortion function in (2.2) to (3.5), noting that

$$\begin{aligned} \int _{t}^{M}u'(w-x+I^*(x)-B)dF_X(x)=\mathbf {E}\left[ u'\left( w-X+I^*(X)-B\right) \mathbbm {1}_{[t,M)}(X)\right] \end{aligned}$$

and

$$\begin{aligned} S_X(t)=\mathbf {E}[\mathbbm {1}_{[t,M)}(X)], \end{aligned}$$

the function L in (3.5) becomes

$$\begin{aligned} L_V(t; I^*, \lambda _1, \lambda _2,\lambda _3)=\left\{ \begin{aligned}&\mathbf {E}\left[ \left\{ u'\left( w-X+I^*(X)-B\right) +\lambda _3\right\} \mathbbm {1}_{[t,M)}(X)\right] +\lambda _1-\lambda _2,&\quad t\le a, \\&\mathbf {E}\left[ \left\{ u'\left( w-X+I^*(X)-B\right) +\lambda _3\right\} \mathbbm {1}_{[t,M)}(X)\right] -\lambda _2,&\quad a< t\le b, \\&\mathbf {E}\left[ \left\{ u'\left( w-X+I^*(X)-B\right) +\lambda _3\right\} \mathbbm {1}_{[t,M)}(X)\right] ,&\quad t>b. \end{aligned} \right. \end{aligned}$$
(4.1)

In general, for \(\lambda _1, \lambda _2>0\), the function \(t\mapsto L_V(t; I^*, \lambda _1, \lambda _2,\lambda _3)\) has a downward jump at a, an upward jump at b, and is continuous on \([0,a)\cup (a,b)\cup (b,M)\). Its shape is illustrated in Fig. 1 left panel. As indicated by Theorem 3.1, the marginal indemnity function is equal to one on \(\left\{ t: L_V>0\right\} \) and zero on \(\left\{ t: L_V<0\right\} \). This is shown on the right panel of Fig. 1.

Fig. 1
figure 1

(Left) An example of \(L_V(t; I^*, \lambda _1, \lambda _1,\lambda _3)\); (Right) The corresponding optimal indemnity function

Full size image

To gain more insights, we next provide more illustrating results for specific cases where neither of the constraints is binding, where only one constraint is binding and where both constraints are binding. This facilitates the comparison of our results with those in the literature.

Proposition 4.1

Under the VaR risk constraints and Assumption 1, the solution to Problem 1a is given by the following.

  1. (1)

    If neither of the risk constraints is binding, then the optimal indemnity function is given by \( I_{d,V1}^*(x)=(x-t_0)_+ \) for some \(0\le t_0\le M\), where \(t_0\) is determined by \(B=(1+\theta )\mathbf {E}[I_{d,V1}^*(x)]\) .

  2. (2)

    If only the insured’s risk constraint is binding, then the optimal indemnity function is given by \( I_{d,V2}^*(x)=(x\wedge a-t_0)_++(x-t_1)_+ \) for some \(t_0, t_1\) such that \(t_0\le a\le t_1\), \(t_0=A_d-B\), and \(B=(1+\theta )\mathbf {E}[I_{d,V2}^*(X)]\).

  3. (3)

    If only the insurer’s risk constraint is binding, then the optimal indemnity function is given by \( I_{d,V3}^*(x)=(x\wedge t_1-t_0)_++(x-b)_+ \) for some \(t_0, t_1\) such that \(t_0\le t_1\le b\), \(t_1=A_r+B+t_0\), and \(B=(1+\theta )\mathbf {E}[I_{d,V3}^*(X)]\).

  4. (4)

    If both the insured and insurer’s risk constraints are binding, then the optimal indemnity function is given by \(I^*_{d,V_4}(x)=(x\wedge a-t_0)_++(x\wedge t_2-t_1)_++(x-b)_+\) for some \(t_0, t_1\) and \(t_2\) such that \(0 \le t_0\le a\le t_1\le t_2\le b\), \(t_0=A_d-B\), \(t_2=A_r+A_d-a+t_1\) and \(B=(1+\theta )\mathbf {E}[I^*_{d,V_4}(X)]\).

In practical application, one may follow the steps (1)\(\rightarrow \)(2)\(\rightarrow \)(4) or (1)\(\rightarrow \)(3)\(\rightarrow \)(4) of Proposition 4.1 to search for the solution to Problem 1a under the VaR constraints. For example, one starts with \(I_{d,V1}^*\). If both risk constraints are satisfied, then it is the optimal solution; if however it violates the insured’s risk constraint, then one tries \(I_{d,V2}^*\). If \(I_{d,V2}^*\) cannot be found because the equations in step (2) conflict with each other, then the risk constraints are not compatible with each other (\(\mathcal {C}_1\cap \mathcal {C}_2 =\emptyset \)) and the problem has no solution. If \(I_{d,V2}^*\) was found, then one needs to further examine whether the insurer’s risk constraint is satisfied. If yes, then \(I_{d,V2}^*\) is the optimal solution; If \(I_{d,V2}^*\) violates the insurer’s risk constraint, then one needs to try step (4). If a solution can be found, then it is optimal; otherwise the risk constraints are not compatible and the problem has no solution.

Remark 4.1

Equation (4.1) tells that \(L_V(t;I^*,\lambda _1,\lambda _2,\lambda _3)\) has a downward jump of size \(\lambda _1\) at point a and an upward jump of size \(\lambda _2\) at point b. Figure 1 shows that the Lagrangian coefficients \(\lambda _1,\lambda _2\) and \(\lambda _3\) directly affect the points \(t_0, t_1\) and \(t_2\), which are the parameters of our optimal indemnity function in Proposition 4.1. Intuitively, optimizing \(\lambda _1,\lambda _2\) and \(\lambda _3\) leads to the optimal \(t_0, t_1\) and \(t_2\). This implies that we can optimize \(t_0, t_1\) and \(t_2\) directly instead of calculating them through the optimal \(\lambda _1, \lambda _2\) and \(\lambda _3\). This reasoning also applies to the TVaR case.

Remark 4.2

To determine the global optimal indemnity function that solves Problem 1, we apply Proposition 4.1 to all possible premium levels in \([0,\pi _0]\) and select an optimal premium level \(B^*\) so that the insured’s EU reaches maximum. The explicit solution for \(B^*\) is difficult to obtain. However, the numerical search is easy to implement and this will be illustrated in Sect. 5.

Remark 4.3

When neither of the risk constraints is binding, then \(\lambda _1=\lambda _2=0\) and the function \(L_V\) has no jump. In this case, Problem 1a reduces to that in Arrow [2], for which the excess-of-loss indemnity function is optimal.

Remark 4.4

The case when only the insured’s risk constraint is binding has been studied in Huang [25], where the indemnity function I(x) is assumed to satisfy \(0\le I(x) \le x\). In their results, the optimal indemnity function has a downward jump at a.

In our model, indemnity function with jumps are not permissible. Consequently, if the function \(L_V\) jumps downward below zero at a, the optimal indemnity function becomes flat starting from a.

A comparison of our result and that of Huang [25] is shown in Fig. 2. Note that the exact shape of the indemnity function depends on other model parameters, e.g. the premium level B and utility function. Figure 2 only provides a representative situation.

Fig. 2
figure 2

A comparison of Arrow’s solution, our solution and that of Huang [25] when only the insured’s risk constraint is binding

Full size image

Note that when the insured’s risk constraint is binding, both ours results and Huang [25] suggest that the decision maker (insured in this case) substitutes coverage above a with coverage below a. This is because the losses above a do not contribute to the insured’s VaR level.

Remark 4.5

The case where only the insurer’s risk constraint is binding has been studied in Bernard and Tian [8], where the indemnity function I(x) is assumed to be non-decreasing and satisfy \(0\le I(x) \le x\).

Bernard and Tian [8] concluded that the optimal indemnity function has an upward jump at b. This is essentially because losses above b do not contribute to the insurer’s VaR. In our result, because of the continuity of the indemnity function, when the insurer’s risk constraint is binding, the indemnity function does not jump at b; instead, its slope changes from zero to one, indicating that losses above b is provided.

A comparison of our result and that in Bernard and Tian [8] is shown in Fig. 3.

Fig. 3
figure 3

A comparison of Arrow’s solution, our solution and that of Bernard and Tian [8] when only the insurer’s risk constraint is binding

Full size image

Remark 4.6

We have assumed that \(a < b\) to simplify the presentation. However, the results can be easily modified for the case \(a \ge b\). In addition, our approach applies to the case when the two parties employ different distortion risk measures. For example, if the insured applies VaR and the insurer applies TVaR, then the shape of the function L changes accordingly and the optimal indemnity function could be determined in a similar way.

We summarize the consequences of imposing VaR-type risk constraints. From the insured’s point of view, without the risk constraint, it prefers an excess-of-loss coverage (Arrow’s model). However, if the excess-of-loss policy violates its own VaR constraint, it will substitute the coverage for losses above level a with the coverage for losses below level a. On the other hand, if the insurer’s VaR risk constraint is violated, it will keep the coverage for losses above level b and reduce the coverage for losses below level b.

4.2 Optimal insurance policy that maximizes the insurer’s EU under the VaR constraints

When the insuer’s EU is to be maximized, as discussed in Sect. 3.2, the optimal indemnity function depends on the Lagrangian augmented function \(\tilde{L}\) defined in (3.7), which under VaR becomes

$$\begin{aligned} \tilde{L}_V(t; I^*, \lambda _1,\lambda _3)=\left\{ \begin{aligned}&\mathbf {E}\left[ \left\{ v'\left( w_r-I^*(X)+B\right) +\lambda _3\right\} \mathbbm {1}_{[t,M)}(X)\right] -\lambda _1+\lambda _2,&\quad t\le a, \\&\mathbf {E}\left[ \left\{ v'\left( w_r-I^*(X)+B\right) +\lambda _3\right\} \mathbbm {1}_{[t,M)}(X)\right] +\lambda _2,&\quad a< t\le b, \\&\mathbf {E}\left[ \left\{ v'\left( w_r-I^*(X)+B\right) +\lambda _3\right\} \mathbbm {1}_{[t,M)}(X)\right] ,&\quad t>b. \end{aligned} \right. \end{aligned}$$

It can be seen that \(\tilde{L}_V\) has an upward jump at a and a downward jump at b if \(\lambda _1, \lambda _2>0\). Other than the directions of the jumps, the shape of function \(\tilde{L}_V\) is similar to \({L}_V\) because both \(I^*(x)\) and \(x-I^*(x)\) are nondecreasing and u and v are strictly concave utility functions.

Recalling that in this case, coverage is provided for the interval when \(\tilde{L}_V\) is negative, we have the following result. As the result is analogous to Proposition 4.1, its proof is omitted.

Proposition 4.2

Under the VaR risk constraints and Assumption 1, the solution to Problem 2a is given by the following.

  1. (1)

    If neither of the risk constraints is binding, then the optimal indemnity function is given by \( I_{r,V1}^*(x)=x\wedge t_0 \) for some \(t_0\) such that \(0\le t_0\le M\) and \(B=(1+\theta )\mathbf {E}[I_{r,V1}^*(X)]\).

  2. (2)

    If only the insured’s risk constraint is binding, then the optimal indemnity function is given by \(I_{r,V2}^*(x)=(x\wedge t_0)+(x\wedge a - t_1)_+\) for some \(t_0\) and \(t_1\) such that \(0\le t_0\le t_1\), \(I^*_{r,V_2}(a)=a+B-A_d\) and \(B=(1+\theta )\mathbf {E}[I_{r,V2}^*(X)]\).

  3. (3)

    If only the insurer’s risk constraint is binding, then the optimal indemnity function is given by \( I_{r,V3}^*(x)=(x\wedge t_0)+(x\wedge t_1 - b)_+ \) for some \(t_0\) and, \(t_1\) such that \(0\le t_0\le b\le t_1\), \(t_0=A_r+B\) and \(B=(1+\theta )\mathbf {E}[I_{r,V3}^*(X)]\).

  4. (4)

    If both the insured and insurer’s risk constraints are binding, then the optimal indemnity function is given by \(I_{r,V3}^*\) for some \(t_0\) and \(t_1\) such that \(a<t_0\le b\le t_1\), \(t_0=B+A_r\) and \(B=(1+\theta )\mathbf {E}[I_{r,V3}^*(X)]=A_d\).

Remark 4.7

The case where only the insurer’s risk constraint is binding was studied by Bernard and Tian [8], where the derived optimal indemnity function has an upward jump at b.

In our result, when only the insurer’s risk constraint is binding, the slope of indemnity function changes from zero to one at b, resulting the coverage for the layer \([b, t_1]\). Meanwhile, the coverage for the layer [0, b] is reduced as the premium is fixed (Fig. 4). This result is intuitive because losses above level b does not contribute to the insurer’s VaR.

Fig. 4
figure 4

A comparison of Arrow’s solution, our solution and that of Bernard and Tian [8] when maximizing the insurer’s EU and only the insurer’s risk constraint is binding

Full size image

4.3 Optimal insurance policy that maximizes the insured’s EU under the TVaR constraints

When TVaR is used as the risk measure, the distortion function \(g_T\) in (2.3) is applied to (3.5). Then the function L in 3.5 becomes

$$\begin{aligned} L_T(t; I^*, \lambda _1,\lambda _2, \lambda _3)=\left\{ \begin{aligned}&\mathbf {E}\left[ \left\{ u'\left( w-X+I^*(X)-B\right) +\lambda _3\right\} \mathbbm {1}_{[t,M)}(X)\right] +\lambda _1-\lambda _2,&\quad t\le a, \\&\mathbf {E}\left[ \left\{ u'\left( w-X+I^*(X)-B\right) +\lambda _3+\frac{\lambda _1}{1-\alpha }\right\} \mathbbm {1}_{[t,M)}(X)\right] -\lambda _2,&\quad a< t\le b, \\&\mathbf {E}\left[ \left\{ u'\left( w-X+I^*(X)-B\right) +\lambda _3+\frac{\lambda _1}{1-\alpha }-\frac{\lambda _2}{1-\beta }\right\} \mathbbm {1}_{[t,M)}(X)\right] ,&\quad t>b. \end{aligned} \right. \end{aligned}$$

which is in fact continuous on [0, M]. With \(\lambda _1, \lambda _2>0\), the slope of \(L_T\) jumps at a and b. Fig. 5 gives an illustration.

Fig. 5
figure 5

(Left) An example of \(L_T(t; I^*, \lambda _1, \lambda _1,\lambda _3)\); (Right) An example of the optimal indemnity function

Full size image

The following proposition gives the solution to Problem 1a under the TVaR risk constraints. We again focus on specific cases where neither of the constraints are binding, where only one constraint is binding and where both constraints are binding. Its proof is provided in the appendix.

Proposition 4.3

Under the TVaR risk constraints, the solution to Problem 1a is given by the following.

  1. (1)

    If neither of the risk constraints is binding, then the optimal indemnity function is given by \( I_{d,T1}^*(x)=(x-t_0)_+ \) for some \(t_0\) such that \(0\le t_0\le M\) and \(B=(1+\theta )\mathbf {E}[I_{d,T1}^*(X)]\).

  2. (2)

    If only the insured’s risk constraint is binding, then the optimal indemnity function is still given by \(I_{d,T1}^*\) with the additional requirement that \(\text {TVaR}_\alpha (I_{d,T1}^*(X))=\text {TVaR}_\alpha (X)+B-A_d\).

  3. (3)

    If only the insurer’s risk constraint is binding, then the optimal indemnity function is given by \(I^*_{d,T_2}(x)=(x\wedge t_1-t_0)_++(x-t_2)_+\) for some \(t_0, t_1, t_2\) such that \(0\le t_0 \le t_1\le b\le t_2\le M\), \(\text {TVaR}_\beta (I_{d,T_2}^*(X))=A_r+B\) and \(B=(1+\theta )\mathbf {E}[I_{d,T_2}^*(X)]\).

  4. (4)

    If both the insured and insurer’s risk constraints are binding, then the optimal indemnity function is given by \(I^*_{d,T_2}\) for some \(t_0, t_1, t_2\) such that \(\text {TVaR}_\alpha (I_{d,T_2}^*(X))=\text {TVaR}_\alpha (X)+B-A_d\), \(\text {TVaR}_\beta (I^*_{d,T_2}(X))=A_r+B\) and \(B=(1+\theta )\mathbf {E}[I^*_{d,T_2}(X)]\).

Remark 4.8

With premium level B, the TVaR of the insured is minimized with the excess-of-loss indemnity function \(I^*_{d,T1}\). Consequently, Problem 1a has no solution (in other words, the compatibility requirement is violated) if \(I^*_{d,T1}\) violates the insured’s risk constraint. On the other hand, if \(I^*_{d,T1}\) violates the insurer’s risk constraint, the insurer could substitute the coverage for the layer \([t_1, t_2]\) (\(t_2> b\)) with that for the lower layer to reduce its TVaR level. A comparison of our result with Arrow’s classical one is illustrated in Fig. 6.

Fig. 6
figure 6

A comparison of Arrow’s solution and ours in Sect. 4.3 when the insurer’s TVaR risk constraint is binding

Full size image

4.4 Optimal insurance policy that maximizes the insurer’s EU under the TVaR constraints

When the insuer’s EU is to be maximized, similarly as the previous section we check the function

$$\begin{aligned} \tilde{L}_T(t; I^*, \lambda _1,\lambda _2, \lambda _3)=\left\{ \begin{aligned}&\mathbf {E}\left[ \left\{ v'\left( w_r-I^*(X)+B\right) +\lambda _3\right\} \mathbbm {1}_{[t,M)}(X)\right] -\lambda _1+\lambda _2,&\quad t\le a, \\&\mathbf {E}\left[ \left\{ v'\left( w_r-I^*(X)+B\right) +\lambda _3-\frac{\lambda _1}{1-\alpha }\right\} \mathbbm {1}_{[t,M)}(X)\right] +\lambda _2,&\quad a< t\le b, \\&\mathbf {E}\left[ \left\{ v'\left( w_r-I^*(X)+B\right) +\lambda _3-\frac{\lambda _1}{1-\alpha }+\frac{\lambda _2}{1-\beta }\right\} \mathbbm {1}_{[t,M)}(X)\right] ,&\quad t>b. \end{aligned} \right. \end{aligned}$$

Coverage is provided when \(\tilde{L}_T(t; I^*, \lambda _1,\lambda _2, \lambda _3)<0\).

The corresponding optimal indemnity function is provided in the following proposition.

Proposition 4.4

Under the TVaR risk constraints, the solution to Problem 2a is given by the following.

  1. (1)

    If neither of the risk constraints is binding, then the optimal indemnity function is given by \( I_{r,T_1}^*(x)=x \wedge t_0 \) for some \(t_0\) such that \(0\le t_0\le M\) and \(B=(1+\theta )\mathbf {E}[I_{r,T1}^*(X)]\) .

  2. (2)

    If only the insured’s risk constraint is binding, then the optimal indemnity function is given by \(I_{r,T_2}^*(x)=(x\wedge t_0)+(x \wedge t_2-t_1)_+\) for some \(t_0, t_1, t_2\) such that \(0\le t_0\le t_1 \le a \le t_2 \le M\), \(TVaR_{\alpha }(I^*_{r,T_2}(X))=TVaR_{\alpha }(X)+B-A_d\) and \(B=(1+\theta )\mathbf {E}[I_{r,T_2}^*(X)]\).

  3. (3)

    If only the insurer’s risk constraint is binding, then the optimal indemnity function is given by \(I^*_{r,T_1}\) for some \(t_0\) such that \(0\le t_0\le M\), \(\text {TVaR}_\beta (I^*_{r,T_1}(X))=A_r+B\) and \(B=(1+\theta )\mathbf {E}[I^*_{r,T_1}(X)]\).

  4. (4)

    If both the insured and insurer’s risk constraints are binding, then the optimal indemnity function is given by \(I_{r,T_2}^*\) for some \(t_0, t_1, t_2\) such that \(0\le t_0\le t_1 \le a \le t_2 \le M\), \(TVaR_{\alpha }(I^*_{r,T2}(X))=TVaR_{\alpha }(X)+B-A_d\), \(\text {TVaR}_\beta (I^*_{r,T_2}(X))=A_r+B\) and \(B=(1+\theta )\mathbf {E}[I_{r,T_2}^*(X)]\).

4.5 Proportional hazard transform

The above four sections discuss the optimal indemnity function from either the insured’s or insurer’s perspective under either the VaR or TVaR risk constraints, of which the distortion function is simple and piece-wise linear. As shown in the above sections, the optimal indemnity functions for these cases are of layered forms, i.e. with slope zero or one for different layers of loss. In this section, we explore the optimal indemnity function when the risk constraints are based on the distortion function that is not piece-wise linear.

We focus on the PH transform proposed in Wang [32], with the distortion function \(g_{PH}(x)=x^{\gamma }\) for some \(\gamma \in (0,\infty )\). This transform is rather flexible because \(g_{PH}\) is convex if \(\gamma >1\) and is concave if \(\gamma <1\). Thus, when applied to the survival function of loss, the former represents risk-seeking behavior, whereas the latter represents risk-aversion.

In particular, we consider the insured’s decision problem (Problem 1), where the insured’s risk constraint is defined in terms of PH transform, i.e. \(g_{PH}(x)=x^{\gamma }\) with \(\gamma \in (0,1)\). For simplicity, we assume that only the insured’s risk constraint is binding (i.e., \(\lambda _1>0\) and \(\lambda _2=0\)).

Under this setting, the function \(L(t;I^*,\lambda _1,\lambda _2,\lambda _3)\) in Theorem 3.1 becomes

$$\begin{aligned} \begin{aligned} L(t;I^*,\lambda _1,\lambda _3)&=\int _{t}^M u'(w-x+I^*(x)-B)dF_X(x)+\lambda _1 g_{PH}(S_X(t))+\lambda _2S_X(t) \\&=\int _t^M u'(w-x+I^*(x)-B)dF_X(x)+\lambda _1 S_X(t)^{\gamma }+\lambda _3 S_X(t), \end{aligned} \end{aligned}$$
(4.2)

where \(\lambda _1\in \mathbf {R}^+\) and \(\lambda _3\in \mathbf {R}\). The first-order derivative of \(L(t;I^*,\lambda _1,\lambda _2)\) is given by

$$\begin{aligned} L'(t;I^*,\lambda _1,\lambda _3)=-K'(t;I^*,\lambda _1,\lambda _3) f_X(t), \end{aligned}$$
(4.3)

where \(K(t;I^*,\lambda _1,\lambda _3)=u'(w-t+I^*(t)-B)+\lambda _1\gamma S_X(t)^{\gamma -1}+\lambda _3\).

Because \(u''(\cdot )<0\), \({I^*}'(t)\in [0,1]\), and \(\gamma \in (0,1)\), we have

$$\begin{aligned} K'(t;I^*,\lambda _1,\lambda _3)=u''(w-t+I^*(t)-B)({I^*}'(t)-1)-\lambda _1\gamma (\gamma -1)S_X(t)^{\gamma -2}f_X(t)>0. \end{aligned}$$
(4.4)

With the monotonicity of \(K(t;I^*,\lambda _1,\lambda _3)\), we are able to derive the following two properties of L.

  • First, \(L(t;I^*,\lambda _1,\lambda _3)=0\) cannot hold on any sub-intervals of [0, M]. This could be proved through contradiction. If there exists a sub-interval, e.g. \([a,b]\subseteq [0,M]\), such that \(L(t;I^*,\lambda _1,\lambda _3)=0\) for \(t\in [a,b]\). Then \(L'(t;I^*,\lambda _1,\lambda _3)=0\) for \(t\in [a,b]\), and this further implies that \(K(t;I^*,\lambda _1,\lambda _3)=0\) for \(t\in [a,b]\). Then for any \(t\in [a,b]\),

    $$\begin{aligned} \begin{aligned}&K'(t;I^*,\lambda _1,\lambda _3)=0\\&\quad \Longrightarrow \ u''(w-t+I^*(t)-B)({I^*}'(t)-1)-\lambda _1\gamma (\gamma -1)S_X(t)^{\gamma -2}f_X(t)=0 \\&\quad \Longrightarrow \ {I^*}'(t)=1+\frac{\lambda _1\gamma (\gamma -1)S_X(t)^{\gamma -2}f_X(t)}{u''(w-t+I^*(t)-B)}>1, \end{aligned} \end{aligned}$$

    which contradicts with the incentive compatibility condition.

  • Second, there does not exist a point \(t^*\) such that

    $$\begin{aligned} L(t^*;I^*,\lambda _1,\lambda _3)<0,\quad L'(t^*;I^*,\lambda _1,\lambda _3)\le 0. \end{aligned}$$

    This implies that L cannot down-cross the t axis. We next prove it by contradiction. If such \(t^*\) exists, then from the second inequality we have \(K(t^*;I^*,\lambda _1,\lambda _3)\ge 0\). Based on (4.4), we get \(K(t;I^*,\lambda _1,\lambda _3)\ge 0\) for any \(t\in [t^*,M]\), which further implies that \(L'(t;I^*,\lambda _1,\lambda _3)\le 0\) for any \(t\in [t^*,M]\). This leads to

    $$\begin{aligned} L(t^*;I^*,\lambda _1,\lambda _3)=\int _{t^*}^M-L'(x;I^*,\lambda _1,\lambda _3)dx\ge 0, \end{aligned}$$

    which contradicts with \(L(t^*;I^*,\lambda _1,\lambda _3)<0\).

With the above two properties, we can define

$$\begin{aligned} \tilde{\mathcal {S}}:=\left\{ t\in [0,M]: L(t;I^*,\lambda _1,\lambda _3)\ge 0\right\} \quad \text {and}\quad d=\inf \tilde{\mathcal {S}} \end{aligned}$$

with the convention \(d=M\) if \(\tilde{\mathcal {S}}=\emptyset \). Then \(L(t;I^*,\lambda _1,\lambda _3)<0\) for \(t\in [0,d)\) and \(L(t;I^*,\lambda _1,\lambda _3)\ge 0\) for \(t\in [d,M]\). Based on Theorem 3.1, the optimal indemnity function is of the excess-of-loss type.

We summarize the above finding in the following corollary.

Corollary 4.1

For the insured’s decision problem with its risk constraint defined in terms of the PH transform \(g_{PH}(x)=x^{\gamma }\) with \(\gamma \in (0,1)\), if only the insured’s risk constraint is binding, then the optimal indemnity function is given by \(I^*(x)=(x-d)_+\) for some \(d\in [0,M]\).

Remark 4.9

Suppose that the insurer’s risk constraint is defined in terms of the PH transform and binding for the insured’s decision problem (Problem 1a), the form of the optimal indemnity function is more complicated. In this situation, it is possible that \(L(t;I^*,\lambda _1,\lambda _2,\lambda _3)=0\) holds on some sub-intervals of [0, M], which may yield coinsurance (i.e., \({I^*}'\in (0,1)\)) on those intervals. Since this current paper mainly focuses on the problem with the VaR or TVaR risk constraints, we will not discuss this issue further herein.

5 Numerical analysis

In Sects. 3 and 4, we derived the optimal parametric form of the indemnity function by analyzing the Lagrangian augmented function (3.5) or (3.7). These results are used in this section to study the sensitivity of the indemnity function with respect to the risk constraints. To save space, we focus on Problem 1a, where the objective is to maximize the insured’s EU under risk constraints.

We assume the following throughout the analysis.

  • The insured’s preference is captured by a quadratic utility function:

    $$\begin{aligned} u(x)=-\frac{1}{2}\delta x^2+x,\quad x\le \frac{1}{\delta }, \end{aligned}$$

    where \(\frac{1}{\delta }\) is called the saturation point representing the maximum wealth of the insured. We set \(\frac{1}{\delta }=10{,}000\).

  • The insured’s initial capital w is 2000.

  • The ground-up loss follows an exponential distribution \(F_X(x)=1-\exp (-\frac{x}{\lambda })\) with \(\lambda =1000\).

  • The budget level \(\pi _0\) is set to be 1000, which is half of the insured’s initial capital.

  • The risk loading \(\theta \) in determining the premium is 0.2.

  • For the VaR and TVaR calculation, the insured and insurer apply probability level \(\alpha =0.95\) and \(\beta =0.99\) respectively. Thus, \(a=VaR_\alpha (X)=2996\) and \(b=VaR_\beta (X)=4605\).

5.1 Maximize the insured’s EU under the VaR constraints

Under Arrow’s model, i.e. maximizing the insured’s expected utility with no risk constraints, the optimal insurance policy is given by \(B_{Arrow}^*=78.3\) and \(I_{Arrow}^*(x) = (x-2730)_+\). With this policy, the VaR of the insured is \(VaR_\alpha (X-I_{Arrow}^*(X)+B_{Arrow}^*)=2808\) and the VaR of the insurer is \(VaR_\beta (I_{Arrow}^*(X)-B_{Arrow}^*)=1797\).

5.1.1 The insured’s VaR constraint is violated

We first examine the case where \(A_d=2500\) and only the insured’ VaR constraint is violated if \(I^*_{Arrow}\) is applied. Then applying Proposition 4.1 (2), the optimal indemnity function is give by

$$\begin{aligned} I^*_{d,V_2}(x)= (x\wedge a-t_0)_++(x-t_1)_+, \end{aligned}$$

where \(t_0=2409\) and \(t_1=3326\). The optimal premium level is found numerically to be \(B^*=91\).

With the insurance policy \((I^*_{d,V_2}, B^*)\) as shown above, the VaR of the insured and insurer are 2500 and 1775 respectively. We observe that, comparing with \(I_{Arrow}^*\), the insured retains the risk above a and purchases more coverage for losses below a, which actually contributes to its VaR.

If \(A_r>1775\), then \(I^*_{d,V_2}\) is the solution to Problem 1. If however \(A_r\le 1775\), e.g. \(A_r=1700\), we need to apply Proposition 4.1 (4). In this situation, the optimal indemnity function is given by

$$\begin{aligned} I^*_{d,V_4}(x)=(x\wedge a-t_0)_++(x\wedge t_2-t_1)_++(x-b)_+, \end{aligned}$$
(5.1)

where \(t_0=2410\), \(t_1=3325\) and \(t_2=4529\). The optimal premium level is \(B^*=90\). As seen in Fig. 7, the insurer sells less coverage below b such that its VaR is reduced to meet the risk constraint.

Fig. 7
figure 7

The optimal indemnity functions with different risk constraints for Sect. 5.1.1

Full size image

5.1.2 The insurer’s VaR constraint is violated

We first examine the case where \(A_r=1700\) and only the insurer’ VaR constraint is violated if \(I_{Arrow}^*\) is applied. Then applying Proposition 4.1 (3), the optimal indemnity function is give by

$$\begin{aligned} I^*_{d,V_3}(x)= (x\wedge t_1-t_0)_++(x-b)_+, \end{aligned}$$

where \(t_0=2715\) and \(t_1=4493\). The optimal premium level is found numerically to be \(B^*= 78\).

With insurance policy \((I^*_{d,V_3}, B^*)\) as shown above, the VaR of the insured and insurer are 2793 and 1700 respectively. We observe that, comparing with \(I_{Arrow}^*\), the insurer will reduce its coverage for the losses below b so as to reduce its VaR level (Fig. 8).

If \(A_d>2793\), then \(I^*_{d,V_3}\) is the solution to Problem 1. If however \(A_d\le 2793\), e.g. \(A_d=2500\), we need to apply Proposition 4.1 (4), which yields the optimal indemnity function \(I^*_{d,V_4}\) given by (5.1).

Fig. 8
figure 8

The optimal indemnity functions with different risk constraints for Sect. 5.1.2

Full size image

5.2 Maximize the insured’s EU under the TVaR constraints

Note that with \(I^*_{Arrow}\) and \(B_{Arrow}^*\), the TVaR of the insured and insurer are 2808 and 2797 respectively.

5.2.1 The insured’s TVaR constraint is violated

First suppose that \(A_d=1600\) and only the insured’s TVaR constraint is violated. Then according to Proposition 4.3 (2), the optimal indemnity function still takes the excess-of-loss form. To make the insured’s TVaR constraint binding, the retention point d needs to satisfy

$$\begin{aligned} \begin{aligned}&TVaR_{\alpha }((X-d)_+)=TVaR_{\alpha }(X)+(1+\theta )\mathbf {E}[(X-d)_+]-A_d \\&\quad \Longrightarrow \ \int _{d}^{\infty }\left\{ g_T(S_X(t))-(1+\theta )S_X(t)\right\} dt=TVaR_{\alpha }(X)-A_d. \end{aligned} \end{aligned}$$

A numerical search gives \(d^*=1259\), which is the only root for the equation above. In other words, the optimal indemnity function is given by \(I^*_{d,T_1}=(x-1259)_+\) and the corresponding premium is \(B^*=341\). With this policy, the TVaR of the insured and insurer are 1600 and 4005 respectively (Fig. 9).

If \(A_r>4005\), then \(I^*_{d,T_1}\) is the solution to Problem 1. If however \(A_r\le 4005\), e.g. \(A_r=2700\), then we apply Proposition 4.3 (4) and obtain that the optimal indemnity function is given by

$$\begin{aligned} I^*_{d,T_2}(x)=(x\wedge t_1-t_0)_++(x-t_2)_+, \end{aligned}$$
(5.2)

where \(t_0=454\), \(t_1=3891\) and \(t_2=12211\), for which the premium is \(B^*=738\).

Fig. 9
figure 9

The optimal indemnity functions with different risk constraints for Sect. 5.2.1

Full size image

5.2.2 The insurer’s TVaR constraint is violated

Now suppose that \(A_r=2700\) and only the insurer’s TVaR constraint is violated. Then according to Proposition 4.3 (3), the optimal indemnity is given by

$$\begin{aligned} I^*_{d,T_2}(x)=(x\wedge t_1-t_0)_++(x-t_2)_+, \end{aligned}$$

where \(t_0=986\), \(t_1=4020\) and \(t_2=6976\). With this indemnity function and the corresponding premium \(B^*=427\), the TVaR of the insured and insurer are 1754 and 2700 respectively (Fig. 10).

If \(A_d>1754\), then \(I^*_{d,T_2}\) is the solution to Problem 1. If however \(A_d\le 1754\), e.g. \(A_d=1600\), then we apply Proposition 4.3 (4) and obtain the optimal indemnity function as given by (5.2).

Fig. 10
figure 10

The optimal indemnity functions with different risk constraints for Sect. 5.2.2

Full size image

6 Concluding remarks and future research

In this paper, we study the optimal insurance policy which maximizes the DM’s expected utility under the risk constraints of both parties. Different from the existing literature, we impose the no-sabotage condition on the indemnity function to mitigate the potential ex post moral hazard. Moreover, we define the risk constraints using general distortion risk measures, which generalizes the VaR constraints used in the literature. We show that the optimal indemnity function has a layered form. Closed-form indemnity functions are obtained when the risk constraints are defined through VaR and TVaR. The impacts of the incentive compatibility condition and the risk constraints on the optimal indemnity function are discussed in great detail.

This paper focuses on the unilateral problem, where the objective is maximizing only one party’s EU. In the future, it would be interesting to extend the model to a bilateral or multi-player one. In those cases, the technical complexity would be enhanced drastically and the results would allow for more interesting economic interpretations.