Abstract
This research article considers a class of distributed stochastic systems where interconnected systems closely keep track of reference signals issued by a coordinator. Much of the existing literature concentrates on conducting decisions and control synthesis based solely on expected utilities and averaged performance. However, research in psychology and behavioral decision theory suggests that performance risk plays an important role in shaping preferences in decisions under uncertainty. Thus motivated, a new equilibrium concept, called “person-by-person equilibrium” for local best responses is proposed for analyzing signaling effects and mutual influences between an incumbent system, its coordinator, and immediate neighbors. Individual member objectives are defined by the multi-attribute utility functions that capture both performance expectation and risk measures to model the satisfaction associated with local best responses with risk-averse attitudes. The problem class and approach of coordination control of distributed stochastic systems proposed here are applicable to and exemplified in military organizations and flexibly autonomous systems.
Access provided by Autonomous University of Puebla. Download conference paper PDF
Similar content being viewed by others
Keywords
- Distributed stochastic system
- Coordination control
- Mutual influences
- Signaling effects
- Performance measure
- Performance risk
- Output feedback
- Risk-value aware performance index
- Person-by-person equilibrium
1 Introduction
Control and coordination of distributed stochastic systems offers a framework to analyzing intertemporal strategic interactions between individual agents or controllers, one for each interconnected systems and based on local observations. The importance of evaluating approaches in a dynamic setting and the broad flexibility and adaptability of the decision and control architectures of distributed control with communications has spurred many large-scale applications such as military command and control hierarchies, spacecraft constellations, remotely piloted platform formations, teams of humans and autonomous robots, etc. where each member can be in best response to its neighbor actions and yet has no influence on other members to which it has no communication supports.
Despite the broad interest in distributed systems, there remain significant hurdles in applying them to practical problems of interest. Interplay between common team objectives and individual member objectives can yield surprises and complex behaviors. Hence, a form of coordination control that helps balance between cooperative goals and adversarial behavior in addition to fundamentals for team and individual decisions, is necessarily required.
Thus motivated, this research article proposes a new framework and analysis to study risk-averse control of a distributed stochastic system, in particular coordination control with risk-averse attitudes toward performance uncertainty and robustness. The approach of noncooperative game-theoretic decision making and optimization is suited to coordination control, where a distributed stochastic system is distinguished into a coordinator (also known as dominant player) with significant reference signals and incumbent systems (also known as nondominant players) with fringe couplings. To account for uncertainty in inherent design problem and in preference assessment, a multi-attribute utility function that enables incumbent systems’ decision makers or controllers to select the best risk-averse strategy for the attribute trade-offs between performance expectation and risks is therefore considered. Notice that this dominant/nondominant game structure is also prevalent in both economics [1] and social sciences [2].
The game-theoretic model of mixed player behaviors considered herein is particularly related to the research [3] that has extended the large population linear-quadratic-Gaussian games to include a major player and a large number of minor players. As such, minor players are more sensitive to variations in the behavior of major player than those of individual minor players. To overcome the curse of dimensionality, computational concerns have typically resorted the analysis to the so-called Nash certainty equivalence method, where the key idea is to break the large population game into a family of limiting two-player games. The synthesis of decentralized strategies is obtained via a set of aggregate quantities giving the mean field approximation. In contrast with such existing literature, this appealing research representing the interplay between stochastics, statistics, and dynamics as well as the extension of the recent accounts [4, 5] investigates: (1) a stochastic dynamic game model of behavior where nondominant players not only keep track closely of the large impact by the dominant player but also monitor rivals from the peers in a less detailed way and (2) a tractable paradigm of performance assessment uncertainty forecast for which sufficient statistics summarize all performance relevant information and thus are used in the person-by-person equilibrium strategies by nondominant players.
In summary, the proposed game-theoretic framework is prevalent in distributed stochastic systems with a dominant/fringe coordination structure, capturing the attributes that are important to inherent design problem and preference assessment uncertainties, their trade-off behavior over these attributes and their risk attitude. The rest of this article is organized as follows. Section 2 introduces a new computationally tractable model for distributed stochastic systems with state-space representations of a dominant coordinator and many nondominant systems. In addition, the preliminary results on sufficient mathematical statistics that summarize all performance measure or utility relevant history and for which the person-by-person equilibrium strategies are optimal for nondominant systems are discussed in great detail. Section 3 contains precise problem statements for coordination control analysis and decision optimization for the person-by-person equilibrium or feedback Nash strategy concerned by autonomous agents and incumbent systems. The construction of person-by-person strategies is established in Sect. 4, while some conclusions and future research directions are drawn in Sect. 5.
2 Problem Formulation
Before going into a formal presentation, it is necessary to consider some conceptual notations in this article. For instance, time t is modeled as continuous and the notation of the time interval is [t 0, t f ]. All random variables are defined on a probability space \((\Omega,\mathcal{F},\mathcal{P})\) which is a triple consisting of a set Ω, a σ-algebra \(\mathcal{F}\), and a probability measure \(\mathcal{P}: \mathcal{F}\mapsto [0,1]\) and is equipped with a filtration \(\{\mathcal{F}_{t}: t \in [t_{0},t_{f}]\}\). In addition, for a given Hilbert space X with norm \(\vert \vert \cdot \vert \vert _{X}\), 1 ≤ p ≤ ∞, a Banach space is defined as follows
with norm
Furthermore, the Banach space of X-valued continuous functionals on [t 0, t f ] with the max-norm induced by \(\vert \vert \cdot \vert \vert _{X}\) is denoted by \(\mathcal{C}(t_{0},t_{f};X)\). The deterministic version of (1) and its associated norm (2) is written as \({\mathcal{L}}^{p}(t_{0},t_{f};X)\) and \(\vert \vert \cdot \vert \vert _{p}\).
A distributed stochastic system that evolves over [t 0, t f ] captures interactions among a coordinator and finite number of incumbent systems. Each incumbent system that enters the distributed system is assigned a unique positive integer-valued index. The set of indices of incumbent systems is denoted by \(\overline{I} \triangleq \{ 1,2,\ldots,N\}\) and a typical element by i. The set of immediate neighbors associated with an incumbent system i is denoted by N i . For concreteness, the heterogeneity of incumbent system i and \(i \in \overline{I}\) is distinguished by an individual state; that is governed by the stochastic differential equation with the initial-value condition \(x_{i}(t_{0}) = x_{i}^{0}\)
where the continuous-time coefficients \(A_{ii} \in \mathcal{C}(t_{0},t_{f}; {\mathbb{R}}^{n_{i}\times n_{i}})\), \(B_{ii} \in \mathcal{C}(t_{0},t_{f}; {\mathbb{R}}^{n_{i}\times m_{i}})\), \(C_{ii} \in \mathcal{C}(t_{0},t_{f}; {\mathbb{R}}^{n_{i}\times q_{i}})\), \(B_{ij} \in \mathcal{C}(t_{0},t_{f}; {\mathbb{R}}^{n_{i}\times r_{i}})\), \(G_{i} \in \mathcal{C}(t_{0},t_{f}; {\mathbb{R}}^{n_{i}\times p_{i}})\) as well as \(C_{i} \in \mathcal{C}(t_{0},t_{f}; {\mathbb{R}}^{r_{i}\times n_{i}})\) are deterministic matrix-valued functions. At time t, the recursive state and output of incumbent system i are denoted by \(x_{i} \in \mathcal{L}_{\mathcal{F}_{i}}^{2}(t_{0},t_{f}; {\mathbb{R}}^{n_{i}})\) and \(y_{i} \in \mathcal{L}_{\mathcal{F}_{i}}^{2}(t_{0},t_{f}; {\mathbb{R}}^{r_{i}})\) with the initial state \(x_{i}^{0} \in {\mathbb{R}}^{n_{i}}\) known. The control policies from agent i to that system i are presented by \(u_{i} \in \mathcal{L}_{\mathcal{F}_{i}}^{2}(t_{0},t_{f}; {\mathbb{R}}^{m_{i}})\) and \(z_{i} \in \mathcal{L}_{\mathcal{F}_{i}}^{2}(t_{0},t_{f}; {\mathbb{R}}^{q_{i}})\). In addition, the interconnection inputs and linkage effects of that incumbent system i supported by the communication paths from immediate neighbors j and j ∈ N i are viewed as the real-valued functions u ij (t)dt of the following random processes
where continuous-time coefficients \(C_{ij} \in \mathcal{C}(t_{0},t_{f}; {\mathbb{R}}^{r_{i}\times n_{j}})\) and \(D_{ij} \in \mathcal{C}(t_{0},t_{f}; {\mathbb{R}}^{r_{i}\times m_{j}})\) are deterministic matrix-valued functions. As the number of incumbent systems grows large, it is unrealistic to believe that binding agents i associated with incumbent systems i and \(i \in \overline{I}\) are capable of monitoring the evolution of their immediate neighbors. Instead, it is reasonable to assume that incumbent systems only keep track of actual interactions or signaling references provided by coordinator c and \(c \in \overline{I}_{c}\), where the set of partaking coordinators is predetermined and does not change over time.
A challenging task for all multiscale modeling and coordination control is to transfer the knowledge gained from one resolution to another. As such, in coordination control there is an ongoing need for a coordinator c issuing reference signals to two or more incumbent systems i and \(i \in \overline{I}\) such that
but the incumbent systems i do not directly send signals to the coordinator c. In practice, it is further desirable to have decentralized decision making without intensive communication overheads. A potential alternative therefore involves the selection of a crude model of reduced order for the interactions among coordinator c and binding agents i associated with incumbent systems i. The actual reference signals imposed by coordinator c are now approximated by an explicit model-following of the type
whereby continuous-time coefficients \(A_{ic} \in \mathcal{C}(t_{0},t_{f}; {\mathbb{R}}^{q_{i}\times q_{i}})\), \(B_{ic} \in \mathcal{C}(t_{0},t_{f}; {\mathbb{R}}^{q_{i}\times m_{c}})\), \(G_{ic} \in \mathcal{C}(t_{0},t_{f}; {\mathbb{R}}^{q_{i}\times p_{ic}})\) and \(C_{ic} \in \mathcal{C}(t_{0},t_{f}; {\mathbb{R}}^{r_{ic}\times q_{i}})\) are deterministic matrix-valued function and potentially come from a structural decomposition of a monolithic distributed system with centralized dynamics. In this exposition, \(z_{ic} \in \mathcal{L}_{\mathcal{F}_{i}}^{2}(t_{0},t_{f}; {\mathbb{R}}^{q_{i}\times q_{i}})\) is the coordinator state, \(u_{c} \in \mathcal{L}_{\mathcal{F}_{i}}^{2}(t_{0},t_{f}; {\mathbb{R}}^{m_{c}})\) is the coordinator control input and \(y_{ic} \in \mathcal{L}_{\mathcal{F}_{i}}^{2}(t_{0},t_{f}; {\mathbb{R}}^{r_{ic}})\) is the coordinator output.
In the state-space representations (3)–(4) and (7)–(8) one postulates uncorrelated Wiener processes \(w_{i}(t) \triangleq w_{i}(t,\omega _{i}): [t_{0},t_{f}] \times \Omega _{i}\mapsto {\mathbb{R}}^{p_{i}}\), \(v_{i}(t) \triangleq v_{i}(t,\omega _{i}): [t_{0},t_{f}] \times \Omega _{i}\mapsto {\mathbb{R}}^{r_{i}}\), \(w_{ic}(t) \triangleq w_{ic}(t,\omega _{ic}): [t_{0},t_{f}] \times \Omega _{ic}\mapsto {\mathbb{R}}^{p_{ic}}\) and \(v_{ic}(t) \triangleq v_{ic}(t,\omega _{ic}): [t_{0},t_{f}] \times \Omega _{ic}\mapsto {\mathbb{R}}^{r_{ic}}\) defined by the underlying filtered probability spaces \((\Omega _{i},\mathcal{F}_{i},\{\mathcal{F}_{i}\}_{t},\mathcal{P}_{i})\) and \((\Omega _{ic},\mathcal{F}_{ic},\{\mathcal{F}_{ic}\}_{t},\mathcal{P}_{ic})\) with the correlations of independent increments
which now approximate the inherent design system uncertainty due to variability and lack of knowledge.
Furthermore, the model primitives of the state recursion (3) in the absence of links from the immediate neighbors and environmental disturbances are also assumed to be uniformly exponentially stable. For instance, there exist positive constants η 1 and η 2 such that the pointwise matrix norm of the closed-loop state transition matrix associated with incumbent system (3) satisfies the inequality
The pair \((A_{ii}(t),[B_{ii}(t),C_{ii}(t)])\) is pointwise stabilizable if there exist bounded matrix-valued functions \(K_{x_{i}}(t)\) and \(K_{z_{i}}(t)\) so that the closed-loop system \(dx_{i}(t) = \left (A_{ii}(t) + B_{ii}(t)K_{x_{i}}(t) + C_{ii}(t)K_{z_{i}}(t)\right )x_{i}(t)dt\) is uniformly exponentially stable.
With the local agent dynamics (3) considered herein, each agent i associated with incumbent system i only plays a local dynamical game with its immediate neighbors j ∈ N i . Mutual influence controlled by the control policies from the immediate neighbors of agent i is defined by \(u_{-i} \triangleq \{ u_{ij}: j \in N_{i}\}\). Assuming its coalition N i conveys mutual influence information u − i , agent i selects, at each time instant, a tuple of control policies to optimize its multi-attribute utility function. The tuple of control laws is defined by the control processes u i and z i , of which z i is supposed to follow the prediction process z ic for the reference signals from coordinator c. Thus, the subsequent states of agent i is determined by its current individual states x i and z ic , its chosen action (u i , z i ) and the coalition effects u − i . In fact, the selected action (u i , z i ) will depend on agent i’s individual states x i and z ic as well as the coalition effects u − i .
To further illustrate the applicability of the coordination control framework as proposed here, the classes of admissible control policies associated with (3) are defined by \(U_{i} \times Z_{i} \subset \mathcal{L}_{\mathcal{F}_{i}}^{2}(t_{0},t_{f}; {\mathbb{R}}^{m_{i}}) \times \mathcal{L}_{\mathcal{F}_{ mi}}^{2}(t_{ 0},t_{f}; {\mathbb{R}}^{q_{i}})\). For any given coalition effects u − i , the 3-tuple \((x_{i}(\cdot ),u_{i}(\cdot ),z_{i}(\cdot ))\) shall be therefore referred to as an admissible 3-tuple if \(x_{i}(\cdot ) \in \mathcal{L}_{\mathcal{F}_{i}}^{2}(t_{0},t_{f}; {\mathbb{R}}^{n_{i}})\) is the solution trajectory of the stochastic differential equation (3) when u i ( ⋅) ∈ U i and z i ( ⋅) ∈ Z i .
In the subsequent analysis, the problem of observation and/or estimation in the distributed stochastic system is investigated with a major emphasis on the design of a set of locally optimal decision and control policies for incumbent agent i and \(i \in \overline{I}\) in a completely decentralized environment with interconnection patterns. More precisely, since (A ii , C i ) are detectable, it is possible to construct the local observers
whereby \(\hat{x}_{i}(t) \in {\mathbb{R}}^{n_{i}}\) is the state estimate of x i (t) for incumbent agent i and \(i \in \overline{I}\) and \(L_{i}(t) \in {\mathbb{R}}^{n_{i}\times r_{i}}\) are the decentralized filtering gains determined by suitably modifying the dynamics of the local observers; for example
It is readily evident that the decentralized observation scheme developed in (9)–(11) incorporates the knowledge of the interconnection functions or the outputs of the other immediate neighbors of agent i and \(i \in \overline{I}\).
In similar to the state-regulation case, independent decentralized optimal estimators may be designed hereafter for certain compensating signals from coordinators c to agent i; e.g.,
whereby \(L_{ic}(t) \in {\mathbb{R}}^{q_{i}\times r_{ic}}\) is given by \(L_{ic}(t) = \Sigma _{ic}(t)C_{ic}^{T}(t)V _{ic}^{-1}\) and \(\Sigma _{ic}(t) \in {\mathbb{R}}^{q_{i}\times q_{i}}\) is the covariance of the error process \(\tilde{z}_{ic}(t) = z_{ic}(t) -\hat{ z}_{ic}(t)\), satisfying the forward-in-time differential equation
In terms of the observation errors \(\tilde{x}_{i}(t) = x_{i}(t) -\hat{ x}_{i}(t)\) and \(\tilde{z}_{ic}(t) = z_{ic}(t) -\hat{ z}_{ic}(t)\), it follows from (3), (4), (7), and (8) that, for \(\tilde{x}_{i}(t_{0}) = 0\) and \(\tilde{z}_{ic}(t_{0}) = 0\)
Indeed, the system (14)–(15) will function as observers for the system (3) and (7) if the design parameters L i (t) and L ic (t) can be selected such that the local observers (9) and (12) are asymptotically stable.
Next, agent i evaluates its performance and makes control policies that are consistent with its preferences. There are performance trade-offs among the closeness of locally accessible states \(\hat{x}_{i}\) from desired states ζ i , the size of local actions u i and the closeness of interaction enforcements between local efforts z i and local estimates \(\hat{z}_{ic}\) of reference signals imposed by coordinator c. Henceforth, agent i must carefully balance the three in order to achieve its local performance measure. Mathematically, there assumes existence of an integral-quadratic form (IQF) performance-measure \(J_{i}: U_{i} \times Z_{i}\mapsto \mathbb{R}_{+}\)
where the deterministic matrix-valued functions \(Q_{i}^{f} \in {\mathbb{R}}^{n_{i}\times n_{i}}\), \(Q_{ii} \in \mathcal{C}(t_{0},t_{f}; {\mathbb{R}}^{n_{i}\times n_{i}})\), \(Q_{i} \in \mathcal{C}(t_{0},t_{f}; {\mathbb{R}}^{n_{i}\times n_{i}})\) \(R_{ii} \in \mathcal{C}(t_{0},t_{f}; {\mathbb{R}}^{m_{i}\times m_{i}})\) and \(R_{zi} \in \mathcal{C}(t_{0},t_{f}; {\mathbb{R}}^{q_{i}\times q_{i}})\) representing design parameters for terminal states, transient state estimates for regulation and tracking, regulating efforts and coordination effort mismatches are positive semidefinite with R ii (t) and R zi (t) invertible.
Control of (collective and aggregated) distributed stochastic systems on coordination levels is a major challenge and research theme. The approach to handling the problem with a tuple of two or more control laws is to use the noncooperative game-theoretic paradigm. Particularly, an N-tuple policy \(\{(u_{1}^{{\ast}},z_{1}^{{\ast}}),(u_{2}^{{\ast}},z_{2}^{{\ast}}),\ldots,(u_{N}^{{\ast}},z_{N}^{{\ast}})\}\) is said to constitute a person-by-person equilibrium solution for the coordination control problem (3) and performance measure (16) if
That is, none of the N agents can deviate unilaterally from the equilibrium policies and gain from doing so. The justification for the restriction to such an equilibrium is that the coalition effects u − i ∗ sent to agent i does not necessarily support its preference optimization. Therefore, they cannot do better than behave as if they strive for this equilibrium. It is reasonable to conclude that a person-by-person equilibrium of distributed control is identical to the concept of a Nash equilibrium within a noncooperative game-theoretic setting.
Because admissible feedback policy sets for agent i are not discussed, the determination of a person-by-person equilibrium for the distributed stochastic system is still not straightforward. Therefore, a further restriction is imposed next. For the moment, it will suffice to say that in the case of incomplete information, an admissible 2-tuple feedback policy (u i , z i ) for local best responses to all other immediate neighbors u − i ∗ must be of the form, for some ∂ i ( ⋅, ⋅) and \(\hslash _{i}(\cdot,\cdot )\)
In general, the conditional density \(p_{i}(x_{i}(t)\vert \mathcal{F}_{t}^{i})\), which is the density of x i (t) conditioned on \(\mathcal{F}_{t}^{i}\) (i.e., induced by the observation {y i (τ): τ ∈ [t 0, t]}) represents the sufficient statistics for describing the conditional stochastic effects of future 2-tuple feedback policies (u i , z i ). It is natural that under the Gaussian assumption, the conditional density \(p_{i}(x_{i}(t)\vert \mathcal{F}_{t}^{i})\) is parameterized by the locally available conditional mean \(\hat{x}_{i}(t) \triangleq E\{x_{i}(t)\vert \mathcal{F}_{t}^{i}\}\) and error-estimate covariance \(\Sigma _{i}(t) \triangleq E\{[x_{i}(t) -\hat{ x}_{i}(t)]{[x_{i}(t) -\hat{ x}_{i}(t)]}^{T}\vert \mathcal{F}_{t}^{i}\}\) by incumbent agent i. With respect to the linear-Gaussian conditions, the error-estimate covariances Σ i (t) are independent of feedback policies u i (t) and z i (t) and observations {y i (τ): τ ∈ [t 0, t]}. Hereafter, to look for observer-based optimal control and/or decision policies u i (t) and z i (t) of the form (18) and (19), it is only required that
In view of the linear-quadratic properties of the state-space description (3) and (16), the search for linear time-varying feedback policies generated from the locally accessible state \(\hat{x}_{i}(t)\) is now proceeded to consider
with the feedback policy parameters \(K_{x_{i}} \in \mathcal{C}(t_{0},t_{f}; {\mathbb{R}}^{m_{i}\times n_{i}})\), \(K_{z_{i}} \in \mathcal{C}(t_{0},t_{f}; {\mathbb{R}}^{q_{i}\times n_{i}})\), \(p_{x_{i}} \in \mathcal{C}(t_{0},t_{f}; {\mathbb{R}}^{m_{i}})\) and \(p_{z_{i}} \in \mathcal{C}(t_{0},t_{f}; {\mathbb{R}}^{q_{i}})\) admissible feedback policy parameters whose further defining properties will be stated shortly.
For the given \((t_{0},x_{ai}^{0})\) and subject to the feedback control policies (20)–(21), agent i forms a local awareness of its state recursion (3) and (7) as follows
in which the aggregate Wiener process w ai has the correlations of independent increments, for all \(\tau _{1},\tau _{2} \in [t_{0},t_{f}]\) and W ai > 0
whereas the augmented state variable x ai , its initial-valued condition x ai 0, the system coefficients and parameters are defined by
Moreover, the sample-path function of the random performance measure (16) is now rewritten as below
whereby the corresponding weightings are given by
In views of the linear-quadratic structure of the problem (22) and (23), the performance measure (23) is clearly a random variable with chi-squared type. To account for performance uncertainty, a methodology that enables agent i to select robust decisions under uncertainty from a Pareto front which is acquired using envelopes of a finite set of higher-order statistics associated with (23). This methodology assists the preferences by agent i to be captured perfectly; i.e., what performance attributes that are important to agent i, their trade-off behavior over these attributes and their risk attitude. Recently, the research [6, 7] show how performance uncertainty affects different aspects of risk-averse decision making which can now serve as a starting point for such a knowledge extraction in terms of performance-measure statistics hereafter.
Theorem 1 (Performance-Measure Statistics).
Let the pairs \((A_{ii},B_{ii})\) and \((A_{ii},C_{ii})\) be uniformly stabilizable on [t 0 ,t f ] in the incumbent system i and \(i \in \overline{I}\) governed by (22) and (23) . Then for the given initial condition \((t_{0},x_{i}^{0})\) , incumbent agent i obtains the k i -th cumulant associated with (23)
whereby the supporting variables \(\{H_{i}(s,r)\}_{r=1}^{k_{i}}\), \(\{\breve{D}_{i}(s,r)\}_{r=1}^{k_{i}}\) and \(\{D_{i}(s,r)\}_{r=1}^{k_{i}}\) satisfy the time-backward differential equations (with the dependence of H i (s,r), \(\breve{D}_{i}(s,r)\) and D i (s,r) upon the admissible \(K_{x_{i}}\), \(K_{z_{i}}\), \(p_{x_{i}}\) and \(p_{z_{i}}\) suppressed)
whereby the terminal-value conditions \(H_{i}(t_{f},1) = Q_{ai}^{f}\), \(H_{i}(t_{f},r) = 0\) for 2 ≤ r ≤ k i; \(\breve{D}_{i}(t_{f},1) = S_{ai}^{f}\), \(\breve{D}_{i}(t_{f},r) = 0\) for 2 ≤ r ≤ k i ; and \(D_{i}(t_{f},1) =\zeta _{ i}^{T}(t_{f})Q_{i}^{f}\zeta _{i}(t_{f})\), \(D_{i}(t_{f},r) = 0\) for 2 ≤ r ≤ k i.
Proof.
A key challenge of the problem at hand is to come up with a tractable way to handle performance uncertainty such that its probabilistic nature is manageable. Therefore, only its statistics can be optimized. Most researchers find it easier to understand or describe a random variable through both moment and cumulant generating functions.
Precisely stated, it is necessary to parameterize the initial condition \((t_{0},x_{ai}^{0})\) as any arbitrary pair (s, x ai s). Then, for the given admissible affine inputs \(p_{x_{i}}\) and \(p_{z_{i}}\) in addition with admissible feedback gains \(K_{x_{i}}\) and \(K_{z_{i}}\), the “running” version of performance measure (23) is introduced as follows
The moment-generating function associated with agent i of (31) is defined by
for some small parameters θ i in an open interval about 0. Thus, the cumulant-generating function immediately follows
for some θ i in some (possibly smaller) open interval about 0 while ln ⋅denotes the natural logarithmic transformation.
For notational simplicity, it is convenient to define \(\varpi _{i}\left (s,x_{ai}^{s};\theta _{i}\right ) \triangleq \) \(\exp \left \{\theta _{i}J_{i}\left (s,x_{ai}^{s}\right )\right \}\) and \(\varphi _{i}\left (s,x_{ai}^{s};\theta _{i}\right ) \triangleq E\left \{\varpi _{i}\left (s,x_{ai}^{s};\theta _{i}\right )\right \}\) together with the time derivative of
Using the standard Ito’s formula, it yields
Furthermore, the moment-generating function of (31) can also be expressed by
whereby all the supporting entities are going to be determined in the sequel. In particular, the partial derivatives of (35) results in
Equating the expression (34) with that of (36) and having both linear and quadratic terms independent of x ai s yield the following results
wherein \(\upsilon _{i}(s;\theta _{i}) \triangleq \ln \{\varrho _{i}(s;\theta _{i})\}\). At the final time s = t f , it follows that
which in turn yields the terminal-value conditions as \(\Upsilon _{i}(t_{f};\theta _{i}) =\theta _{i}Q_{ai}^{f}\); \(\eta _{i}(t_{f};\theta _{i}) =\theta _{i}S_{ai}^{f}\); and \(\upsilon _{i}(t_{f};\theta _{i}) =\theta _{i}\zeta _{i}^{T}(t_{f})Q_{i}^{f}\zeta _{i}(t_{f})\).
Hereafter, all the higher-order performance-measure statistics associated with the chi-squared random performance measure (31) will be utilized to generate a Pareto front with which incumbent agent i and \(i \in \overline{I}\) is enabled to choose one or more trade-offs between multiple performance attributes and risk attitude. In views of the expression (35) and the definition of (33), the cumulant-generating function or second-order characteristic function of (31) is rewritten as follows
Subsequently, higher-order statistics of the random performance measure (31) that depict the performance uncertainty can now be determined by a Maclaurin series expansion of the cumulant-generating function (40); e.g.,
from which all \(\kappa _{r} \triangleq \left. \frac{{\partial }^{(r)}} {\partial \theta _{i}^{(r)}} \psi _{i}(s,x_{ai}^{s};\theta _{i})\right \vert _{\theta _{i}=0}\) are known as the mathematical statistics or cumulants of the performance measure (31).
Moreover, the series expansion coefficients are computed by using the cumulant-generating function (40)
In view of the definition (41), the rth performance-measure statistic is given by
for any finite 1 ≤ r < ∞. For notational convenience, the change of notations
is introduced. What remains is to show that the solutions H i (s, r), \(\breve{D}_{i}(s,r)\), and D i (s, r) for 1 ≤ r ≤ k i and \(k_{i} \in \mathbb{N}\) indeed satisfy the time-backward matrix, vector, and scalar-valued differential equations (25)–(30). Notice that these differential equations (25)–(30) are readily obtained by successively taking derivatives with respect to θ i of the cumulant-supporting equations (37)–(39) under the assumption of (A ii , B ii ) and (A ii , C ii ) uniformly stabilizable on the interval [t 0, t f ]. □
Furthermore, some attractive properties of the solutions to the cumulant-generating equations (25)–(30), for which the problem of coordination control with risk-averse performance of the class of distributed stochastic systems considered here is therefore well posed, are presented as follows.
Theorem 2 (Existence of Solutions for Performance-Measure Statistics).
Let the pairs (A ii (⋅),B ii (⋅)) and (A ii (⋅),C ii (⋅)) be uniformly stabilizable. Then, for any given \(k_{i} \in \mathbb{N}\) , the cumulant-generating equations (25)–(30) admit unique and bounded solutions \(\left \{H_{i}(\cdot,r)\right \}_{r=1}^{k_{i}}\), \(\left \{\breve{D}_{i}(\cdot,r)\right \}_{r=1}^{k_{i}}\) and \(\left \{D_{i}(\cdot,r)\right \}_{r=1}^{k_{i}}\) on [t 0 ,t f ].
Proof.
Under the assumption of stabilizability, there always exist some feedback parameters \(K_{x_{i}}(\cdot )\) and \(K_{z_{i}}(\cdot )\) such that the continuous-time aggregate state matrix A ai ( ⋅) is exponentially stable on [t 0, t f ]. According to the results in [8], the state transition matrix Φ ai (t, t 0), associated with the continuous-time composite state matrix A ai ( ⋅), has the following properties
By the matrix variation of constant formula, the unique solutions to the time-backward matrix differential equations (25)–(30) together with the terminal-value conditions are then written as follows
As long as the growth rates of the integrals are not faster than those of exponentially decreasing Φ ai ( ⋅, ⋅) and Φ ai T( ⋅, ⋅) factors, it is therefore concluded that there exist upper bounds on the nonnegative and monotically increasing solutions H i ( ⋅, r), \(\breve{D}_{i}(\cdot,r)\) and D i ( ⋅, r) for any time interval [t 0, t f ]. □
3 Problem Statements
The problem of adapting to performance uncertainty is now addressed by leveraging increased insight into the roles played by performance-measure statistics (24). It is interesting to note that all the performance-measure statistics (24) are functions of time-backward evolutions and do not depend on intermediate recursive state values x ai (t) governed by the state-space representation (22)–(23) for incumbent agent i at each point of time t ∈ [t 0, t f ]. Henceforth, these time-backward evolutions (25)–(30) of which the admissible decision variables \(K_{x_{i}}\), \(K_{z_{i}}\), \(p_{x_{i}}\), and \(p_{z_{i}}\) from the 2-tuple person-by-person equilibrium strategy (20)–(21) are embedded, are therefore considered as the new dynamical equations with the associated state variables H i ( ⋅, r), \(\breve{D}_{i}(\cdot,r)\) and D i ( ⋅, r), not the traditional system states x ai ( ⋅).
To properly develop the problem statements within the concept of the person-by-person equilibrium strategy for agent i and \(i \in \overline{I}\), the new dynamics (25)–(30) based upon the performance-measure statistics of (24) is rewritten in accordance with the following matrix partitions, for 1 ≤ r ≤ k i and \(k_{i} \in \mathbb{N}\)
For notational simplicity, it is now useful to denote the right members of the dynamics (25)–(30) as the mappings
with the rules of action
whereby the k i -tuple \(\mathcal{H}_{i}\), k i -tuple \(\breve{\mathcal{D}}_{i}\), and k i -tuple \(\mathcal{D}_{i}\) variables are defined by
Hence, the product system of dynamical equations in coordination control of the problem class with performance risk aversion becomes
whereby
in addition to the product system of the terminal-value conditions
Once immediate neighbors j ∈ N i of agent i fix the corresponding person-by-person equilibrium strategies u j ∗ and thus the signaling or coordination effects u − i ∗ , agent i then obtains an optimal stochastic control problem with risk-averse performance considerations. The construction of agent i’s person-by-person policy now involves the 4-tuple \((K_{x_{i}}\), \(K_{z_{i}},p_{x_{i}},p_{z_{i}})\). Furthermore, the solutions of the equations (44)–(46) also depend on the admissible feedback gains \(K_{x_{i}}\) and \(K_{z_{i}}\), in addition to the affine inputs \(p_{x_{i}}\) and \(p_{z_{i}}\). In the sequel and elsewhere, when this dependence is needed to be clear, then the notations \(\mathcal{H}_{i}(s,K_{x_{i}},K_{z_{i}};u_{-i}^{{\ast}})\), \(\breve{\mathcal{D}}_{i}(s,K_{x_{i}},K_{z_{i}},p_{x_{i}},p_{z_{i}};u_{-i}^{{\ast}})\) and \(\mathcal{D}_{i}(s,K_{x_{i}},K_{z_{i}},p_{x_{i}},p_{z_{i}};u_{-i}^{{\ast}})\) should be used to denote the solution trajectories of the dynamics (44)–(46) with the admissible 5-tuple \((K_{x_{i}}\), \(K_{z_{i}},p_{x_{i}},p_{z_{i}};u_{-i}^{{\ast}})\).
For the given terminal data \((t_{f},\mathcal{H}_{i}^{f},\breve{\mathcal{D}}_{i}^{f},\mathcal{D}_{i}^{f})\), the theoretical framework for risk-averse control of the distributed stochastic system with possibly noncooperative u − i ∗ , is then analyzed by a class of admissible feedback policies employed by agent i.
Definition 1 (Admissible Feedback Policies).
Let compact subsets \({\overline{K}}^{x_{i}} \subset {\mathbb{R}}^{m_{i}\times n_{i}}\), \({\overline{K}}^{z_{i}} \subset {\mathbb{R}}^{q_{i}\times n_{i}}\), \({\overline{P}}^{x_{i}} \subset {\mathbb{R}}^{m_{i}}\), and \({\overline{P}}^{z_{i}} \subset {\mathbb{R}}^{q_{i}}\) be the sets of allowable feedback form values available at agent i and \(i \in \overline{I}\). For the given \(k_{i} \in \mathbb{N}\) and sequence \(\mu _{i} =\{\mu _{ r}^{i} \geq 0\}_{r=1}^{k_{i}}\) with μ 1 i > 0, the set of feedback gains \(\mathcal{K}_{t_{f},\mathcal{H}_{i}^{f},\breve{\mathcal{D}}_{i}^{f},\mathcal{D}_{i}^{f};\mu _{i}}^{x_{i}}\), \(\mathcal{K}_{t_{f},\mathcal{H}_{i}^{f},\breve{\mathcal{D}}_{i}^{f},\mathcal{D}_{i}^{f};\mu _{i}}^{z_{i}}\), \(\mathcal{P}_{t_{f},\mathcal{H}_{i}^{f},\breve{\mathcal{D}}_{i}^{f},\mathcal{D}_{i}^{f};\mu _{i}}^{x_{i}}\) and \(\mathcal{P}_{t_{f},\mathcal{H}_{i}^{f},\breve{\mathcal{D}}_{i}^{f},\mathcal{D}_{i}^{f};\mu _{i}}^{z_{i}}\) are assumed to be the classes of \(\mathcal{C}(t_{0},t_{f}; {\mathbb{R}}^{m_{i}\times n_{i}})\), \(\mathcal{C}(t_{0},t_{f}; {\mathbb{R}}^{q_{i}\times n_{i}})\), \(\mathcal{C}(t_{0},t_{f}; {\mathbb{R}}^{m_{i}})\) and \(\mathcal{C}(t_{0},t_{f}; {\mathbb{R}}^{q_{i}})\) with values \(K_{x_{i}}(\cdot ) \in {\overline{K}}^{x_{i}}\), \(K_{z_{i}}(\cdot ) \in {\overline{K}}^{z_{i}}\), \(p_{x_{i}}(\cdot ) \in {\overline{P}}^{x_{i}}\) and \(p_{z_{i}}(\cdot ) \in {\overline{P}}^{z_{i}}\), for which the solutions to the dynamic equations (44)–(46) with the terminal-value conditions \(\mathcal{H}_{i}(t_{f}) = \mathcal{H}_{i}^{f}\), \(\breve{\mathcal{D}}_{i}(t_{f}) =\breve{ \mathcal{D}}_{i}^{f}\) and \(\mathcal{D}_{i}(t_{f}) = \mathcal{D}_{i}^{f}\) exist on the interval of optimization [t 0, t f ].
To determine agent i’s the person-by-person equilibrium strategy with risk bearing so as to minimize its performance vulnerability of (23) against all the sample-path realizations from uncertain environments w ai and noncooperative coordination u − i ∗ from immediate neighbors j and j ∈ N i , performance risks are henceforth interpreted as worries and fears about certain undesirable characteristics of performance distributions of (23) and thus are proposed to manage through a finite set of selective weights. This custom set of design freedoms representing particular uncertainty aversions is hence different from the ones with aversion to risk captured in risk-sensitive optimal control [9, 10].
On \(\mathcal{K}_{t_{f},\mathcal{H}_{i}^{f},\breve{\mathcal{D}}_{i}^{f},\mathcal{D}_{i}^{f};\mu _{i}}^{x_{i}}\), \(\mathcal{K}_{t_{f},\mathcal{H}_{i}^{f},\breve{\mathcal{D}}_{i}^{f},\mathcal{D}_{i}^{f};\mu _{i}}^{z_{i}}\), \(\mathcal{P}_{t_{f},\mathcal{H}_{i}^{f},\breve{\mathcal{D}}_{i}^{f},\mathcal{D}_{i}^{f};\mu _{i}}^{x_{i}}\) and \(\mathcal{P}_{t_{f},\mathcal{H}_{i}^{f},\breve{\mathcal{D}}_{i}^{f},\mathcal{D}_{i}^{f};\mu _{i}}^{z_{i}}\), the performance index with risk-value considerations in risk-averse decision making is subsequently defined as follows.
Definition 2 (Risk-Value Aware Performance Index).
Let incumbent agent i and \(i \in \overline{I}\) select \(k_{i} \in \mathbb{N}\) and the sequence of scalar coefficients \(\mu _{i} =\{\mu _{ r}^{i} \geq 0\}_{r=1}^{k_{i}}\) with μ 1 i > 0. Then for the given initial condition \((t_{0},x_{i}^{0})\), the risk-value aware performance index, \(\phi _{i}^{0}:\{ t_{0}\} \times {({\mathbb{R}}^{n_{i}\times n_{i}})}^{k_{i}} \times {({\mathbb{R}}^{n_{i}})}^{k_{i}} \times {\mathbb{R}}^{k_{i}}\mapsto {\mathbb{R}}^{+}\) pertaining to risk-averse decision making of agent i over [t 0, t f ] is defined by
wherein the additional design freedom by means of μ r i’s utilized by agent i with risk-averse attitudes are sufficient to meet and exceed different levels of performance-based reliability requirements, for instance, mean (i.e., the average of performance measure), variance (i.e., the dispersion of values of performance measure around its mean), skewness (i.e., the anti-symmetry of the density of performance measure), kurtosis (i.e., the heaviness in the density tails of performance measure), etc., pertaining to closed-loop performance variations and uncertainties while the supporting solutions \(\{\mathcal{H}_{i}^{r}(s)\}_{r=1}^{k_{i}}\), \(\{\breve{\mathcal{D}}_{i}^{r}(s)\}_{r=1}^{k_{i}}\) and \(\{\mathcal{D}_{i}^{r}(s)\}_{r=1}^{k_{i}}\) evaluated at s = t 0 satisfy the dynamical equations (44)–(46).
To specifically indicate the dependence of the risk-value aware performance index (47) expressed in Mayer form on \((K_{x_{i}},K_{z_{i}},p_{x_{i}},p_{z_{i}})\) and the signaling effects u − i ∗ issued by all immediate neighbors j from N i , the multi-attribute utility function or performance index (47) for agent i is now rewritten explicitly as \(\phi _{i}^{0}(K_{x_{i}},K_{z_{i}},p_{x_{i}},p_{z_{i}};u_{-i}^{{\ast}})\).
Definition 3 (Nash Equilibrium Solution).
An N-tuple of policies \(\{(K_{x_{1}}^{{\ast}},K_{z_{1}}^{{\ast}},p_{x_{1}}^{{\ast}},p_{z_{1}}^{{\ast}}),\ldots,(K_{x_{N}}^{{\ast}},K_{z_{N}}^{{\ast}},p_{x_{N}}^{{\ast}},p_{z_{N}}^{{\ast}})\}\) is said to constitute a Nash equilibrium solution for the distributed N-agent stochastic game if, for all \(i \in \overline{N}\), the Nash inequality condition holds
For the sake of time consistency and subgame perfection, a Nash equilibrium solution is required to have an additional property that its restriction on the interval [t 0, τ] is also a Nash solution to the truncated version of the original problem, defined on [t 0, τ]. With such a restriction so defined, the Nash equilibrium solution is now termed as a feedback Nash equilibrium solution, which is now free of any informational nonuniqueness, and thus whose derivation allows a dynamic programming type argument.
Definition 4 (Feedback Nash Equilibrium).
Let \((K_{x_{i}}^{{\ast}},K_{z_{i}}^{{\ast}},p_{x_{i}}^{{\ast}},p_{z_{i}}^{{\ast}})\) constitute a feedback Nash strategy for agent i such that
for admissible \(K_{x_{i}} \in \mathcal{K}_{t_{f},\mathcal{H}_{i}^{f},\breve{\mathcal{D}}_{i}^{f},\mathcal{D}_{i}^{f};\mu _{i}}^{x_{i}}\), \(K_{z_{i}} \in \mathcal{K}_{t_{f},\mathcal{H}_{i}^{f},\breve{\mathcal{D}}_{i}^{f},\mathcal{D}_{i}^{f};\mu _{i}}^{z_{i}}\), \(p_{x_{i}} \in \mathcal{P}_{t_{f},\mathcal{H}_{i}^{f},\breve{\mathcal{D}}_{i}^{f},\mathcal{D}_{i}^{f};\mu _{i}}^{x_{i}}\) and \(p_{z_{i}} \in \mathcal{P}_{t_{f},\mathcal{H}_{i}^{f},\breve{\mathcal{D}}_{i}^{f},\mathcal{D}_{i}^{f};\mu _{i}}^{z_{i}}\), upon which the solutions to the dynamical systems (44)–(46) exist on [t 0, t f ].
Then, \(\{(K_{x_{1}}^{{\ast}},K_{z_{1}}^{{\ast}},p_{x_{1}}^{{\ast}},p_{z_{1}}^{{\ast}}),\ldots,(K_{x_{N}}^{{\ast}},K_{z_{N}}^{{\ast}},p_{x_{N}}^{{\ast}},p_{z_{N}}^{{\ast}})\}\) when restricted to the interval [t 0, τ] is still an N-tuple feedback Nash equilibrium solution for the multiperson Nash decision problem with the appropriate terminal-value condition \((\tau,\mathcal{H}_{i}^{{\ast}}(\tau ),\breve{\mathcal{D}}_{i}^{{\ast}}(\tau ),\mathcal{D}_{i}^{{\ast}}(\tau ))\) for all τ ∈ [t 0, t f ].
In conformity with the rigorous formulation of dynamic programming, the following development is important. Let the terminal time t f and 3-tuple states \((\mathcal{H}_{i}^{f},\breve{\mathcal{D}}_{i}^{f},\mathcal{D}_{i}^{f})\), the other end condition involved the initial time t 0 and 3-tuple states \((\mathcal{H}_{i}^{0},\breve{\mathcal{D}}_{i}^{0},\mathcal{D}_{i}^{0})\) be specified by a target set requirement.
Definition 5 (Target Sets).
\((t_{0},\mathcal{H}_{i}^{0},\breve{\mathcal{D}}_{i}^{0},\mathcal{D}_{i}^{0}) \in \mathcal{M}_{i}\), where the target set \(\mathcal{M}_{i}\) is a closed subset of \([t_{0},t_{f}] \times {({\mathbb{R}}^{n_{i}\times n_{i}})}^{k_{i}} \times {({\mathbb{R}}^{n_{i}})}^{k_{i}} \times {\mathbb{R}}^{k_{i}}\).
Now, the decision optimization residing at incumbent agent i is to minimize the risk-value aware performance index (47) over admissible feedback strategies composed by \(K_{x_{i}} \equiv K_{x_{i}}(\cdot ) \in \mathcal{K}_{t_{f},\mathcal{H}_{i}^{f},\breve{\mathcal{D}}_{i}^{f},\mathcal{D}_{i}^{f};\mu _{i}}^{x_{i}}\), \(K_{z_{i}} \equiv K_{z_{i}}(\cdot ) \in \mathcal{K}_{t_{f},\mathcal{H}_{i}^{f},\breve{\mathcal{D}}_{i}^{f},\mathcal{D}_{i}^{f};\mu _{i}}^{z_{i}}\), \(p_{x_{i}} \equiv p_{x_{i}}(\cdot ) \in \mathcal{P}_{t_{f},\mathcal{H}_{i}^{f},\breve{\mathcal{D}}_{i}^{f},\mathcal{D}_{i}^{f};\mu _{i}}^{x_{i}}\) and \(p_{z_{i}} \equiv p_{z_{i}}(\cdot ) \in \mathcal{P}_{t_{f},\mathcal{H}_{i}^{f},\breve{\mathcal{D}}_{i}^{f},\mathcal{D}_{i}^{f};\mu _{i}}^{z_{i}}\) while subject to interconnection links from all immediate neighbors with corresponding feedback Nash policies u − i ∗ .
Definition 6 (Optimization of Mayer Problem).
Given the sequence of scalars \(\mu _{i} =\{\mu _{ r}^{i} \geq 0\}_{r=1}^{{k}^{i} }\) with μ 1 i > 0, the decision optimization over [t 0, t f ] is given by
subject to the dynamical equations (44)–(46) on [t 0, t f ].
Notice that the optimization considered here is in Mayer form and can be solved by applying an adaptation of the Mayer form verification results as given in [11]. To embed this optimization facing agent i into a larger problem, the terminal time and states \((t_{f},\mathcal{H}_{i}^{f},\breve{\mathcal{D}}_{i}^{f},\mathcal{D}_{i}^{f})\) are parameterized as \((\varepsilon,\mathcal{Y}_{i},\breve{\mathcal{Z}}_{i},\mathcal{Z}_{i})\), whereby \(\mathcal{Y}_{i} \triangleq \mathcal{H}_{i}(\varepsilon )\), \(\breve{\mathcal{Z}}_{i} \triangleq \breve{\mathcal{D}}_{i}(\varepsilon )\) and \(\mathcal{Z}_{i} \triangleq \mathcal{D}_{i}(\varepsilon )\). Thus, the value function for this optimization problem is now depending on the parameterization of terminal-value conditions.
Definition 7 (Value Function).
Suppose \((\varepsilon,\mathcal{Y}_{i},\breve{\mathcal{Z}}_{i},\mathcal{Z}_{i}) \in [t_{0},t_{f}] \times {({\mathbb{R}}^{n_{i}\times n_{i}})}^{k_{i}} \times {({\mathbb{R}}^{n_{i}})}^{k_{i}} \times {\mathbb{R}}^{k_{i}}\) is given and fixed. Then, the value function \(\mathcal{V}_{i}(\varepsilon,\mathcal{Y}_{i},\breve{\mathcal{Z}}_{i},\mathcal{Z}_{i})\) is defined by
For convention, \(\mathcal{V}_{i}(\varepsilon,\mathcal{Y}_{i},\breve{\mathcal{Z}}_{i},\mathcal{Z}_{i}) \triangleq \infty \) when \(\mathcal{K}_{t_{f},\mathcal{H}_{i}^{f},\breve{\mathcal{D}}_{i}^{f},\mathcal{D}_{i}^{f};\mu _{i}}^{x_{i}} \times \mathcal{K}_{t_{ f},\mathcal{H}_{i}^{f},\breve{\mathcal{D}}_{i}^{f},\mathcal{D}_{i}^{f};\mu _{i}}^{z_{i}} \times \mathcal{P}_{t_{ f},\mathcal{H}_{i}^{f},\breve{\mathcal{D}}_{i}^{f},\mathcal{D}_{i}^{f};\mu _{i}}^{x_{i}} \times \mathcal{P}_{t_{ f},\mathcal{H}_{i}^{f},\breve{\mathcal{D}}_{i}^{f},\mathcal{D}_{i}^{f};\mu _{i}}^{z_{i}}\) is empty. Next, some candidates for the value function are constructed with the help of the concept of reachable set.
Definition 8 (Reachable Sets).
Let a reachable set be defined by \(\mathcal{Q}_{i} \triangleq \Big\{ (\varepsilon,\mathcal{Y}_{i},\breve{\mathcal{Z}}_{i},\mathcal{Z}_{i}) \in [t_{0},t_{f}] \times {({\mathbb{R}}^{n_{i}\times n_{i}})}^{k_{i}} \times {({\mathbb{R}}^{n_{i}})}^{k_{i}} \times {\mathbb{R}}^{k_{i}}\) such that the Cartesian product \(\mathcal{K}_{t_{f},\mathcal{H}_{i}^{f},\breve{\mathcal{D}}_{i}^{f},\mathcal{D}_{i}^{f};\mu _{i}}^{x_{i}}\times \mathcal{K}_{t_{ f},\mathcal{H}_{i}^{f},\breve{\mathcal{D}}_{i}^{f},\mathcal{D}_{i}^{f};\mu _{i}}^{z_{i}}\times \mathcal{P}_{t_{ f},\mathcal{H}_{i}^{f},\breve{\mathcal{D}}_{i}^{f},\mathcal{D}_{i}^{f};\mu _{i}}^{x_{i}}\times \mathcal{P}_{t_{ f},\mathcal{H}_{i}^{f},\breve{\mathcal{D}}_{i}^{f},\mathcal{D}_{i}^{f};\mu _{i}}^{z_{i}}\neq \varnothing \Big\}\).
Moreover, it can be shown that the value function associated with agent i is satisfying a partial differential equation at interior points of \(\mathcal{Q}_{i}\), at which it is differentiable.
Theorem 3 (Hamilton–Jacobi–Bellman Equation–Mayer Problem).
Let \((\varepsilon,\mathcal{Y}_{i},\breve{\mathcal{Z}}_{i},\mathcal{Z}_{i})\) be any interior point of the reachable set \(\mathcal{Q}_{i}\) , at which the value function \(\mathcal{V}_{i}(\varepsilon,\mathcal{Y}_{i},\breve{\mathcal{Z}}_{i},\mathcal{Z}_{i})\) is differentiable. If there exists a feedback Nash strategy which is supported by \(K_{x_{i}}^{{\ast}}(\cdot ) \in \mathcal{K}_{t_{f},\mathcal{H}_{i}^{f},\breve{\mathcal{D}}_{i}^{f},\mathcal{D}_{i}^{f};\mu _{i}}^{x_{i}}\), \(K_{z_{i}}^{{\ast}}(\cdot ) \in \mathcal{K}_{t_{f},\mathcal{H}_{i}^{f},\breve{\mathcal{D}}_{i}^{f},\mathcal{D}_{i}^{f};\mu _{i}}^{z_{i}}\), \(p_{x_{i}}^{{\ast}}(\cdot ) \in \mathcal{P}_{t_{f},\mathcal{H}_{i}^{f},\breve{\mathcal{D}}_{i}^{f},\mathcal{D}_{i}^{f};\mu _{i}}^{x_{i}}\) and \(p_{z_{i}}^{{\ast}}(\cdot ) \in \mathcal{P}_{t_{f},\mathcal{H}_{i}^{f},\breve{\mathcal{D}}_{i}^{f},\mathcal{D}_{i}^{f};\mu _{i}}^{z_{i}}\) , then the differential equation
is satisfied whereby \(\mathcal{V}_{i}(t_{0},\mathcal{Y}_{i}(t_{0}),\breve{\mathcal{Z}}_{i}(t_{0}),\mathcal{Z}_{i}(t_{0})) =\phi _{ i}^{0}(\mathcal{H}_{i}(t_{0}),\breve{\mathcal{D}}_{i}(t_{0}),\mathcal{D}_{i}(t_{0}))\).
Proof.
By what have been shown in the recent results by the author [7], the proof for the result herein is readily proven. □
Finally, the following result gives the sufficient condition used to verify a feedback Nash strategy for incumbent agent i and \(i \in \overline{I}\).
Theorem 4 (Verification Theorem).
Let \(\mathcal{W}_{i}(\varepsilon,\mathcal{Y}_{i},\breve{\mathcal{Z}}_{i},\mathcal{Z}_{i})\) be continuously differentiable solution of the Hamilton–Jacobi–Bellman (HJB) equation (51) , which satisfies the boundary condition
Let \((t_{f},\mathcal{H}_{i}^{f},\breve{\mathcal{D}}_{i}^{f},\mathcal{D}_{i}^{f})\) be a 4-tuple point in \(\mathcal{Q}_{i}\) ; let \(K_{x_{i}} \in \mathcal{K}_{t_{f},\mathcal{H}_{i}^{f},\breve{\mathcal{D}}_{i}^{f},\mathcal{D}_{i}^{f};\mu _{i}}^{x_{i}}\), \(K_{z_{i}} \in \mathcal{K}_{t_{f},\mathcal{H}_{i}^{f},\breve{\mathcal{D}}_{i}^{f},\mathcal{D}_{i}^{f};\mu _{i}}^{z_{i}}\), \(p_{x_{i}} \in \mathcal{P}_{t_{f},\mathcal{H}_{i}^{f},\breve{\mathcal{D}}_{i}^{f},\mathcal{D}_{i}^{f};\mu _{i}}^{x_{i}}\), \(p_{z_{i}} \in \mathcal{P}_{t_{f},\mathcal{H}_{i}^{f},\breve{\mathcal{D}}_{i}^{f},\mathcal{D}_{i}^{f};\mu _{i}}^{z_{i}}\) ; and let \(\mathcal{H}_{i}(\cdot )\), \(\breve{\mathcal{D}}_{i}(\cdot )\) and \(\mathcal{D}_{i}(\cdot )\) be the corresponding solutions of the equations of motion (44)–(46).Then, \(\mathcal{W}_{i}(s,\mathcal{H}_{i}(s),\breve{\mathcal{D}}_{i}(s),\mathcal{D}_{i}(s))\) is time-backward increasing function of s.
If \(K_{x_{i}}^{{\ast}}\in \mathcal{K}_{t_{f},\mathcal{H}_{i}^{f},\breve{\mathcal{D}}_{i}^{f},\mathcal{D}_{i}^{f};\mu _{i}}^{x_{i}}\), \(K_{z_{i}}^{{\ast}}\in \mathcal{K}_{t_{f},\mathcal{H}_{i}^{f},\breve{\mathcal{D}}_{i}^{f},\mathcal{D}_{i}^{f};\mu _{i}}^{z_{i}}\), \(p_{x_{i}}^{{\ast}}\in \mathcal{P}_{t_{f},\mathcal{H}_{i}^{f},\breve{\mathcal{D}}_{i}^{f},\mathcal{D}_{i}^{f};\mu _{i}}^{x_{i}}\) and \(p_{z_{i}}^{{\ast}}\in \mathcal{P}_{t_{f},\mathcal{H}_{i}^{f},\breve{\mathcal{D}}_{i}^{f},\mathcal{D}_{i}^{f};\mu _{i}}^{z_{i}}\) defining a person-by-person equilibrium or feedback Nash strategy for agent i with the corresponding solutions \(\mathcal{H}_{i}^{{\ast}}(\cdot )\), \(\breve{\mathcal{D}}_{i}^{{\ast}}(\cdot )\) and \(\mathcal{D}_{i}^{{\ast}}(\cdot )\) of the dynamical equations (44)–(46) such that, for s ∈ [t 0 ,t f ]
then \((K_{x_{i}}^{{\ast}},K_{z_{i}}^{{\ast}},p_{x_{i}}^{{\ast}},p_{z_{i}}^{{\ast}})\) results in a feedback Nash strategy or person-by-person equilibrium for agent i in \(\mathcal{K}_{t_{f},\mathcal{H}_{i}^{f},\breve{\mathcal{D}}_{i}^{f},\mathcal{D}_{i}^{f};\mu _{i}}^{x_{i}} \times \mathcal{K}_{t_{ f},\mathcal{H}_{i}^{f},\breve{\mathcal{D}}_{i}^{f},\mathcal{D}_{i}^{f};\mu _{i}}^{z_{i}} \times \mathcal{P}_{t_{ f},\mathcal{H}_{i}^{f},\breve{\mathcal{D}}_{i}^{f},\mathcal{D}_{i}^{f};\mu _{i}}^{x_{i}} \times \mathcal{P}_{t_{ f},\mathcal{H}_{i}^{f},\breve{\mathcal{D}}_{i}^{f},\mathcal{D}_{i}^{f};\mu _{i}}^{z_{i}}\) . Furthermore, it follows that
whereby \(\mathcal{V}_{i}(\varepsilon,\mathcal{Y}_{i},\breve{\mathcal{Z}}_{i},\mathcal{Z}_{i})\) is the value function associated with incumbent agent i.
Proof.
With the aid of the recent development [7], the proof then follows for the verification theorem herein. □
4 Distributed Person-by-Person Equilibrium Strategies
Reflecting on the Mayer-form optimization problem of the person-by-person equilibrium strategy concerned by incumbent agent i and \(i \in \overline{I}\), the technical approach is to apply an adaptation of the Mayer-form verification theorem of dynamic programming as given in [11]. In the framework of dynamic programming, it is often required to denote the terminal time and states of a family of optimization problems as \((\varepsilon,\mathcal{Y}_{i},\breve{\mathcal{Z}}_{i},\mathcal{Z}_{i})\) rather than \((t_{f},\mathcal{H}_{i}^{f},\breve{\mathcal{D}}_{i}^{f},\mathcal{D}_{i}^{f})\). Stating precisely, for \(\varepsilon \in [t_{0},t_{f}]\) and 1 ≤ r ≤ k i , the states of the performance robustness (44)–(46) defined on the interval \([t_{0},\varepsilon ]\) have the terminal values denoted by \(\mathcal{H}_{i}(\varepsilon ) \equiv \mathcal{Y}_{i}\), \(\breve{\mathcal{D}}_{i}(\varepsilon ) \equiv \breve{\mathcal{Z}}_{i}\) and \(\mathcal{D}_{i}(\varepsilon ) \equiv \mathcal{Z}_{i}\).
Since the performance index (47) is quadratic affine in terms of arbitrarily fixed x i 0, the resulting insight suggests a solution to the adapted Hamilton–Jacobi–Bellman equation (51) is of the form as follows. It is assumed that \((\varepsilon,\mathcal{Y}_{i},\breve{\mathcal{Z}}_{i},\mathcal{Z}_{i})\) is any interior point of the reachable set \(\mathcal{Q}_{i}\) at which the real-valued function
is differentiable. The parametric functions of time \(\mathcal{E}_{i}^{r} \in {\mathcal{C}}^{1}(t_{0},t_{f}; {\mathbb{R}}^{n_{i}\times n_{i}})\), \(\breve{\mathcal{T}}_{i}^{r} \in {\mathcal{C}}^{1}(t_{0},t_{f}; \mathbb{R}_{i}^{n})\) and \(\mathcal{T}_{i}^{r} \in {\mathcal{C}}^{1}(t_{0},t_{f}; \mathbb{R})\) are yet to be determined. Furthermore, the time derivative of \(\mathcal{W}_{i}(\varepsilon,\mathcal{Y}_{i},\breve{\mathcal{Z}}_{i},\mathcal{Z}_{i})\) can be shown to be
The substitution of this hypothesized solution (55) into the HJB equation (51) and making use of (56) results in
Differentiating the expression within the bracket of (57) with respect to \(K_{x_{i}}\), \(K_{z_{i}}\), \(p_{x_{i}}\) and \(p_{z_{i}}\) yields the necessary conditions for an extremum of (51) on \([t_{0},\varepsilon ]\),
Because all \(x_{i}^{0}{(x_{i}^{0})}^{T}\), \({(x_{i}^{0})}^{T}\) and x i 0 have arbitrary ranks of one, it must be true that
Replacing these results (58)–(61) into the right member of the HJB equation (51) yields the value of the minimum whose mathematical details are omitted herein for the purpose of brevity.
For each agent i and \(i \in \overline{I}\), it is necessary to exhibit \(\left \{\mathcal{E}_{i}^{r}(\cdot )\right \}_{r=1}^{k_{i}}\), \(\{\breve{\mathcal{T}}_{i}^{r}(\cdot )\}_{r=1}^{k_{i}}\) and \(\left \{\mathcal{T}_{i}^{r}(\cdot )\right \}_{r=1}^{k_{i}}\) which render the left side of the HJB equation (51) equal to zero for \(\varepsilon \in [t_{0},t_{f}]\), when \(\{\mathcal{Y}_{i}^{r}\}_{r=1}^{k_{i}}\), \(\{\breve{\mathcal{Z}}_{i}^{r}\}_{r=1}^{k_{i}}\) and \(\{\mathcal{Z}_{i}^{r}\}_{r=1}^{k_{i}}\) are evaluated along the solution trajectories of the dynamical equations (44)–(46). With a careful examination of the expression (57), it reveals that
will work. Furthermore, the boundary condition associated with the verification theorem requires that
Thus, matching the boundary condition yields the initial value conditions \(\mathcal{E}_{i}^{r}(t_{0}) = 0\), \(\breve{\mathcal{T}}_{i}^{r}(t_{0}) = 0\) and \(\mathcal{T}_{i}^{r}(t_{0}) = 0\) for the forward-in-time differential equations (62)–(67).
Applying the 4-tuple \((K_{x_{i}},K_{z_{i}},p_{x_{i}},p_{z_{i}})\) in (58)–(61) that is defining the person-by-person equilibrium for each agent i and \(i \in \overline{I}\) along the solution trajectories of the backward-in-time differential equations (44)–(46), these equations become the backward-in-time Riccati-type differential equations
where the terminal-value conditions \((\mathcal{H}_{i}^{1})_{11}(t_{f}) = Q_{i}^{f}\), \((\mathcal{H}_{i}^{r})_{11}(t_{f}) = 0\) for 2 ≤ r ≤ k i ; \((\breve{\mathcal{D}}_{i}^{r})_{11}(t_{f}) = -Q_{i}^{f}\zeta _{i}(t_{f})\), \((\breve{\mathcal{D}}_{i}^{r})_{11}(t_{f}) = 0\) for 2 ≤ r ≤ k i ; and \(\mathcal{D}_{i}^{r}(t_{f}) =\zeta _{i}(t_{f})Q_{i}^{f}\zeta _{i}(t_{f})\), \(\mathcal{D}_{i}^{r}(t_{f})\) for 2 ≤ r ≤ k i . Thus, whenever the coupled backward-in-time differential equations (68)–(73) admit the matrix-valued solutions \(\{(\mathcal{H}_{i}^{r})_{11}(\cdot )\}_{r=1}^{k_{i}}\), vector-valued solutions \(\left \{(\breve{\mathcal{D}}_{i}^{r})_{11}(\cdot )\right \}_{r=1}^{k_{i}}\), and scalar-valued solutions \(\{\mathcal{D}_{i}^{r}(\cdot )\}_{r=1}^{k_{i}}\), then the existence of the matrix-valued solutions \(\{\mathcal{E}_{i}^{r}(\cdot )\}_{r=1}^{k_{i}}\), vector-valued solutions \(\left \{\breve{\mathcal{T}}_{i}^{r}(\cdot )\right \}_{r=1}^{k_{i}}\), and scalar-valued solutions \(\{\mathcal{T}_{i}^{r}(\cdot )\}_{r=1}^{k_{i}}\) satisfying the coupled forward-in-time differential equations (62)–(67) are assured.
By comparing the time-forward differential equations (62)–(67) to those of time-backward differential equations (68)–(73), one may recognize that these sets of differential equations are related to one another by
Enforcing the initial-value conditions of \(\mathcal{E}_{i}^{r}(t_{0}) = 0\), \(\breve{\mathcal{T}}_{i}^{r}(t_{0}) = 0\) and \(\mathcal{T}_{i}^{r}(t_{0}) = 0\) uniquely implies the following results
for all \(\varepsilon \in [t_{0},t_{f}]\) and yields a value function
for which the sufficient condition (53) of the verification theorem is satisfied. Therefore, the extremal person-by-person equilibrium policy (58)–(61) minimizing (47) become optimal
The goals in this research investigation have been methodological. A noncooperative game-theoretic methodology for coordination control of distributed stochastic systems is successfully sought for theory building in contexts in which signaling effects are issued by a coordinator and distributed person-by-person equilibrium strategies by autonomous agents i and \(i \in \overline{I}\) are placed toward performance robustness. At this point, it makes sense to integrate all of the contending results into the following unified theorem.
Theorem 5 (Person-by-Person Equilibrium Strategies).
Consider a distributed stochastic system governed by (3)–(16) whose pairs (A ii ,B ii ) and (A ii ,C ii ) are uniformly stabilizable on [t 0 ,t f ]. An N-tuple \(\{(u_{1}^{{\ast}},z_{1}^{{\ast}}),\ldots,(u_{N}^{{\ast}},z_{N}^{{\ast}})\}\) of control policies constitutes a feedback Nash equilibrium for the class of distributed stochastic system considered here. Furthermore, 2-tuple \((u_{i}^{{\ast}},z_{i}^{{\ast}})\) imposing a person-by-person equilibrium strategy for the corresponding agent i and \(i \in \overline{I}\) is implemented forwardly in time by
which strives to optimize the risk-value aware performance index (47)composed by a preferential set of mathematical statistics of the chi-squared cost random variable (16).The construction of the person-by-person equilibrium for each agent i is determined backwardly in time; e.g.,
wherein the normalized preferences \(\hat{\mu }_{i}^{r} \triangleq \mu _{i}^{r}/\mu _{i}^{1}\) ’s are mutually chosen by each incumbent agent i for risk-averse coordinations toward co-design of individual performance robustness. The optimal set of supporting solutions satisfies the time-backward, matrix, vector, and scalar-valued differential equations
Notice that as for comparison with other state-of-the-art research, the principal distinguishing feature of the research investigation herein is the pervasive use of noncooperative game theory and person-by-person equilibrium strategies (78) and (79) across the hierarchy for coordinated control of distributed systems. The emphasis is the recognition of the presence of a coordinator and incumbent systems and thus, addressing an important challenge in performance analysis supported by (84)–(123) for intra- and inter-interactions considered at the outset to achieve the attributes of “desired effects” and “tailored performance.”
5 Conclusions
The present research investigation results in significant contributions to coordination control science’s existing portfolio of methodologies. This portfolio contains a coordinator which directs two or more interconnected stochastic systems. Thinking about risk-averse attitudes toward performance uncertainty suggests new ideas for extending existing theories of distributed control and multiperson decision analysis. In this sense, the present research article suggested that making decisions using the proposed method protects decision makers and/or controller designers from overly optimistic design decisions that may not be the best under uncertainty. To account for mutual influence from immediate neighbors that give rise to interaction complexity such as potential noncooperation, each incumbent system or self-directed agent autonomously focuses on the search for a person-by-person equilibrium which is in turn remotely supported by local observers. Further discussions showed that the person-by-person equilibrium is equivalent to the concept of feedback Nash strategy. Another research issue discussed includes adjusting risk-averse attitudes via risk-value aware performance indices. The process of adjustment for performance risk aversion imposes some computational requirements as needed by the construction of the states of the person-by-person equilibrium.
Future work will focus on distributed multiscale modeling and control with explicit communications and partial information patterns, wherein research issues are: (a) how the feedback of incumbent systems would affect macroscales and macrostates of dominant coordinators? (b) how fast, small-scale behavior of incumbent systems could potentially trigger conformation changes of dominant coordinators? and (c) reliable and effective pathways for transferring information and knowledge from dominant players to fringe players and vice versa?
References
Friedman JW (1990) Game theory with applications to economics. 2nd edn. New York: Oxford University Press
Fudenberg D and Levine DK (1998) The theory of learning in games. Cambridge, MA: MIT Press
Huang M (2010) Large-population LQG games involving a major player: the Nash certainty equivalence principle. SIAM Journal of Control Optimization, 48(5):3318–3353
Pham KD (2008) Non-cooperative outcomes for stochastic nash games: decision strategies towards multi-attribute performance robustness. The 17th World Congress International Federation on Automatic Control, pp. 11750–11756
Pham KD (2008) New results in stochastic cooperative games: strategic coordination for multi-resolution performance robustness. In: Hirsch MJ, Pardalos PM, Murphey R, Grundel D (eds.) Optimization and Cooperative Control Strategies. Series Lecture Notes in Control and Information Sciences, Vol. 381, pp. 257–285, Springer Berlin, Heidelberg
Pham KD, Liberty SR and Jin G (2008) Multi-cumulant and Pareto solutions for tactics change prediction and performance analysis in stochastic multi-team non-cooperative games. In: Won C-H, Schrader C and Michel AN (eds.) Advances in statistical control, algebraic systems theory and dynamic systems characteristics. Systems & Control: Foundations and Applications, pp. 65–97. Birkhauser, Boston
Pham KD (2011) Performance-reliability aided decision making in multiperson quadratic decision games against jamming and estimation confrontations. Journal of Optimization Theory and Applications, 149(3):559–629
Brockett RW (1970) Finite dimensional linear systems. Wiley, New York
Jacobson DH (1973) Optimal stochastic linear systems with exponential performance criteria and their relation to deterministic games. IEEE Transactions on Automatic Control, 18: 124–131
Whittle P (1990) Risk sensitive optimal control. John Wiley & Sons, New York
Fleming WH and Rishel RW (1975) Deterministic and stochastic optimal control. New York: Springer-Verlag
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2013 Springer Science+Business Media New York
About this paper
Cite this paper
Pham, K.D. (2013). A Framework for Coordination in Distributed Stochastic Systems: Output Feedback and Performance Risk Aversion. In: Sorokin, A., Pardalos, P. (eds) Dynamics of Information Systems: Algorithmic Approaches. Springer Proceedings in Mathematics & Statistics, vol 51. Springer, New York, NY. https://doi.org/10.1007/978-1-4614-7582-8_6
Download citation
DOI: https://doi.org/10.1007/978-1-4614-7582-8_6
Published:
Publisher Name: Springer, New York, NY
Print ISBN: 978-1-4614-7581-1
Online ISBN: 978-1-4614-7582-8
eBook Packages: Mathematics and StatisticsMathematics and Statistics (R0)