Abstract
We consider a stochastic control problem over an infinite horizon where the state process is influenced by an unobservable environment process. In particular, the Hidden-Markov-model and the Bayesian model are included. This model under partial information is transformed into an equivalent one with complete information by using the well-known filter technique. In particular, the optimal controls and the value functions of the original and the transformed problem are the same. An explicit representation of the filter process which is a piecewise-deterministic process, is also given. Then we propose two solution techniques for the transformed model. First, a generalized verification technique (with a generalized Hamilton–Jacobi–Bellman equation) is formulated where the strict differentiability of the value function is weaken to local Lipschitz continuity. Second, we present a discrete-time Markovian decision model by which we are able to compute an optimal control of our given problem. In this context we are also able to state a general existence result for optimal controls. The power of both solution techniques is finally demonstrated for a parallel queueing model with unknown service rates. In particular, the filter process is discussed in detail, the value function is explicitly computed and the optimal control is completely characterized in the symmetric case.
Article PDF
Similar content being viewed by others
Avoid common mistakes on your manuscript.
References
Altman E, Jimenez T, Nunez-Queija R, Yechiali U (2003) Optimal routing among ./M/1 queues with partial information. INRIA Research Report No. 4985
Asmussen S (2003) Applied probabilities and queues. Springer, Berlin
Bäuerle N, Rieder U (2007) Portfolio optimization with jumps and unobservable intensity process. Math Finance 17(2): 205–224
Bensoussan A, Cakanyildirim M, Sethi S (2003) Partially observed inventory systems. In: Proceedings of the 44th IEEE conference on decision and control, pp 1023–1028
Bertsekas D, Shreve S (1978) Stochastic optimal control: the discrete time case. Academic Press, Dublin
Brémaud P (1981) Point processes and queues. Springer, Berlin
Clarke F (1983) Optimization and nonsmooth analysis. Wiley, New York
Davis D (1993) Markov models and optimization. Chapman & Hall, London
Donchev D (1998) On the two-armed bandit problem with non-observed Poissonian switching of arms. Math Methods Oper Res 47: 401–422
Donchev D (1999) Exact solution of the Bellman equation for a β-discounted reward in a two-armed bandit with switching arms. J Appl Math Stoch Anal 12(2): 151–160
Elliott R, Aggoun R, Moore J (1997) Hidden Markov models: estimation and control. Springer, Berlin
Honhon D, Seshadri S (2007) Admission control with incomplete information to a finite buffer queue. Probab Eng Inform Sci 21(1): 19–46
Lin K, Ross S (2003) Admission control with incomplete information of a queueing system. INFORMS Oper Res 51: 645–654
Liptser R, Shiryayev A (2004) Statistics of random processes. Springer, Berlin
Rogers L, Williams D (2003) Diffusions, Markov processes and martingales. Cambridge University Press, Cambridge
Winter J (2008) Optimal control of Markovian jump processes with different information structures. PhD-Thesis, Universität Ulm
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Rieder, U., Winter, J. Optimal control of Markovian jump processes with partial information and applications to a parallel queueing model. Math Meth Oper Res 70, 567–596 (2009). https://doi.org/10.1007/s00186-009-0284-7
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00186-009-0284-7