Journal Article10.1109/TAC.2004.826725
Dynamic programming equations for discounted constrained stochastic control
R. C. Chen,G.L. Blankenship +1 more
58
TL;DR: The application of the dynamic programming approach to constrained stochastic control problems with expected value constraints is demonstrated and optimality equations are obtained for these problems.
read more
Abstract: In this paper, the application of the dynamic programming approach to constrained stochastic control problems with expected value constraints is demonstrated. Specifically, two such problems are analyzed using this approach. The problems analyzed are the problem of minimizing a discounted cost infinite horizon expectation objective subject to an identically structured constraint, and the problem of minimizing a discounted cost infinite horizon minimax objective subject to a discounted expectation constraint. Using the dynamic programming approach, optimality equations, which are the chief contribution of this paper, are obtained for these problems. In particular, the dynamic programming operators for problems with expectation constraints differ significantly from those of standard dynamic programming and problems with worst-case constraints. For the discounted cost infinite horizon cases, existence and uniqueness of solutions to the dynamic programming equations are explicitly shown by using the Banach fixed point theorem to show that the corresponding dynamic programming operators are contractions. The theory developed is illustrated by numerically solving the constrained stochastic control dynamic programming equations derived for simple example problems. The example problems are based on a two-state Markov model that represents an error prone system that is to be maintained.
read more
Chat with Paper
AI Agents for this Paper
Find similar papers on Google Scholar, PubMed and Arxiv
Write a critical review of this paper
Analyze citations of this paper to find unaddressed research gaps
Citations
•Book
Control Techniques for Complex Networks
Sean P. Meyn
- 01 Jan 2007
TL;DR: The workload model that is the basis of traditional analysis of the single queue becomes a foundation for workload relaxations used in the treatment of complex networks and Lyapunov functions and dynamic programming equations lead to the celebrated MaxWeight policy.
614
•Journal Article
Dynamic programming in constrained Markov decision processes
TL;DR: It is shown that the problem can be reformulated as a standard MDP and solved using the Dynamic Programming approach, and an example on arolled queue is presented.
Stochastic optimal control with dynamic, time-consistent risk constraints
Yinlam Chow,Marco Pavone +1 more
- 17 Jun 2013
TL;DR: A dynamic programming approach to stochastic optimal control problems with dynamic, time-consistent risk constraints, which allows to compute the optimal costs by value iteration and a procedure to construct optimal policies.
Stochastic Control Bounds on Sensor Network Performance
David A. Castanon
- 12 Dec 2005
TL;DR: A computable lower bound on the classification error that can be achieved by a causal adaptive sensing schedule is described, based on solving a partially observed stochastic control problem.
22
Convex analytic approach to constrained discounted Markov decision processes with non-constant discount factors
TL;DR: In this article, the authors developed the convex analytic approach to a discounted discrete-time Markov decision process (DTMDP) in Borel state and action spaces with N constraints, and proved that every extreme point of the space of occupation measures can be generated by a deterministic stationary policy for the DTMDP.
17
References
•Book
Dynamic Programming and Optimal Control
Dimitri P. Bertsekas
- 01 May 1995
TL;DR: The leading and most up-to-date textbook on the far-ranging algorithmic methododogy of Dynamic Programming, which can be used for optimal control, Markovian decision problems, planning and sequential decision making under uncertainty, and discrete/combinatorial optimization.
Optimal stochastic linear systems with exponential performance criteria and their relation to deterministic differential games
TL;DR: In this article, two stochastic optimal control problems are solved whose performance criteria are the expected values of exponential functions of quadratic forms, and the optimal controller is linear in both cases but depends upon the covariance matrix of the additive process noise so that the certainty equivalence principle does not hold.
811
Risk-sensitive linear/quadratic/gaussian control
TL;DR: In this paper, the conventional linear/quadratic/Gaussian assumptions are modified in that minimization of the expectation of cost G defined by (2) is replaced by minimisation of the criterion function (5).
579
•Book
Adaptive Markov control processes
Onésimo Hernández-Lerma
- 01 May 1989
TL;DR: In this paper, the authors present an inventory/production system for control of water reservoir management, and a semi-Markov control model for estimating the value of a water reservoir.
413
Related Papers (5)
Eitan Altman
- 30 Mar 1999
Dimitri P. Bertsekas
- 01 May 1995
E. Fernandez-Gaucherand,Ari Arapostathis,S.I. Marcus +2 more
- 16 Dec 1992