Adaptive Markov control processes

TL;DR: In this paper, the authors present an inventory/production system for control of water reservoir management, and a semi-Markov control model for estimating the value of a water reservoir.

Abstract: 1 Controlled Markov Processes.- 1.1 Introduction.- 1.2 Stochastic Control Problems.- Control Models.- Policies.- Performance Criteria.- Control Problems.- 1.3 Examples.- An Inventory/Production System.- Control of Water Reservoirs.- Fisheries Management.- Nonstationary MCM's.- Semi-Markov Control Models.- 1.4 Further Comments.- 2 Discounted Reward Criterion.- 2.1 Introduction.- Summary.- 2.2 Optimality Conditions.- Continuity of ?*.- 2.3 Asymptotic Discount Optimality.- 2.4 Approximation of MCM's.- Nonstationary Value-Iteration.- Finite-State Approximations.- 2.5 Adaptive Control Models.- Preliminaries.- Nonstationary Value-Iteration.- The Principle of Estimation and Control.- Adaptive Policies.- 2.6 Nonparametric Adaptive Control.- The Parametric Approach.- New Setting.- The Empirical Distribution Process.- Nonparametric Adaptive Policies.- 2.7 Comments and References.- 3 Average Reward Criterion.- 3.1 Introduction.- Summary.- 3.2 The Optimality Equation.- 3.3 Ergodicity Conditions.- 3.4 Value Iteration.- Uniform Approximations.- Successive Averagings.- 3.5 Approximating Models.- 3.6 Nonstationary Value Iteration.- Nonstationary Successive Averagings.- Discounted-Like NVI.- 3.7 Adaptive Control Models.- Preliminaries.- The Principle of Estimation and Control (PEC).- Nonstationary Value Iteration (NVI).- 3.8 Comments and References.- 4 Partially Observable Control Models.- 4.1 Introduction.- Summary.- 4.2 PO-CM: Case of Known Parameters.- The PO Control Problem.- 4.3 Transformation into a CO Control Problem.- I-Policies.- The New Control Model.- 4.4 Optimal I-Policies.- 4.5 PO-CM's with Unknown Parameters.- PEC and NVI I-Policies.- 4.6 Comments and References.- 5 Parameter Estimation in MCM's.- 5.1 Introduction.- Summary.- 5.2 Contrast Functions.- 5.3 Minimum Contrast Estimators.- 5.4 Comments and References.- 6 Discretization Procedures.- 6.1 Introduction.- Summary.- 6.2 Preliminaries.- 6.3 The Non-Adaptive Case.- A Non-Recursive Procedure.- A Recursive Procedure.- 6.4 Adaptive Control Problems.- Preliminaries.- Discretization of the PEC Adaptive Policy.- Discretization of the NVI Adaptive Policy.- 6.5 Proofs.- The Non-Adaptive Case.- The Adaptive Case.- 6.6 Comments and References.- Appendix A. Contraction Operators.- Appendix B. Probability Measures.- Total Variation Norm.- Weak Convergence.- Appendix C. Stochastic Kernels.- Appendix D. Multifunctions and Measurable Selectors.- The Hausdorff Metric.- Multifunctions.- References.- Author Index.