Markov chain approximation method

Abstract: Introduction 1 Review of Continuous Time Models 1.1 Martingales and Martingale Inequalities 1.2 Stochastic Integration 1.3 Stochastic Differential Equations: Diffusions 1.4 Reflected Diffusions 1.5 Processes with Jumps 2 Controlled Markov Chains 2.1 Recursive Equations for the Cost 2.2 Optimal Stopping Problems 2.3 Discounted Cost 2.4 Control to a Target Set and Contraction Mappings 2.5 Finite Time Control Problems 3 Dynamic Programming Equations 3.1 Functionals of Uncontrolled Processes 3.2 The Optimal Stopping Problem 3.3 Control Until a Target Set Is Reached 3.4 A Discounted Problem with a Target Set and Reflection 3.5 Average Cost Per Unit Time 4 Markov Chain Approximation Method: Introduction 4.1 Markov Chain Approximation 4.2 Continuous Time Interpolation 4.3 A Markov Chain Interpolation 4.4 A Random Walk Approximation 4.5 A Deterministic Discounted Problem 4.6 Deterministic Relaxed Controls 5 Construction of the Approximating Markov Chains 5.1 One Dimensional Examples 5.2 Numerical Simplifications 5.3 The General Finite Difference Method 5.4 A Direct Construction 5.5 Variable Grids 5.6 Jump Diffusion Processes 5.7 Reflecting Boundaries 5.8 Dynamic Programming Equations 5.9 Controlled and State Dependent Variance 6 Computational Methods for Controlled Markov Chains 6.1 The Problem Formulation 6.2 Classical Iterative Methods 6.3 Error Bounds 6.4 Accelerated Jacobi and Gauss-Seidel Methods 6.5 Domain Decomposition 6.6 Coarse Grid-Fine Grid Solutions 6.7 A Multigrid Method 6.8 Linear Programming 7 The Ergodic Cost Problem: Formulation and Algorithms 7.1 Formulation of the Control Problem 7.2 A Jacobi Type Iteration 7.3 Approximation in Policy Space 7.4 Numerical Methods 7.5 The Control Problem 7.6 The Interpolated Process 7.7 Computations 7.8 Boundary Costs and Controls 8 Heavy Traffic and Singular Control 8.1 Motivating Examples &nb

Abstract: A powerful and usable class of methods for numerically approximating the solutions to optimal stochastic control problems for diffusion, reflected diffusion, or jump-diffusion models is discussed. The basic idea involves uconsistent approximation of the model by a Markov chain, and then solving an appropriate optimization problem for the Murkoy chain model. A general method for obtaining a useful approximation is given. All the standard classes of cost functions can be handled here, for illustrative purposes, discounted and average cost per unit time problems with both reflecting and nonreflecting diffusions are concentrated on. Both the drift and the variance can be controlled. Owing to its increasing importance and to lack of material on numerical methods, an application to the control of queueing and production systems in heavy traffic is developed in detail. The methods of proof of convergence are relatively simple, using only some basic ideas in the theory of weak convergence of a sequence of probabi...

Abstract: Mijatovic and Pistorius proposed an efficient Markov chain approximation method for pricing European and barrier options in general one-dimensional Markovian models. However, sharp convergence rates of this method for realistic financial payoffs, which are nonsmooth, are rarely available. In this paper, we solve this problem for general one-dimensional diffusion models, which play a fundamental role in financial applications. For such models, the Markov chain approximation method is equivalent to the method of lines using the central difference. Our analysis is based on the spectral representation of the exact solution and the approximate solution. By establishing the convergence rate for the eigenvalues and the eigenfunctions, we obtain sharp convergence rates for the transition density and the price of options with nonsmooth payoffs. In particular, we show that for call-/put-type payoffs, convergence is second order, while for digital-type payoffs, convergence is generally only first order. Furthermore, we provide theoretical justification for two well-known smoothing techniques that can restore second-order convergence for digital-type payoffs and explain oscillations observed in the convergence for options with nonsmooth payoffs. As an extension, we also establish sharp convergence rates for European options for a rich class of Markovian jump models constructed from diffusions via subordination. The theoretical estimates are confirmed using numerical examples.

Abstract: Multistate physics modeling (MSPM) of degradation processes is an approach proposed for estimating the failure probability of components and systems. This approach integrates multistate modeling, which describes the degradation process through transitions among discrete states (e.g., initial, microcrack, rupture, etc.), and physics modeling by (physics) equations that describe the degradation process within the states. In reality, the degradation process is non-Markovian, its transition rates are time-dependent, and the degradation is possibly influenced by uncertain external factors such as temperature and stress. Under these conditions, it is in general difficult to derive the state probabilities analytically. In this paper, we overcome this difficulty by building a simulation model supported by a stochastic Petri net representing the multistate degradation process. The proposed modeling approach is applied to the problem of a nuclear component undergoing stress corrosion cracking. The results are compared with those derived from the state-space enrichment Markov chain approximation method applied in a previous work of literature.