Markov models

TL;DR: In this paper, the authors define stochastic processes as a sequence of non-i.i.d. random variables, and define the joint density using the chain rule.

Abstract: 1 Stochastic processes A stochastic processis an indexed collection of random variables, {Xt}, t ∈ T . If the index setT is discrete, we will often writet ∈ {1, 2, . . .}, to represent discrete time steps. For a finite number of vari ables, we will assume t ∈ 1 : d as usual, where d is the length of the sequence. If the state space X is finite, we will writeXt ∈ {1, 2, . . . , K}, where K is the number of states. If the state space is countably infini te, we will writeXt ∈ {0, 1, 2, . . .}. If the state space is continuous, we will writeXt ∈ IR, althoughXt could also be a vector. Here are some examples of stochastic processes: • A finite sequence of i.i.d. discrete random variables, {X1, X2, . . . , Xn}, whereXt ∈ {1, . . . , K}. This is discrete (finite) time and discrete (finite) state. • An infinite sequence of non i.i.d. random variables {X1, X2, . . .}, Xt ∈ IR, representing, for example, the daily temperature or stock price. This is discrete time but contin uous state. • An infinite sequence of non i.i.d. random variables {X1, X2, . . .}, Xt ∈ {0, 1, 2, . . .}, representing, for example, the number of people in a queue at time t. This is discrete time and discrete state. • Brownian motion, which models a particle performing a Gaussian random walk a long the real line. This is continuous-time and continuous-state. For the rest of this Chapter, we shall restrict attention to d iscrete-time, discrete-state stochastic processes. 2 Markov chains Recall that for any set of random variables X1, . . . , Xd, we can write the joint density using the chain rule as