Differential dynamic programming

Abstract: Dynamic programming is a rich area with numerous books already available. Professor Powell’s book on approximate dynamic programming sheds a completely new light on this exciting area. Instead of focusing on traditional approaches to dynamic programming, most of the content is devoted to recent advances in solving large-scale dynamic programs. Professor Powell definitely breaks the longstanding myth of the curse of dimensionality by presenting several state-ofthe-art solution methodologies capable of solving problems of unthinkable size. The book exposes the reader to an excellent mixture of operations research, mathematical programming and artificial intelligence techniques. The transitions and the interplay among these three areas are presented in a coherent way, thus making it an extremely readable book. The spread of topics is also amazing, and yet the book is focused on approximate dynamic programming. Stochastic optimization, machine learning, mathematical programming modeling, finite and infinite time horizon problems, and value function approximations are all nicely blended within a single book. The material is presented at various levels and several different aspects are addressed. The book can easily be used for either an undergraduate or a graduate course textbook. There is abundant technical material to design a comprehensive graduate level course. The book also includes a wide range of exercises. Perhaps the most appealing aspect of Professor Powell’s book is the fact that it spans both theory and practice. On the theoretical side, treatment of convergence is provided for several algorithms and various technical statements are rigorously proved, including aspects of the optimality equation. On the practical side, an important contribution is the modeling framework. Throughout the book several realworld examples are discussed, from transportation to the energy sector and finance. All of them are presented in the same modeling framework. From a practitioner’s perspective, it is perhaps even more important to stress the computational tractability of the presented solution methodologies. Problems deemed intractable a few years ago are now easily solved by using the exhibited techniques in Professor Powell’s book. This clears the way to more comprehensive models and to real-time dynamic decision making. I would strongly recommend the book to any practitioner facing complex, dynamic models involving constantly changing information streams. The first two chapters expose the reader to the basics of dynamic programming. They nicely outline the computational challenges in solving large-scale dynamic programs and several illustrative examples are provided. A more traditional treatment of dynamic programming is provided in the third chapter. The standard value and policy iteration algorithms are the bulk of this chapter. Convergence proofs are also provided. Modern aspects of approximate dynamic programming span the remaining chapters. The important concept of post-decision modeling is introduced in the fourth chapter. This modeling paradigm is the basis for most of the algorithms. The first exposure to reinforcement learning, and thus artificial intelligence, is also provided in this chapter. Through the evolution of dynamic programming, several modeling styles have developed. In the fifth chapter the book discusses a brand new modeling approach, suitable for many complex problems, with intriguing decision making and exogenous information processes. Every reader should bear in mind the importance of this modeling approach. It makes the remaining chapters substantially more readable and easier to understand. The modeling principles were developed by focusing on practical aspects and the ease of modeling the ever evolving real-world situations. Chapter 6 serves as the springboard to the following chapters by covering the stochastic approximation methods. The focus is on the so-called stochastic gradient algorithm. An important parameter of this algorithm is the step size. The book provides a thorough treatment of the step size selection process. In addition to standard step size formulas, a substantial portion of the chapter is devoted to selecting an optimal step size. The chapter is concluded with two convergence results, which are presented in a tutorial fashion, as an introduction to stochastic convergence theory. The two proofs illustrate an older proof technique and a more modern technique. The main concept behind approximate dynamic programming is the notion of value function approximations. Chapter 7 provides a comprehensive treatment of this topic. A very recent development particularly suited for extremely

Abstract: An algorithm for the direct numerical solution of an optimal control problem is given. The method employs cubic polynomials to represent state variables, linearly interpolates control variables, and uses collocation to satisfy the differential equations. This representation transforms the optimal control problem to a mathematical programming problem which is solved by sequential quadratic programming. The method is easy to program for a very general trajectory optimization problem and is shown to be very efficient for several sample problems. Results are compared with solutions obtained with other methods.

Abstract: What do you do to start reading dynamic programming and stochastic control? Searching the book that you love to read first or find an interesting book that will make you want to read? Everybody has difference with their reason of reading a book. Actuary, reading habit must be from earlier. Many people may be love to read, but not a book. It's not fault. Someone will be bored to open the thick book with small words to read. In more, this is the real condition. So do happen probably with this dynamic programming and stochastic control.