Reactive programming

Abstract: From the Publisher: This is the first textbook that fully explains the neuro-dynamic programming/reinforcement learning methodology, which is a recent breakthrough in the practical application of neural networks and dynamic programming to complex problems of planning, optimal decision making, and intelligent control.

Abstract: Conventional programming languages are growing ever more enormous, but not stronger. Inherent defects at the most basic level cause them to be both fat and weak: their primitive word-at-a-time style of programming inherited from their common ancestor—the von Neumann computer, their close coupling of semantics to state transitions, their division of programming into a world of expressions and a world of statements, their inability to effectively use powerful combining forms for building new programs from existing ones, and their lack of useful mathematical properties for reasoning about programs.An alternative functional style of programming is founded on the use of combining forms for creating programs. Functional programs deal with structured data, are often nonrepetitive and nonrecursive, are hierarchically constructed, do not name their arguments, and do not require the complex machinery of procedure declarations to become generally applicable. Combining forms can use high level programs to build still higher level ones in a style not possible in conventional languages.Associated with the functional style of programming is an algebra of programs whose variables range over programs and whose operations are combining forms. This algebra can be used to transform programs and to solve equations whose “unknowns” are programs in much the same way one transforms equations in high school algebra. These transformations are given by algebraic laws and are carried out in the same language in which programs are written. Combining forms are chosen not only for their programming power but also for the power of their associated algebraic laws. General theorems of the algebra give the detailed behavior and termination conditions for large classes of programs.A new class of computing systems uses the functional programming style both in its programming language and in its state transition rules. Unlike von Neumann languages, these systems have semantics loosely coupled to states—only one state transition occurs per major computation.

Abstract: Dynamic programming is a rich area with numerous books already available. Professor Powell’s book on approximate dynamic programming sheds a completely new light on this exciting area. Instead of focusing on traditional approaches to dynamic programming, most of the content is devoted to recent advances in solving large-scale dynamic programs. Professor Powell definitely breaks the longstanding myth of the curse of dimensionality by presenting several state-ofthe-art solution methodologies capable of solving problems of unthinkable size. The book exposes the reader to an excellent mixture of operations research, mathematical programming and artificial intelligence techniques. The transitions and the interplay among these three areas are presented in a coherent way, thus making it an extremely readable book. The spread of topics is also amazing, and yet the book is focused on approximate dynamic programming. Stochastic optimization, machine learning, mathematical programming modeling, finite and infinite time horizon problems, and value function approximations are all nicely blended within a single book. The material is presented at various levels and several different aspects are addressed. The book can easily be used for either an undergraduate or a graduate course textbook. There is abundant technical material to design a comprehensive graduate level course. The book also includes a wide range of exercises. Perhaps the most appealing aspect of Professor Powell’s book is the fact that it spans both theory and practice. On the theoretical side, treatment of convergence is provided for several algorithms and various technical statements are rigorously proved, including aspects of the optimality equation. On the practical side, an important contribution is the modeling framework. Throughout the book several realworld examples are discussed, from transportation to the energy sector and finance. All of them are presented in the same modeling framework. From a practitioner’s perspective, it is perhaps even more important to stress the computational tractability of the presented solution methodologies. Problems deemed intractable a few years ago are now easily solved by using the exhibited techniques in Professor Powell’s book. This clears the way to more comprehensive models and to real-time dynamic decision making. I would strongly recommend the book to any practitioner facing complex, dynamic models involving constantly changing information streams. The first two chapters expose the reader to the basics of dynamic programming. They nicely outline the computational challenges in solving large-scale dynamic programs and several illustrative examples are provided. A more traditional treatment of dynamic programming is provided in the third chapter. The standard value and policy iteration algorithms are the bulk of this chapter. Convergence proofs are also provided. Modern aspects of approximate dynamic programming span the remaining chapters. The important concept of post-decision modeling is introduced in the fourth chapter. This modeling paradigm is the basis for most of the algorithms. The first exposure to reinforcement learning, and thus artificial intelligence, is also provided in this chapter. Through the evolution of dynamic programming, several modeling styles have developed. In the fifth chapter the book discusses a brand new modeling approach, suitable for many complex problems, with intriguing decision making and exogenous information processes. Every reader should bear in mind the importance of this modeling approach. It makes the remaining chapters substantially more readable and easier to understand. The modeling principles were developed by focusing on practical aspects and the ease of modeling the ever evolving real-world situations. Chapter 6 serves as the springboard to the following chapters by covering the stochastic approximation methods. The focus is on the so-called stochastic gradient algorithm. An important parameter of this algorithm is the step size. The book provides a thorough treatment of the step size selection process. In addition to standard step size formulas, a substantial portion of the chapter is devoted to selecting an optimal step size. The chapter is concluded with two convergence results, which are presented in a tutorial fashion, as an introduction to stochastic convergence theory. The two proofs illustrate an older proof technique and a more modern technique. The main concept behind approximate dynamic programming is the notion of value function approximations. Chapter 7 provides a comprehensive treatment of this topic. A very recent development particularly suited for extremely

Abstract: (2007). Nonlinear Programming Theory and Algorithms. Technometrics: Vol. 49, No. 1, pp. 105-105.