Doubly Recursive Multivariate Automatic Differentiation

TL;DR: The purpose of this paper is to present the recursive automatic differentiation system, an amazingly elegant extension of the one-variable/one-derivative system that handles essentially any number of variables and derivatives that is recursively defined.

Abstract: system. That's right. The automatic differentiation system never formulates a symbolic expression for the derivative. Automatically calling on something like Mathematica to produce a symbolic derivative, and then plugging in a value for x is the wrong image entirely. Automatic differentiation is something completely different. Well OK, but so what? Symbolic algebra systems are so prevalent and powerful today, why should we be concerned with avoiding symbolic methods? There are two answers. The first is practical. Symbolic generation of derivatives can lead to exponential growth in the length of expressions. That causes computational problems in real applications. Accordingly, there is a practical applied side to the subject of automatic differentiation, as witnessed by the serious attention of computer scientists and numerical analysts [3, 4]. The second answer is more mathematical. It is a relatively easy task to create a single variable automatic differentiation system capable of evaluating first derivatives. In fact, writing in this MAGAZINE in 1986, Rall [10] gives a beautiful presentation of just such a system. What is mathematically interesting is an amazingly elegant extension of the one-variable/one-derivative system that handles essentially any number of variables and derivatives. The extension is recursively defined, employing an induction on both the number of variables and the number of derivatives, and using fundamental definitions that are virtually identical to the ones used in Rall's system. The purpose of this paper is to present the recursive automatic differentiation system. To set the stage, we will begin with a brief review of Rall's one-variable/onederivative system, followed by an example of the recursive system in action. Then the mathematical formulation of the recursive system will be presented. The paper will end with a brief discussion of practical issues related to the recursive system.