Numerical Methods for Unconstrained Optimization

TL;DR: The Hooke and Jeeves method as mentioned in this paper is one such method, suitable for small problems with little programming effort, and it forms the basis for methods of constrained optimization in the next chapter.

Abstract: Unconstrained optimization is the search for the maximum or minimum of a function with no restriction on the values of the variables. At the same time, it forms the basis for methods of constrained optimization in the next chapter. Zero-order methods use only function values, progress made in the previous step pointing the way to the next step. The Hooke and Jeeves method is one such method, suitable for small problems with little programming effort. First-order methods employ the gradient of the function, usually obtained by finite difference, to derive a search direction. This is followed by a line search along this direction for the current maximum or minimum, performed either by progressive reduction of the region in which the maximum or minimum is to be found or by polynomial interpolation. In its simplest form, this is the steepest descent method. However, by the use of gradient data from the previous iteration, an improved search direction can be found, with faster convergence. This is the Fletcher–Reeves method. A more general formulation is based on a quadratic approximation to the objective function, referred to as a second-order method or quasi-Newton method. This involves progressively building up an approximation to the inverse of the Hessian matrix of second derivatives to deduce a search direction. A spreadsheet program for the Hooke and Jeeves method is also used in the next chapter for the penalty function method for constrained optimization.