Gauss–Newton algorithm

Abstract: A direct method in handling incomplete data in general covariance structural models is investigated. Asymptotic statistical properties of the generalized least squares method are developed. It is shown that this approach has very close relationships with the maximum likelihood approach. Iterative procedures for obtaining the generalized least squares estimates, the maximum likelihood estimates, as well as their standard error estimates are derived. Computer programs for the confirmatory factor analysis model are implemented. A longitudinal type data set is used as an example to illustrate the results.

Abstract: The Gauss-Newton algorithm is often used to minimize a nonlinear least-squares loss function instead of the original Newton-Raphson algorithm. The main reason is the fact that only first-order derivatives are needed to construct the Jacobian matrix. Some applications as, for instance multivariable system identification, give rise to "weighted" nonlinear least-squares problems for which it can become quite hard to obtain an analytical expression of the Jacobian matrix. To overcome that struggle, a pseudo-Jacobian matrix is introduced, which leaves the stationary points untouched and can be calculated analytically. Moreover, by slightly changing the pseudo-Jacobian matrix, a better approximation of the Hessian can be obtained resulting in faster convergence.

Abstract: Kautz and Laguerre filters are effective linear regression models that can describe accurately an unknown linear system with a fewer parameters than finite-impulse response (FIR) filters. This is achieved by expanding the transfer functions of the Kautz and Laguerre filters around some a priori knowledge, concerning the dominating time constants or resonant modes of the system to be identified. When the estimation of these filters is based on a minimization of the least-squares error criterion, the minimization problem becomes separable with respect to the linear coefficients. Therefore, the original unseparated problem can be reduced to a separated problem in only the nonlinear poles, which is numerically better conditioned than the original unseparated one. This paper proposed batch and recursive algorithms that are derived using this separable nonlinear least-squares method, for the estimation of the coefficients and poles of Kautz and Laguerre filters. They have similar computational loads, but better convergence properties than their corresponding algorithms that solve the unseparated problem. The performance of the suggested algorithms is compared to alternative batch and recursive algorithms in some system identification examples. Generally, it is shown that the proposed batch and recursive algorithms have better convergence properties than the alternatives.