On multiple regression analysis

TL;DR: In this article, the authors present a simple and convenient computational lay-out which can be used for both procedures and apply to a variety of practical examples in order to see what results they lead to and what pitfalls may be encountered.

Abstract: Summary The sums of squares associated with the independent variables in a multiple regression equation depend on the order in which these variables are introduced. Two methods have been proposed in the literature to avoid this inconvenience: “forward selection” or “backward elimination”. With forward selection the independent variables are introduced in successive stages. The order is not predetermined but at each stage that variable is taken as the next one which produces the highest reduction in the residual sum of squares of the dependent variable. With backward elimination on the other hand, we start with the complete regression equation and eliminate the independent variables from it in the order in which they produce the smallest increases in the residual sum of squares. This paper describes a simple and convenient computational lay-out which can be used for both procedures. In forward selection we start with the matrix of product sums, and in bacward elimination we work from the inverse matrix. In addition these techniques are applied to a variety of practical examples in order to see what results they lead to and what pitfalls may be encountered.