Questioning Multilevel Models

TL;DR: It is concluded that more studies need to be done to establish the claimed superiority of restricted versus unrestricted maximum likelihood, to study the effects of shrinkage on the estimators, and to explore the merits of simpler methods such as weighted least squares.

Abstract: In this article, practical problems with multilevel techniques are discussed. These problems, brought to our attention by the National Center for Education Statistics (NCES), have to do with terminology, computer programs employing different algorithms, and interpretations of the coefficients in one or two steps. We discuss the usefulness of the hierarchical linear model (HM) in the most common situation in education-that of a large number of relatively small groups. We also point to situations where the more complicated HMs can be replaced with simpler models, with statistical properties that are easier to study. We conclude that more studies need to be done to establish the claimed superiority of restricted versus unrestricted maximum likelihood, to study the effects of shrinkage on the estimators, and to explore the merits of simpler methods such as weighted least squares. Finally, distinctions must be made between choice of model, choice of technique, choice of algorithm, and choice of computer program. While HMs are an elegant conceptualization, they are not always necessary. Traditional techniques perform as well, or better, if there are large groups and small intraclass correlations, and if the researcher is interested only in the fixedlevel regression coefficients. In this article, we discuss some of the practical problems in using multilevel techniques, by looking into the choices users of these techniques have to make. It is difficult, of course, to define "user." Different users have different degrees of statistical background, computer literacy, experience, and so on. We adopt a particular operational definition of a "user" in this article, which certainly does not cover all users. Our "user" is defined by the set of questions asked by the Statistical Standards and Methodology Division of the National Center for Education Statistics (NCES). These questions were asked in the context of a grant, which has as one of its specific purposes to evaluate the practical usefulness of multilevel modeling in educational statistics. We cannot discuss, let alone answer, all the questions from NCES in this article. Even the ones we discuss will usually require additional statistical and computational research, but they illustrate some of the practical methodological problems in using hierarchical linear models.' The statistics and mathematics will be kept as simple as possible. We shall concentrate on the situation in which we have a relatively large number of relatively small groups. The situation in which we have only two or three groups does not really interest us here, and the situation in which we have