Regression with multiple candidate models: selecting or mixing?

TL;DR: An improved risk bound for ARM is obtained and it is demonstrated that when AIC and BIC are combined, the mixed estimator automatically behaves like the better one, and ARM also performs better than BMA techniques based on BIC approximation.

Abstract: Model combining (mixing) provides an alternative to model selection. An algorithm ARM was recently proposed by the author to combine different regres- sion models/methods. In this work, an improved risk bound for ARM is obtained. In addition to some theoretical observations on the issue of selection versus com- bining, simulations are conducted in the context of linear regression to compare performance of ARM with the familiar model selection criteria AIC and BIC, and also with some Bayesian model averaging (BMA) methods. The simulation suggests the following. Selection can yield a smaller risk when the random error is weak relative to the signal. However, when the random noise level gets higher, ARM produces a better or even much better estimator. That is, mixing appropriately is advantageous when there is a certain degree of uncer- tainty in choosing the best model. In addition, it is demonstrated that when AIC and BIC are combined, the mixed estimator automatically behaves like the better one. A comparison with bagging (Breiman (1996)) suggests that ARM does better than simply stabilizing model selection estimators. In our simulation, ARM also performs better than BMA techniques based on BIC approximation.