Regression with multiple candidate models: Selecting or mixing?

Research output: Contribution to journalArticlepeer-review

71 Scopus citations

Abstract

Model combining (mixing) provides an alternative to model selection. An algorithm ARM was recently proposed by the author to combine different regression models/methods. In this work, an improved risk bound for ARM is obtained. In addition to some theoretical observations on the issue of selection versus combining, simulations are conducted in the context of linear regression to compare performance of ARM with the familiar model selection criteria AIC and BIC, and also with some Bayesian model averaging (BMA) methods. The simulation suggests the following. Selection can yield a smaller risk when the random error is weak relative to the signal. However, when the random noise level gets higher, ARM produces a better or even much better estimator. That is, mixing appropriately is advantageous when there is a certain degree of uncertainty in choosing the best model. In addition, it is demonstrated that when AIC and BIC are combined, the mixed estimator automatically behaves like the better one. A comparison with bagging (Breiman (1996)) suggests that ARM does better than simply stabilizing model selection estimators. In our simulation, ARM also performs better than BMA techniques based on BIC approximation.

Original languageEnglish (US)
Pages (from-to)783-809
Number of pages27
JournalStatistica Sinica
Volume13
Issue number3
StatePublished - Jul 2003

Keywords

  • ARM
  • Combining procedures
  • Model averaging
  • Model selection

Fingerprint Dive into the research topics of 'Regression with multiple candidate models: Selecting or mixing?'. Together they form a unique fingerprint.

Cite this