Good and Hardin, Common Errors in Statistics (and How to
Avoid Them), p. 3, p. 152

Various algorithms have been developed for aiding in model selection. Many of them are "automatic", in the sense that they have a "stopping rule" (which it might be possible for the researcher to set or change from a default value) based on criteria such as value of a t-statistic or an F-statistic. Others might be better termed "semi-automatic," in the sense that they automatically list various options and values of measures that might be used to help evaluate them.

Caution: Different regression softwares may use the same name (e.g.,"Forward Selection" or "Backward Elimination") to designate different algorithms. Be sure to read the documentation to know find out just what the algorithm does in the software you are using -- in particular, whether it has a stopping rule or is of the "semi-automatic" variety.

Cook and Weisberg

"We do not recommend such
stopping rules for routine use since they can reject perfectly
reasonable submodels from further consideration. Stepwise procedures
are easy to explain, inexpensive to compute, and widely used. The
comparative simplicity of the results from stepwise regression with
model selection rules appeals to many analysts. But, such algorithmic
model selection methods must be used with caution."

They give an example (pp. 280 - 281) illustrating how
stepwise regression algorithms will generally result in models
suggesting that the remaining terms are more important than they really
are, and that the RRyan

Alternatives to Stepwise Selection Methods

- There are various criteria that may be considered in
evaluating models.
One that has intuitive appeal is Mallow's C-statistic. It is an
estimate of Mean Square Error, and can also be regarded as a measure
that accounts for both bias and variance.
^{3}Other aids include Akaike's Information Criterion (AIC) and variations^{4}and Added Variable Plots. - And, of course, context can be important to consider in deciding on a model. For example, the questions of interest can dictate that certain variables need to remain in the model; or quality of data can help decide which of two variables to retain. Several considerations may come into play in deciding on a model.
- Wei Liu has described methods using simultaneous
confidence bands that are useful in some situations for variable
selection or comparing regression models more generally.
^{5} - Also, other regression methods (e.g., Ridge Regression) may be useful instead of Least Squares Regression.
- For more discussion of model selection methods, see Cook and Weisberg (Chapters 10, 11 and 17 - 20); Ryan (Chapters 7, 11, 12 and references therein); Berk (pp. 126 - 135); and P. I. Good and J. W. Hardin (2006). Common Errors in Statistics (And How to Avoid Them), Wiley (Chapters 10, 11, Appendix A).

Notes:

1. R.D. Cook and S. Weisberg (1999), Applied Regression Including Computing and Graphics, Wiley

2. T. Ryan (2009), Modern Regression Methods, Wiley

3. Mallow's statistic is discussed in, e.g., Cook and Weisberg (pp. 272 - 280), Ryan (pp. 273 - 277 and 279 - 283), R. Berk (2004) Regression Analysis: A Constructive Critique, Sage (pp.130 - 135); see also Lecture Notes on Selecting Terms

4. K. P. Burnham and D. R. Anderson (2002), Model selection and Multimodel Inference: A Practical Information-Theoretic Approach, 2nd ed., SpringerK. P. Burnham and D. R. Anderson (2002), Model selection and Multimodel Inference: A Practical Information-Theoretic Approach, 2nd ed., SpringerK. P. Burnham and D. R. Anderson (2002), Model selection and Multimodel Inference: A Practical Information-Theoretic Approach, 2nd ed., Springer has an extensive discussion of Akaike's Information Criterion and related methods. For some common mistakes in using AIC, see pp. 63, 66, 108, 119

5. W. Liu (2011) Simultaneous Inference in Regression, CRC Press. Liu also has Matlab® programs for implementing procedures available from his website. (Click on the link to the book.)

Last updated January 20, 2012