The advent of formal definitions of the simplicity of a theory has important
implications for model selection. But what is the best way to define
simplicity? Forster and Sober ([1994]) advocate the use of Akaike's Information
Criterion (AIC), a non-Bayesian formalisation of the notion of simplicity. This
forms an important part of their wider attack on Bayesianism in the philosophy
of science. We defend a Bayesian alternative: the simplicity of a theory is to
be characterised in terms of Wallace's Minimum Message Length (MML). We show
that AIC is inadequate for many statistical problems where MML performs well.
Whereas MML is always defined, AIC can be undefined. Whereas MML is not known
ever to be statistically inconsistent, AIC can be. Even when defined and
consistent, AIC performs worse than MML on small sample sizes. MML is
statistically invariant under 1-to-1 re-parametrisation, thus avoiding a common
criticism of Bayesian approaches. We also show that MML provides answers to
many of Forster's objections to Bayesianism. Hence an important part of the
attack on Bayesianism fails.
Additional note (not in the abstract):
At the very end of the paper in sec. 8 (Conclusion) on page 52
(before Appendices and References), there is a question/conjecture
[originally from
Dowe, Baxter, Oliver & Wallace (1998, p93), Edwards & Dowe (1998, sec. 5.3),
Wallace & Dowe (1999a, p282), Wallace & Dowe (2000, p78) and
Comley & Dowe (2005, sec. 11.3.1 p269)]
which asks whether or not only MML
and closely related Bayesian methods can, in general, infer fully-specified
models with both statistical consistency and invariance.