5 August 2008
Brian Dennis commented to me at ESA today that he doesn't agree with my categorization of AIC etc.-based methods (what I call "model selection") as not being frequentist. I see what he means, but I think the distinction is a bit subtle. Here's a fuller explanation of his point and my feeling about it:
- AIC is actually an estimate of the expected difference in the "K-L discrepancy" between two models, that is of the difference in the distances between those models and the true model (there are further deep waters here about whether we really need to assume that a "true" model exists at all, but I'm going to skim over them). Therefore, it is an estimate of the change in distance on average, across many hypothetical realizations of the real world — that is, it is really a frequentist point estimate of the change in distance.
- However, when you actually use AIC you don't say anything about probabilities, or think about averages across many realizations — you just say "this model is estimated to be better than this other model". So it definitely doesn't feel like a frequentist method, since you aren't making statements about the fraction of the time that some specified outcome would happen in many repeated trials (i.e., probability, according to the frequentist definition). So I thought it was easier when introducing it not to call it a frequentist method.
- The challenge with all these sorts of things is to give a definition that (1) most people can understand and (2) is technically correct even if it glosses over some details. I think I might have failed on criterion #2 here, I will try to find a better way to describe it that still gets the point across.
(Updated 6 August, BD kindly suggested some changes.)