The Information Criterion I(g : f) that measures the deviation of a model specified by the probability distribution f from the true distribution g is defined by the formula

$$I(g : f) = E\log g(X) - E\log f(X).$$

Here E denotes the expectation with respect to the true distribution g of X. The criterion is a measure of the deviation of the model f from the true model g, or the best possible model for the handling of the present problem.

The following relation illustrates the significant characteristic of the log likelihood:

$$I(g : {f}_{1}) - I(g : {f}_{2}) = -E(\log {f}_{1}(X) -\log {f}_{2}(X)).$$

This formula shows that for an observation x of X the log likelihood logf(x) provides a relative measure of the closeness of the model f to the truth, or the goodness of the model. This measure is useful even when the true structure g is unknown.

For a model f(X ∕ a) with unknown parameter a the maximum likelihood estimate a(x) is defined as the value of a that maximizes the likelihood f(x ∕ a) for a given observation x. Due to this process the value of logf(x ∕ a(x)) shows an upward bias as an estimate of logf(X ∕ a). Thus to use logf(x ∕ a(x)) as the measure of the goodness of the model f(X ∕ a), it must be corrected for the expected bias.

In typical application of the method of maximum likelihood this expected bias is equal the dimension, or the number of components, of the unknown parameter a. Thus the relative goodness of a model determined by the maximum likelihood estimate is given by

AIC = − 2 (log maximum likelihood − (number of parameters)).

Here log denotes natural logarithm. The coefficient − 2 is used to make the quantity similar to the familiar chi-square statistic in the test of dimensionality of the parameter.

AIC is the abbreviation of An Information Criterion.

About the Author

Professor Akaike died of pneumonia in Tokyo on 4th August 2009, aged 81. He was the Founding Head of the first Department of Statistical Science in Japan. “Now that he has left us forever, the world has lost one of its most innovative statisticians, the Japanese people have lost the finest statistician in their history and many of us a most noble friend” (Professor Howell Tong, from “The Obituary of Professor Hirotugu Akaike.” Journal of the Royal Statistical Society, Series A, March, 2010). Professor Akaike had sent his Encyclopedia entry on May 14 2009, adding the following sentence in his email: “This is all that I could do under the present physical condition.”

Cross References

Akaike’s Information Criterion: Background, Derivation, Properties, and Refinements

Cp Statistic

Kullback-Leibler Divergence

Model Selection