Bar 05 · Inference

Estimation and inference

Inference turns a record into a statement about the law that produced it. This page introduces estimators, examines bias and variance, defines consistency, sketches maximum likelihood, and explains the meaning of a confidence interval.

Estimators

An estimator is a rule that maps an observed record to a number or a vector intended to approximate an unknown feature of the law. A sample mean estimates a population mean; a sample variance estimates a variance; a fitted slope estimates a coefficient of dependence. Because the record is random, the estimator is itself a random variable, and its own law is called its sampling distribution.

The sampling distribution is the heart of the subject. Once it is known, questions about accuracy become ordinary probability questions. Two summaries of that law are especially useful: its centre, which measures whether the estimator aims at the right target, and its spread, which measures how tightly it is likely to land near that target.

Bias and variance

An estimator is unbiased when its expectation equals the feature it is meant to estimate. Bias is the difference between the two. Unbiasedness is a virtue but not the only one, because a slightly biased estimator with a small spread may beat an unbiased one with a large spread. The balance between these two quantities is summarised by the mean squared error, which equals the variance plus the square of the bias.

Sample variance is usually divided by one less than the count of observations, rather than by the count itself, precisely so that the result is unbiased. This small correction is one of the earliest lessons in the subject and a good illustration of how the sampling distribution guides the choice of a rule.

Consistency

An estimator is consistent when it converges in probability to the feature it estimates as the record grows. Consistency is a large-sample property and says nothing about any single record, but it is the weakest guarantee one should demand. Many consistent estimators exist for the same feature, and the choice among them turns on the shape of their sampling distributions at the lengths actually available.

Maximum likelihood

The method of maximum likelihood chooses the value of the unknown feature that makes the observed record most probable. Under mild regularity conditions the resulting estimator is consistent, is asymptotically normal, and attains the smallest possible spread among a broad class of rules. The method is attractive because it applies uniformly across families and because its large-sample law is known in general terms.

Confidence intervals

A confidence interval is a random range constructed so that, across repeated records, it covers the unknown feature with a stated long-run frequency. The frequency refers to the rule, not to any single interval. A ninety per cent interval is one whose covering rule succeeds ninety times in a hundred, and this reading should be kept firmly in mind whenever an interval is quoted.

A special case of inference is the test of a claim, which is the subject of the next page.