Bar 01 · Foundations
Foundations of mathematical statistics
Every honest treatment of probability begins with a measure. This page lays out the axiomatic base from which the rest of the reference is built: the sample space, the family of events, the probability measure, and the random variable that carries outcomes into the number line.
The sample space and its events
An experiment is described by a set Ω whose members are the possible outcomes. A single outcome is written ω. From these outcomes we form events, which are subsets of Ω. Not every subset need be admitted; it is enough to choose a family ℱ that contains Ω itself, is closed under complement, and is closed under countable unions. Such a family is called a sigma-field, and its members are exactly the events to which we shall assign a measure.
The restriction to a sigma-field is not a fussy detail. It prevents the construction of subsets so irregular that no consistent measure can be placed upon them. In ordinary study the requirement rarely binds, because the sigma-field generated by intervals of the line already contains every set one wishes to discuss.
The probability measure
A probability measure is a map from ℱ to the real line satisfying three requirements. First, it is never negative. Second, the measure of the whole space equals one. Third, for any countable collection of pairwise disjoint events, the measure of their union equals the sum of their measures. These are the axioms stated by Kolmogorov, and almost every result in the subject follows from them by plain reasoning.
Immediate consequences include the addition rule for two overlapping events, the rule for complements, and the monotonicity of measure: if one event is contained in another, its measure cannot be larger. The measure of an empty event is zero. None of these facts requires a picture; each is a short deduction.
Random variables
A random variable is a measurable map from the sample space to the real line. Measurability means that the pre-image of every interval is an event, so that probabilities may be attached to statements such as "the value exceeds five". The cumulative distribution function of a random variable is the probability that it does not exceed a given level; this function is non-decreasing and right-continuous, and it rises from zero to one.
Two random variables are independent when the joint probability of any pair of events factorises into the product of their separate probabilities. Independence is a strong condition and should never be assumed by habit; much of the interest of the subject lies in dependence, which the later pages examine through covariance and correlation.
Expectation and variance
The expectation of a random variable is its integral against the probability measure. It is calculated first for variables that take only finitely many values and then extended by a limiting argument. Expectation is linear: the expectation of a sum equals the sum of the expectations, whether or not the variables are independent. This single property carries an enormous amount of the theory.
The variance measures the spread of a variable about its mean; it is the expectation of the squared deviation. Its square root is the standard deviation, written σ. A useful identity expresses the variance as the second moment minus the square of the first. The covariance between two variables measures how they move together, and it reduces to the variance when the two variables coincide.
Conditional probability
Conditional probability restricts attention to a sub-event and rescales the measure so that the total remains one. From the definition of conditional probability one obtains the multiplication rule for joint events and Bayes' rule, which reverses the order of conditioning. Bayes' rule is a theorem, not a philosophy; it follows directly from the symmetry of the joint measure.
The companion page on distributions continues from here, separating the discrete families from the continuous ones and showing how a density and a mass function each describe the same cumulative object.