Bar 05 · Inference

Hypothesis testing notes

Testing weighs a record against a claim. This page sets out the null and alternative hypotheses, the two kinds of error, the meaning of significance, and the careful reading of a p-value as a probability statement.

Null and alternative

A test begins with two hypotheses about the law. The null states a precise value or a precise restriction, such as a mean equal to zero or two variances equal. The alternative states what holds when the null fails, and may be one-sided or two-sided. The record is then summarised by a statistic whose sampling distribution is known when the null is true.

The logic of the test is asymmetric. The null is kept unless the record speaks strongly against it. Absence of evidence is not read as evidence of absence; failing to reject the null is not the same as proving it. This asymmetry is deliberate and is the source of much of the misunderstanding that surrounds the subject.

Two kinds of error

Because the record is random, either decision may be wrong. An error of the first kind rejects a true null; an error of the second kind fails to reject a false one. The probability of the first kind is the size of the test and is fixed by the analyst in advance. The probability of the second kind depends on the alternative and on the length of the record; one minus that probability is called the power of the test against that alternative.

The two errors pull in opposite directions. Making the test stricter lowers the chance of a false rejection but raises the chance of a missed effect. The only clean way to reduce both at once is to gather a longer record, which is why the power of a study is a matter of design rather than of arithmetic after the fact.

Significance

A test is conducted at a chosen significance level, usually a small figure such as five parts in a hundred. The statistic is reduced to a p-value, the probability, calculated under the null, of a result at least as extreme as the one observed. When the p-value falls below the level, the null is rejected; otherwise it is retained.

The level is a property of the rule, fixed before the record is seen. Choosing it after the fact, or reporting only the tests that happened to cross it, undermines the whole exercise. A level fixed in advance is what gives the rejection its force.

Reading a p-value

A p-value is the probability of the observed record, or something more extreme, assuming the null is true. It is not the probability that the null is true. It is not the probability that the result arose by chance alone in any loose sense. And it says nothing about the size or the practical importance of the effect under study; a minute difference may be detected with a long enough record.

A small p-value means the record sits uncomfortably under the null. It does not by itself tell us which alternative is correct, and it does not measure the magnitude of the departure.

Several comparisons

When many tests are carried out at once, the chance of at least one false rejection grows quickly with the number of tests. Procedures that control the family-wide error, or that bound the expected proportion of false discoveries, adjust the threshold to account for the multiplicity. Report the count of tests alongside the count of rejections, or the picture will mislead.

Tests and intervals

Testing and interval estimation are two faces of one coin. A two-sided test at a given level rejects exactly those values that fall outside a confidence interval at the matching level. Keeping this correspondence in mind removes a great deal of needless duplication and makes clear that both rest on the same sampling distribution.

The glossary that follows gathers the vocabulary used throughout the reference in one place, with a short reading of each term.