Bar 03 · Dependence

Covariance and correlation matrices

Dependence among several variables is summarised by a matrix. This page defines the covariance and the correlation, shows why the resulting arrangement is always symmetric and positive semi-definite, and explains how standardising turns covariance into the denser object called correlation.

Covariance

The covariance of two variables is the expectation of the product of their deviations from their means. It is positive when the two tend to rise together, negative when one tends to rise as the other falls, and zero when they carry no linear association. Covariance is symmetric, and the covariance of a variable with itself is its variance.

Covariance is bilinear: it is linear in each argument when the other is held fixed. This property is what allows a whole family of variables to be gathered into a single matrix whose entry in row i and column j is the covariance of the i-th and j-th variables. The diagonal entries are variances, and the symmetry of the covariance carries over to the symmetry of the matrix.

Correlation

The covariance depends on the units in which the variables are measured, which makes it hard to compare across different pairs. Dividing the covariance by the two standard deviations removes the units and yields the correlation coefficient. This coefficient always lies between minus one and one, equal to one when one variable is an increasing linear map of the other and to minus one when the map decreases.

Applying the same division to every entry of a covariance matrix produces a correlation matrix. Its diagonal entries are all one, and its off-diagonal entries are the coefficients just described. The correlation matrix is therefore a standardised version of the covariance matrix, and it is the object that carries the pattern of linear dependence with the units removed.

Figure 1. A correlation arrangement for three variables. The diagonal is one, the off-diagonal cells mirror one another, and every entry lies between minus one and one.

Symmetry and positive semi-definiteness

Not every symmetric arrangement of numbers can be a covariance matrix. The additional requirement is that the quadratic form built from the matrix and any vector of coefficients is never negative. This condition is called positive semi-definiteness. It follows directly from the covariance of a sum, which is the variance of a linear combination of the variables and can never be less than zero.

Positive semi-definiteness has several visible consequences. Every eigenvalue is non-negative. Every principal block is itself positive semi-definite. And the matrix fails to be invertible exactly when some non-trivial linear combination of the variables has zero variance, which happens when the variables are bound by an exact linear relation.

A correlation matrix is positive semi-definite with a unit diagonal. Those two facts are the whole of its algebra, and they rule out a great many patterns that a careless list of coefficients might suggest.

Reading a matrix

A large positive entry signals a strong tendency to move together; a large negative entry signals movement in opposite directions; an entry near zero signals little linear association, though it does not rule out a curved relationship. The arrangement as a whole describes the geometry of the scatter of observations, whose principal directions are the eigenvectors of the matrix and whose extents along those directions are the square roots of the eigenvalues.

The stochastic series pages take these ideas into the time dimension, where the covariance between values at two moments becomes the kernel of the whole analysis.