Moments and Variance#

Notation

Let \(\P\) be a probability function defined over the probability space \(\pspace\).

Let \(X\) be a discrete random variable with \(\S = \lset \xi_1, \xi_2, \ldots \rset\) unless otherwise stated.

Moments#

Definition 22 (Moments)

The \(k\)-th moment of \(X\) is defined as:

\[ \expectation \lsq X^k \rsq = \sum_{x \in X(\S)} x^k \cdot \P \lsq X = x \rsq \]

This definition follows from Property 1 in Expectation.

Essentially, this means that the \(k\)-th moment is the expectation of \(X^k\).

Variance#

Definition 23 (Variance)

The variance of \(X\) is defined as:

(7)#\[ \var \lsq X \rsq = \expectation \lsq \lpar X - \mu \rpar^2 \rsq \]

where \(\mu = \expectation \lsq X \rsq\) is the expectation of \(X\).

We denote \(\sigma^2\) as \(\var\) for short-hand notation.

We also have an equivalent definition of variance, which is more used in practice.

Definition 24 (Variance (Alternative))

The variance of \(X\) is defined as:

(8)#\[ \var \lsq X \rsq = \expectation \lsq X^2 \rsq - \expectation \lsq X \rsq^2 \]

Standard Deviation#

Definition 25 (Standard Deviation)

In the definition of Definition 23, we have \(\var \lsq X \rsq\) to have a different unit than \(X\). If \(X\) is measured in meters, then \(\var \lsq X \rsq\) is measured in meters squared. To solve this issue, we define a new measure called the standard deviation, which is the square root of the variance [Pishro-Nik, 2014].

(9)#\[ \std \lsq X \rsq = \sqrt{\var \lsq X \rsq} \]

Properties of Moments and Variance#

The properties of moments and variance are as follows:

Property 6 (Scaling)

For any constant \(c\), we have:

(10)#\[ \expectation \lsq c \cdot X \rsq = c^k \cdot \expectation \lsq X \rsq \]

where \(k\) is the order of the moment.

Property 7 (DC Shift)

For any constant \(c\), we have:

(11)#\[ \expectation \lsq (X + c) \rsq = \expectation \lsq X \rsq \]

The intuition is that shifting the random variable by a constant does not change the spread of the random variable.

Property 8 (Linearity)

Combining Property 6 and Property 7, we have:

(12)#\[ \expectation \lsq a \cdot X + b \rsq = a^k \cdot \expectation \lsq X \rsq \]

where \(k\) is the order of the moment.

Concept#

Concept

  • Variance is a measure of how spread out a distribution is. More concretely, it is the expectation of the squared deviation from the expectation of the distribution. One can think of for every data point in the distribution, how far is each data point from the expectation (population mean). The variance is the average of these data points (squared to make it positive).

  • Variance is deterministic and is synonymous with Population Variance.

  • Sample Variance is the variance of a random sample from the true population, which is a random variable.

Further Readings#

  • Pishro-Nik, Hossein. “Chapter 3.2.4. Variance.” In Introduction to Probability, Statistics, and Random Processes, 202-206. Kappa Research, 2014.

  • Chan, Stanley H. “Chapter 3.4.4. Momenets and variance.” In Introduction to Probability for Data Science, 133-136. Ann Arbor, Michigan: Michigan Publishing Services, 2021.