Gaussian Distribution#

Perhaps the most important distribution is the Gaussian Distribution, also known as the Normal Distribution.

General Gaussian Distribution#

Definition#

Definition 54 (Gaussian Distribution (PDF))

\(X\) is a continuous random variable with a Gaussian distribution if the probability density function is given by:

(22)#\[ \pdf(x) = \frac{1}{\sqrt{2 \pi \sigma^2}} \e \lcurl -\frac{(x - \mu)^2}{2 \sigma^2} \rcurl \]

where \(\mu\) is the mean and \(\sigma^2\) is the variance and are parameters of the distribution.

Some conventions:

  1. We write \(X \sim \gaussian(\mu, \sigma^2)\) to indicate that \(X\) has a Gaussian distribution with mean \(\mu\) and variance \(\sigma^2\).

  2. We sometimes denote \(\gaussian\) as \(\normal\) or \(\gaussiansymbol\).

Definition 55 (Gaussian Distribution (CDF))

There is no closed form for the CDF of the Gaussian distribution. But we will see that it is easy to compute the CDF of the Gaussian distribution using the CDF of the standard normal distribution later.

Therefore, the CDF of the Gaussian distribution is given by:

(23)#\[ \cdf(x) = \int_{-\infty}^x \pdf(x) \, \mathrm{d}x \]

Yes, you have to integrate the PDF to get the CDF but there is no “general” solution.

The PDF of the Gaussian distribution is shown below. The code is referenced and modified from [Shea, 2021].

 1import numpy as np
 2import matplotlib.pyplot as plt
 3import scipy.stats as stats
 4
 5mean_1, mean_2, mean_3, mean_4, mean_5 = -3, -0.3, 0, 1.2, 3
 6sigma = 1
 7
 8x = np.linspace(-10, 10, 1000)
 9
10all_params = [
11    (mean_1, sigma),
12    (mean_2, sigma),
13    (mean_3, sigma),
14    (mean_4, sigma),
15    (mean_5, sigma),
16]
17
18for params in all_params:
19    N = stats.norm(params[0], params[1])
20    label = f"Normal({N.mean()}, $\sigma=${N.std() :.1f})"
21    plt.plot(x, N.pdf(x), label=label)
22
23plt.xlim(-12, 12)
24plt.ylim(0, 0.5)
25plt.legend(prop={'size': 6})
26plt.title("Fixing $\sigma$ = 1, and varying $\mu$")
27plt.show()
28
29mean = 0
30sigma_1, sigma_2, sigma_3, sigma_4, sigma_5 = 0.3, 0.5, 1, 4, 6
31
32x = np.linspace(-10, 10, 1000)
33
34all_params = [
35    (mean, sigma_1),
36    (mean, sigma_2),
37    (mean, sigma_3),
38    (mean, sigma_4),
39    (mean, sigma_5),
40]
41
42for params in all_params:
43    N = stats.norm(params[0], params[1])
44    label = f"Normal({N.mean()}, $\sigma=${N.std() :.1f})"
45    plt.plot(x, N.pdf(x), label=label)
46
47plt.xlim(-12, 12)
48plt.ylim(0, 1.5)
49plt.legend(prop={"size": 6})
50plt.title("Fixing $\mu$ = 0, and varying $\sigma$")
51plt.show()
../_images/31aad40e8a2b99daa511729a8f95a2f932af26b91edc2a47f6a2725254c56952.png ../_images/fc9f985e6e7684b0c32f2876052161ecb3e385055d22aeeac962761849f3b9fa.png

Remark 5 (Gaussian Distribution (PDF))

Note in the plots above, the PDF of the Gaussian distribution is symmetric about the mean \(\mu\), and given a small enough \(\sigma\), the PDF can be greater than 1, as long as the area under the curve is 1.

Note that the PDF curve moves left and right as \(\mu\) increases and decreases, respectively. Similarly, the PDF curve gets narrower and wider as \(\sigma\) increases and decreases, respectively.

Expectation and Variance#

Theorem 15 (Expectation and Variance of the Gaussian Distribution)

The observant reader may notice that the mean and variance of the Gaussian distribution are parameters of the distribution. However, this is not proven previously.

If \(X\) is a continuous random variable with an gaussian distribution with mean \(\mu\) and variance \(\sigma^2\), then the expectation and variance of \(X\) are given by:

(24)#\[ \expectation \lsq X \rsq = \mu \qquad \text{and} \qquad \var \lsq X \rsq = \sigma^2 \]

Linear Transformation of Gaussian Distribution#

Theorem 16 (Linear Transformation of Gaussian Distribution)

Let \(X \sim \gaussiansymbol(\mu, \sigma^2)\) be a random variable with a Gaussian distribution with mean \(\mu\) and variance \(\sigma^2\).

Let \(Y = aX + b\) be a linear transformation of \(X\).

Then, \(Y \sim \gaussiansymbol(a\mu + b, a^2\sigma^2)\) is a Gaussian distribution with mean \(a\mu + b\) and variance \(a^2\sigma^2\).

In other words, \(Y\) experiences a shift and scale change in coordinates, and while the distribution may not longer be the same as \(X\), it still belongs to the same family.

Standard Gaussian Distribution#

Motivation#

In continuous distributions, we are often asked to evaluate the probability of an interval \([a, b]\). For example, if \(X \sim \gaussian(\mu, \sigma^2)\), what is the probability that \(X\) is between \(a\) and \(b\)? If we use PDF to compute the probability, we will have to integrate the PDF over the interval \([a, b]\).

\[ \P \lsq a \leq X \leq b \rsq = \int_a^b \pdf(x) \, \mathrm{d}x \]

We have seen that there is no closed form solution for this integral.

Furthermore, in practice, we use the CDF to compute the probability of an interval.

\[ \P \lsq a \leq X \leq b \rsq = \cdf(b) - \cdf(a) \]

However, we have also seen that there is no closed form solution for the CDF of the Gaussian distribution since the CDF is an integral of the PDF.

This prompted the existence of the standard normal distribution.

Definition#

Definition 56 (Standard Gaussian Distribution)

Let \(X\) be a continuous random variable with a Gaussian distribution with mean \(\mu\) and variance \(\sigma^2\). Then \(X\) is said to be a Standard Gaussian if it has a Gaussian distribution with mean 0 and variance 1.

(25)#\[ \pdf(x) = \frac{1}{\sqrt{2 \pi}} \exp \lcurl -\frac{x^2}{2} \rcurl \]

In notations, it is \(X \sim \gaussiansymbol(0, 1)\), with \(\mu = 0\) and \(\sigma^2 = 1\).

Definition 57 (Standard Normal Distribution (CDF))

The CDF of the Standard Gaussian is given by:

\[ \Phi(x) \defeq \cdf(x) = \frac{1}{\sqrt{2 \pi}} \int_{-\infty}^x \exp \lcurl -\frac{t^2}{2} \rcurl \, \mathrm{d}t \]

Take note of the \(\Phi\) notation, which is the standard notation for the CDF of the Standard Gaussian.

 1import sys
 2from pathlib import Path
 3parent_dir = str(Path().resolve().parent)
 4sys.path.append(parent_dir)
 5
 6import numpy as np
 7import scipy.stats as stats
 8
 9from utils import seed_all, plot_continuous_pdf_and_cdf
10seed_all()
11
12# x = np.linspace(0, 10, 5000)
13mean, sigma = 0, 1
14X = stats.norm(mean, sigma)
15
16plot_continuous_pdf_and_cdf(X, -5, 5, title="Normal Distribution $\mu=0, \sigma=1$", figsize=(15, 5))

Using Seed Number 1992
../_images/00ed8fde2e7015c87aed6113eb4a397d7b2b7f2665301081f1f5985bcb0131e1.png

Error Function#

Definition 58 (Error Function)

The error function is defined as:

\[ \text{erf}(x) \defeq \frac{2}{\sqrt{\pi}} \int_0^x \exp \lcurl -t^2 \rcurl \, \mathrm{d}t \]

The connection of the error function to the CDF of the Standard Gaussian is given by:

(26)#\[ \Phi(x) = \frac{1}{2} \left( 1 + \text{erf} \lcurl \frac{x}{\sqrt{2}} \rcurl \right) \qquad \text{and} \qquad \text{erf}(x) = 2 \Phi(x\sqrt{2}) - 1 \]

CDF of Arbitrary Gaussian Distribution#

Theorem 17 (CDF of Arbitrary Gaussian Distribution)

Let \(X\) be a continuous random variable with a Gaussian distribution with mean \(\mu\) and variance \(\sigma^2\). Then we can convert the CDF of \(X\) to the CDF of the Standard Gaussian by:

(27)#\[ \cdf(x) = \Phi \lpar \frac{x - \mu}{\sigma} \rpar \]

Proof. CDF of Arbitrary Gaussian Distribution The definition of a Gaussian distribution is given by:

\[\begin{split} \begin{aligned} \cdf(x) &= \P \lsq X \leq x \rsq \\ &= \int_{-\infty}^x \pdf(t) \, \mathrm{d}t \\ &= \int_{-\infty}^x \frac{1}{\sqrt{2 \pi \sigma^2}} \exp \lcurl -\frac{(t - \mu)^2}{2 \sigma^2} \rcurl \, \mathrm{d}t \\ \end{aligned} \end{split}\]

We can use the (integration by) substitution \(u = \frac{t - \mu}{\sigma}\) to get [Chan, 2021]:

\[\begin{split} \begin{aligned} \int_{-\infty}^x \frac{1}{\sqrt{2 \pi \sigma^2}} \exp \lcurl -\frac{(t - \mu)^2}{2 \sigma^2} \rcurl \, \mathrm{d}t &= \int_{-\infty}^{\frac{x - \mu}{\sigma}} \frac{1}{\sqrt{2 \pi}} \exp \lcurl -\frac{u^2}{2} \rcurl \, \mathrm{d}u \\ &= \Phi \lpar \frac{x - \mu}{\sigma} \rpar \end{aligned} \end{split}\]

Corollary 8 (Probability of an Interval)

As a consequence of Theorem 17, we can compute the probability of an interval for an arbitrary Gaussian distribution. This solves our problem of computing the probability of an interval for a Gaussian Distribution earlier.

(28)#\[\begin{split} \begin{aligned} \P \lsq a \leq X \leq b \rsq &= \cdf(b) - \cdf(a) \\ &= \P \lsq X \leq b \rsq - \P \lsq X \leq a \rsq \\ &= \Phi \lpar \frac{b - \mu}{\sigma} \rpar - \Phi \lpar \frac{a - \mu}{\sigma} \rpar \end{aligned} \end{split}\]

Corollary 9 (Corollary of Standard Gaussian)

As a consequence of Theorem 17, we have the following corollaries [Chan, 2021]:

  • \(\Phi(y) = 1 - \Phi(-y)\)

  • \(\P \lsq X \geq b \rsq = 1 - \Phi \lpar \frac{b - \mu}{\sigma} \rpar\)

  • \(\P \lvert X \rvert \geq b = 1 - \Phi \lpar \frac{b - \mu}{\sigma} \rpar + \Phi \lpar \frac{-b - \mu}{\sigma} \rpar\)

Further Readings#