06. Gaussian Distribution — The Distribution of Universality

Distribution 1/3. Understand the Gaussian (normal) distribution not as a "bell-shaped curve," but as the unique answer reached by three independent derivations. Background: probability spaces, moments and characteristic functions, and the CLT.

Definition

The probability density function of $X \sim \mathcal{N}(\mu, \sigma^2)$ is

$$f(x) = \frac{1}{\sigma\sqrt{2\pi}} \exp\Big( -\frac{(x-\mu)^2}{2\sigma^2} \Big), \qquad x \in \mathbb{R}$$

• $\mu$: location (mean, median, and mode—the three coincide because of perfect symmetry)
• $\sigma^2$: scale (variance)
• The normalization constant $\sigma\sqrt{2\pi}$ comes from the Gaussian integral $\int e^{-u^2} du = \sqrt{\pi}$, or equivalently $\Gamma(1/2) = \sqrt{\pi}$.

Standardization: $Z = (X - \mu)/\sigma \sim \mathcal{N}(0,1)$. Every Gaussian is a location-scale transformation of the standard normal distribution, so one table (or one function $\Phi$) handles the entire family.

$$P(X \le x) = \Phi\Big( \frac{x-\mu}{\sigma} \Big), \qquad \Phi(z) = \frac{1}{\sqrt{2\pi}} \int_{-\infty}^{z} e^{-t^2/2}\, dt$$

$\Phi$ cannot be expressed using elementary functions—even the Gaussian has no closed-form CDF, which makes the alpha-stable distribution's "lack of a closed-form density" seem less unusual.

Why this particular shape? Three derivations

Route 1: central limit theorem (limit of sums)

As the CLT states, standardized sums of finite-variance i.i.d. factors converge to $\mathcal{N}(0,1)$ regardless of the original distribution. This is why quantities generated as "the sum of countless small independent factors," such as measurement error, biological signals, and aggregate statistics, are close to Gaussian.

Route 2: maximum entropy (minimum assumptions)

Suppose only two facts, the mean $\mu$ and variance $\sigma^2$, are known and nothing else. Under those constraints, the distribution that maximizes the differential entropy

$$h(f) = -\int f(x) \ln f(x)\, dx$$

is exactly $\mathcal{N}(\mu, \sigma^2)$ (derived with Lagrange multipliers). In other words, the Gaussian assumes the least additional structure beyond the given mean and variance. This is the information-theoretic justification for the common practice of modeling noise as Gaussian.

Route 3: stability (closure under sums)

The sum of independent Gaussians is again Gaussian:

$$\mathcal{N}(\mu_1, \sigma_1^2) + \mathcal{N}(\mu_2, \sigma_2^2) = \mathcal{N}(\mu_1+\mu_2,\ \sigma_1^2+\sigma_2^2)$$

It satisfies the definition of stability, and the Gaussian is the only stable distribution with finite variance. It is the $\alpha = 2$ vertex of the alpha-stable family.

Core properties

Characteristic function and MGF

$$\varphi(t) = \exp\Big( i\mu t - \frac{\sigma^2 t^2}{2} \Big), \qquad M(t) = \exp\Big( \mu t + \frac{\sigma^2 t^2}{2} \Big)$$

Moments of every order exist, and the central moments are

$$\mathbb{E}[(X-\mu)^{2k}] = \sigma^{2k} (2k-1)!!, \qquad \mathbb{E}[(X-\mu)^{2k+1}] = 0$$

Skewness 0 and kurtosis 3—the reference value of 3 for kurtosis comes from the Gaussian.

Closure under linear transformations and combinations

$aX + b \sim \mathcal{N}(a\mu + b, a^2\sigma^2)$, and any linear combination of independent Gaussians is also Gaussian. Thanks to this closure, every intermediate calculation in linear models, Kalman filters, and Gaussian processes stays within the family. "Start Gaussian, end Gaussian" is the rule of the linear world.

The 68–95–99.7 rule

$$P(\lvert X - \mu \rvert \le \sigma) \approx 0.6827, \quad P(\lvert X - \mu \rvert \le 2\sigma) \approx 0.9545, \quad P(\lvert X - \mu \rvert \le 3\sigma) \approx 0.9973$$

The tail decays extremely quickly as $e^{-x^2/2}$. The probability outside $6\sigma$ is about $2 \times 10^{-9}$. This rapid decay is both an advantage (concentration inequalities) and a trap: when extreme values in real data occur far more often than a Gaussian predicts (as with financial returns), a Gaussian model systematically underestimates risk. The alpha-stable distribution is an alternative.

Independence coincides with uncorrelatedness (joint Gaussians only)

In general, uncorrelatedness ($\mathrm{Cov} = 0$) is weaker than independence, but the two concepts coincide for jointly Gaussian vectors. This greatly simplifies multivariate analysis.

Multivariate Gaussian (briefly)

$\mathbf{X} \sim \mathcal{N}(\boldsymbol{\mu}, \Sigma)$, where $\Sigma$ is a positive-definite covariance matrix:

$$f(\mathbf{x}) = \frac{1}{(2\pi)^{d/2} \lvert \Sigma \rvert^{1/2}} \exp\Big( -\frac{1}{2} (\mathbf{x}-\boldsymbol{\mu})^{\top} \Sigma^{-1} (\mathbf{x}-\boldsymbol{\mu}) \Big)$$

• Contours are ellipses (or ellipsoids): the eigenvectors of $\Sigma$ give the axes, and its eigenvalues give the squared axis lengths.
• Marginal distributions, conditional distributions, and linear maps are all Gaussian—these three statements are the entire mathematical foundation of Kalman filters and Gaussian-process regression.
• The conditional mean is linear: $\mathbb{E}[X_1 \mid X_2 = x_2] = \mu_1 + \Sigma_{12}\Sigma_{22}^{-1}(x_2 - \mu_2)$—this is why linear regression is "the exact answer in the Gaussian world."

Sampling

From uniform random variables $U_1, U_2 \sim \mathrm{Uniform}(0,1)$ (Box–Muller):

$$Z_1 = \sqrt{-2 \ln U_1}\, \cos(2\pi U_2), \qquad Z_2 = \sqrt{-2 \ln U_1}\, \sin(2\pi U_2)$$

$Z_1, Z_2$ are independent standard normal variables. This uses the fact that the squared radius in polar coordinates follows an exponential distribution; it is the algorithmic version of the classic trick of squaring the Gaussian integral and solving it in polar coordinates. Practical libraries mainly use improved variants (the polar method and Ziggurat).

Summary of roles in statistics and ML

Area	What the Gaussian does
Least squares	MLE = OLS under a Gaussian noise assumption
Bayesian inference	Self-conjugate for the mean parameter
Kalman filter	Posterior remains closed form in a linear-Gaussian state space
Gaussian process	Prior over function space
VAE / diffusion model	Standard choice for latent spaces and noise schedules (reparameterization: $z = \mu + \sigma \epsilon$)
Statistical tests	Supplies approximate distributions of test statistics through the CLT

Limitations — when to question a Gaussian

1. Heavy tails: if extreme values occur more often than $e^{-x^2}$ predicts (log returns and latency), a Gaussian is unsuitable. → alpha-stable, Student-t
2. Bounded support: quantities confined to $[0,1]$, such as proportions and probabilities. → Beta
3. Asymmetry: clearly skewed data. → log-normal, Gamma, skew-normal
4. Multimodality: a single Gaussian cannot represent several peaks. → mixture models (GMM)

What to remember from this page

1. The Gaussian is the unique answer reached by three routes: the CLT (limit of sums), maximum entropy (minimum assumptions), and stability (closure under sums).
2. Closure under linear operations keeps every calculation in the linear world (regression, Kalman filters, and GPs) in closed form.
3. Its tail decays as $e^{-x^2/2}$—it can systematically underestimate extreme values.
4. Kurtosis 3, $\alpha = 2$: the Gaussian is a special vertex of larger families (the exponential and stable families).

Next step

• Beta Distribution — from infinite support to $[0,1]$
• Alpha-Stable Distributions — the world beyond $\alpha = 2$