04. Law of Large Numbers, Central Limit Theorem, and Stability — Universal Laws of Sums

Background 4/5. The law of large numbers (LLN), central limit theorem (CLT), and stable distributions emerge in sequence from one question: "what shape does the sum of many independent factors take?" This page covers why Gaussians appear everywhere—and why the Gaussian is not the only answer.

Problem setup

Let $X_1, X_2, \dots$ be independent and identically distributed (i.i.d.) random variables, and define the partial sum $S_n = X_1 + \cdots + X_n$. The subject is the behavior of $S_n$ as $n$ grows.

Law of large numbers (LLN)

If the mean $\mu = \mathbb{E}[X_i]$ exists, the sample mean converges to the true mean.

$$\frac{S_n}{n} \xrightarrow{\ \text{a.s.}\ } \mu \qquad (n \to \infty)$$

• The weak LLN gives convergence in probability, while the strong LLN (Kolmogorov) gives almost-sure convergence.
• If the variance is finite, Chebyshev's inequality gives a three-line proof:

$$P\Big( \Big\lvert \frac{S_n}{n} - \mu \Big\rvert \ge \varepsilon \Big) \le \frac{\mathrm{Var}(X_1)}{n \varepsilon^2} \to 0$$

• Notice the precondition: $\mathbb{E}[\lvert X_1 \rvert]$ must be finite. The Cauchy distribution violates this condition, and in fact the mean $S_n / n$ of Cauchy samples again follows a Cauchy distribution—nothing improves regardless of how large $n$ becomes. The belief that "collecting more data stabilizes the mean" stands on a distributional assumption.

Central limit theorem (CLT)

The LLN only says that $S_n/n$ collapses to a constant; it says nothing about the fluctuations around it. The CLT magnifies and examines those fluctuations.

$$\frac{S_n - n\mu}{\sigma\sqrt{n}} \xrightarrow{\ d\ } \mathcal{N}(0, 1) \qquad (\sigma^2 = \mathrm{Var}(X_1) \lt \infty)$$

Whatever the original distribution of $X_i$—coin tosses, a uniform distribution, or a skewed discrete distribution—the standardized sum approaches the same limit. This universality is why Gaussians appear throughout nature and engineering.

Proof sketch (characteristic functions)

With characteristic functions, the skeleton of the proof takes only a few lines. Standardize with $Y_i = (X_i - \mu)/\sigma$. Since $\mathbb{E}[Y_i] = 0$ and $\mathbb{E}[Y_i^2] = 1$, near the origin

$$\varphi_Y(t) = 1 - \frac{t^2}{2} + o(t^2)$$

The characteristic function of an independent sum is a product, so

$$\varphi_{S_n^*}(t) = \Big[ \varphi_Y\Big(\frac{t}{\sqrt{n}}\Big) \Big]^n = \Big[ 1 - \frac{t^2}{2n} + o\Big(\frac{1}{n}\Big) \Big]^n \longrightarrow e^{-t^2/2}$$

The limit $e^{-t^2/2}$ is the characteristic function of the standard normal distribution, and Lévy's continuity theorem guarantees convergence in distribution. The crucial point is that the existence of the second moment enabled the second-order term of the Taylor expansion.

Rate of convergence — Berry–Esseen

If the third moment exists, the convergence rate is $O(1/\sqrt{n})$:

$$\sup_x \big\lvert F_n(x) - \Phi(x) \big\rvert \le \frac{C\,\mathbb{E}[\lvert Y_1 \rvert^3]}{\sqrt{n}}$$

Practical lesson: there is no magic number such as $n = 30$. The more asymmetric or heavy-tailed the original distribution, the more slowly the Gaussian approximation improves, and tail probabilities in particular converge far later than the center.

Stability — when the CLT's assumptions break

What happens if the CLT condition $\sigma^2 \lt \infty$ fails? To answer, first reframe the question.

Definition of a stable distribution

A distribution is stable if, for independent copies $X_1, X_2$ and arbitrary $a, b > 0$, there exist suitable $c > 0$ and $d$ such that

$$aX_1 + bX_2 \overset{d}{=} cX + d$$

In other words, taking independent sums does not change the shape of the distribution after adjusting scale and location. The Gaussian is a representative example:

$$\mathcal{N}(0,a^2) + \mathcal{N}(0,b^2) = \mathcal{N}(0, a^2+b^2)$$

Why every limit must be a stable distribution

Suppose $\frac{S_n - b_n}{a_n}$ converges to some distribution $Z$. Since $S_{2n}$ is the sum of two independent blocks of size $n$, in the limit $Z$ must have the same distribution as a normalized sum of two independent copies of itself. Pushing this argument through yields:

Only stable distributions can arise as limits of normalized i.i.d. sums.

The Gaussian is merely one point in this family selected by the privileged condition of "finite variance." The entire stable family is parameterized by an exponent $\alpha \in (0, 2]$, with $\alpha = 2$ giving the Gaussian.

Generalized central limit theorem

If the tails of $X_i$ decay polynomially—$P(\lvert X_1 \rvert > x) \sim C x^{-\alpha}$, $0 \lt \alpha \lt 2$—the classical CLT does not apply because the variance is infinite. Instead (Gnedenko–Kolmogorov),

$$\frac{S_n - b_n}{n^{1/\alpha} L(n)} \xrightarrow{\ d\ } \text{(alpha-stable distribution with exponent } \alpha\text{)}$$

• The scaling is $n^{1/\alpha}$ rather than $\sqrt{n} = n^{1/2}$. The sum grows much faster because a few enormous terms dominate it.
• The tail exponent $\alpha$ of the original distribution determines the limiting distribution. This is described by saying that it "belongs to the domain of attraction of an $\alpha$-stable distribution."
• The classical CLT is subsumed as the special case $\alpha = 2$.

Intuition: what dominates the sum?

When the variance is finite, the contribution of each term to the sum $S_n$ is evenly spread and no single term dominates—the result is a smooth Gaussian. When the tail is thick, decaying as $x^{-\alpha}$ ($\alpha \lt 2$), a single maximum has the same order of magnitude as the entire sum—the result is a jump-dominated alpha-stable distribution. The boundary between the two regimes of "accumulating average factors" and "domination by extremes" is $\alpha = 2$.

What to remember from this page

1. The LLN and CLT are theorems resting on assumptions about moment existence. Distributions that violate those assumptions really exist (Cauchy).
2. The CLT proof takes three lines with characteristic functions: independent sum = product of characteristic functions, second-order Taylor expansion, and $e^{-t^2/2}$.
3. Stable distributions are all the possible limits of i.i.d. sums, and the Gaussian is the $\alpha = 2$ case.
4. Sums of data with tail exponent $\alpha \lt 2$ scale as $n^{1/\alpha}$ and converge to an alpha-stable distribution.

Next step

• Bayesian Inference and Conjugate Priors — from frequentist limit theorems to Bayesian updating
• Gaussian Distribution — the protagonist of the universality proved today
• Alpha-Stable Distributions — a full exploration of the $\alpha \lt 2$ world