02. Expectation, Moments, and Characteristic Functions — Numerical Summaries of Distributions

Background 2/5. This page organizes the tools for summarizing the infinite-dimensional object called a distribution with finitely many numbers (moments) and one function (the characteristic function). The characteristic function is the only practical means of defining alpha-stable distributions, so it receives particular attention.

Expectation

The expectation of a random variable $X$ is an integral with respect to a probability measure.

$$\mathbb{E}[X] = \int_{\Omega} X(\omega)\, dP(\omega) = \int_{-\infty}^{\infty} x \, d\mu_X(x)$$

In practical terms:

• Discrete: $\mathbb{E}[X] = \sum_x x \, p_X(x)$
• Continuous: $\mathbb{E}[X] = \int_{-\infty}^{\infty} x f_X(x)\, dx$

LOTUS (law of the unconscious statistician)

For a function $g$, the following holds without having to derive the distribution of $g(X)$ anew.

$$\mathbb{E}[g(X)] = \int_{-\infty}^{\infty} g(x) f_X(x)\, dx$$

Properties of expectation

• Linearity: $\mathbb{E}[aX + bY] = a\,\mathbb{E}[X] + b\,\mathbb{E}[Y]$ — this holds even without independence.
• Product under independence: if $X \perp Y$, then $\mathbb{E}[XY] = \mathbb{E}[X]\,\mathbb{E}[Y]$
• Existence condition: saying that $\mathbb{E}[X]$ "exists" means that $\mathbb{E}[\lvert X \rvert]$ is finite. Not every distribution has an expectation.

Distributions without expectations are not pathological exceptions

The Cauchy distribution $f(x) = \frac{1}{\pi(1+x^2)}$ has no defined mean because $\int \lvert x \rvert f(x)\, dx = \infty$. The Cauchy distribution is the $\alpha = 1$ special case of an alpha-stable distribution, and it arises naturally when modeling the heavy tails of real data such as financial returns and network latency. The intuition that "we can just take the sample mean" collapses for these distributions—the sample mean does not converge regardless of how many samples are collected.

Variance and higher moments

The $k$th moment is $\mathbb{E}[X^k]$, and the $k$th central moment is $\mathbb{E}[(X-\mu)^k]$ ($\mu = \mathbb{E}[X]$).

Variance (second central moment)

$$\mathrm{Var}(X) = \mathbb{E}[(X-\mu)^2] = \mathbb{E}[X^2] - \mu^2$$

• $\mathrm{Var}(aX + b) = a^2 \mathrm{Var}(X)$
• Under independence, $\mathrm{Var}(X+Y) = \mathrm{Var}(X) + \mathrm{Var}(Y)$ — this is why the $\sqrt{n}$ scaling appears in the CLT.

Skewness and kurtosis (third and fourth standardized moments)

$$\text{skewness} = \mathbb{E}\Big[\Big(\frac{X-\mu}{\sigma}\Big)^3\Big], \qquad \text{kurtosis} = \mathbb{E}\Big[\Big(\frac{X-\mu}{\sigma}\Big)^4\Big]$$

• Skewness measures asymmetry. Depending on its parameters, a Beta distribution can lean in either direction.
• Kurtosis measures tail thickness. A Gaussian has kurtosis exactly 3, which serves as the reference point for measuring excess kurtosis.

The hierarchy of moment existence

If $\mathbb{E}[\lvert X \rvert^k]$ is finite, every lower-order moment is also finite. The converse does not hold. Each distribution differs in "the highest order that exists," making this a quantitative indicator of tail thickness.

Distribution	Existing moments
Gaussian	Every order
Beta	Every order (because its support is bounded)
Alpha-stable ($\alpha \lt 2$)	Only orders $k \lt \alpha$ — the variance is always infinite
Cauchy ($\alpha = 1$)	Not even the first moment (mean)

Moment-generating function (MGF)

$$M_X(t) = \mathbb{E}[e^{tX}]$$

If it exists near 0, every moment can be obtained through differentiation: $M_X^{(k)}(0) = \mathbb{E}[X^k]$. It also factors into a product for independent sums.

$$M_{X+Y}(t) = M_X(t)\, M_Y(t) \quad (X \perp Y)$$

Limitation: the MGF does not exist if the integral of $e^{tX}$ diverges. Distributions whose tails decay polynomially (including alpha-stable distributions) have no MGF. We therefore need a more powerful tool.

Characteristic function

$$\varphi_X(t) = \mathbb{E}[e^{itX}] = \int_{-\infty}^{\infty} e^{itx} f_X(x)\, dx$$

This is the Fourier transform of the density. Its decisive advantages are:

1. It always exists. Because $\lvert e^{itX} \rvert = 1$, the integral converges for every distribution. Even a Cauchy distribution without a mean has a perfectly valid characteristic function.
2. It uniquely determines the distribution (Lévy's inversion theorem). If $\varphi_X = \varphi_Y$, then $X$ and $Y$ have the same distribution.
3. An independent sum becomes a product: $\varphi_{X+Y}(t) = \varphi_X(t)\varphi_Y(t)$
4. It determines convergence (Lévy's continuity theorem): pointwise convergence of characteristic functions is equivalent to convergence in distribution. The proof of the CLT stands on this theorem.

Representative characteristic functions

Distribution	$\varphi(t)$
Gaussian $\mathcal{N}(\mu, \sigma^2)$	$\exp\big(i\mu t - \tfrac{1}{2}\sigma^2 t^2\big)$
Cauchy (location 0, scale 1)	$\exp(-\lvert t \rvert)$
Alpha-stable (symmetric, scale 1)	$\exp(-\lvert t \rvert^{\alpha})$

The third row makes it immediately apparent that the Gaussian ($\alpha=2$) and Cauchy ($\alpha=1$) distributions are two ends of the same family. The unity invisible in density functions is exposed in characteristic-function space—this is why an alpha-stable distribution is defined through its characteristic function.

Relationship to moments

If $\mathbb{E}[\lvert X \rvert^k]$ is finite, then $\varphi_X$ is $k$ times differentiable and $\varphi_X^{(k)}(0) = i^k \mathbb{E}[X^k]$. Conversely, a characteristic function that is not smooth at the origin signals that the corresponding moment does not exist. The fact that $\exp(-\lvert t \rvert)$ is not differentiable at the origin and the fact that the Cauchy distribution has no mean are two expressions of the same fact.

Inequality toolbox

• Markov: $P(\lvert X \rvert \ge a) \le \mathbb{E}[\lvert X \rvert]/a$
• Chebyshev: $P(\lvert X - \mu \rvert \ge k\sigma) \le 1/k^2$ — holds regardless of the distribution as long as the variance exists
• Jensen: if $g$ is convex, then $g(\mathbb{E}[X]) \le \mathbb{E}[g(X)]$

Chebyshev gives the shortest proof of the law of large numbers.

What to remember from this page

1. Expectations and variances may not exist. Their existence is itself an important property of a distribution.
2. The range of existing moments is a quantitative indicator of tail thickness.
3. The MGF is powerful but fails for heavy tails. A characteristic function always exists, uniquely determines the distribution, and turns independent sums into products.
4. Gaussian and Cauchy distributions are the $\alpha = 2, 1$ cases of the characteristic-function family $\exp(-\lvert t \rvert^{\alpha})$.

Next step

• Gamma and Beta Functions — special functions that create the normalization constant of the Beta distribution
• Law of Large Numbers, Central Limit Theorem, and Stability — where characteristic functions are put to work