01. Probability Spaces and Random Variables — Where Understanding Distributions Begins

Background 1/5. Every distribution in this series is ultimately "a measure induced by a random variable defined on a probability space." The goal of this page is to make that sentence completely interpretable.

Why begin with axioms?

Starting from intuition, such as "each face has probability 1/6 when a die is rolled," causes no problem in finite cases. But to handle objects such as the alpha-stable distributions that are the final goal of this series—where even the mean may not exist—we need a rigorous foundation for "what probability is." That foundation is Kolmogorov's axiomatic system, on top of which Gaussian, Beta, and alpha-stable distributions are all described in the same language.

Probability space

A probability space is a triple $(\Omega, \mathcal{F}, P)$.

1. Sample space $\Omega$

The set of every possible outcome.

• One coin toss: $\Omega = \lbrace H, T \rbrace$
• One die roll: $\Omega = \lbrace 1,2,3,4,5,6 \rbrace$
• Lifetime of a component: $\Omega = [0, \infty)$ — an uncountably infinite set is also possible.

2. Event space $\mathcal{F}$ (sigma-algebra)

This is the collection of "questions whose probability can be asked." A collection of subsets of $\Omega$ satisfying the following three conditions is called a sigma-algebra (σ-algebra).

1. $\Omega \in \mathcal{F}$ — it must always be possible to ask whether "something happens."
2. If $A \in \mathcal{F}$, then $A^c \in \mathcal{F}$ — if an event can be asked about, so can its complement.
3. If $A_1, A_2, \dots \in \mathcal{F}$, then $\bigcup_{i=1}^{\infty} A_i \in \mathcal{F}$ — a countable union can also be asked about.

Why not use every subset?

When $\Omega$ is uncountable, like an interval of real numbers, it is known to be impossible to assign a consistent probability (a notion of length) to every subset (Vitali sets and the Banach–Tarski paradox). A σ-algebra therefore contains only "sets for which probability can safely be asked." On the real line, the Borel σ-algebra $\mathcal{B}(\mathbb{R})$ generated by open intervals is the standard choice.

3. Probability measure $P$ — Kolmogorov's axioms

$P: \mathcal{F} \to [0,1]$ is a function satisfying three axioms.

1. Non-negativity: $P(A) \ge 0$ for every $A \in \mathcal{F}$
2. Normalization: $P(\Omega) = 1$
3. Countable additivity: for pairwise disjoint $A_1, A_2, \dots$,

$$P\Big(\bigcup_{i=1}^{\infty} A_i\Big) = \sum_{i=1}^{\infty} P(A_i)$$

These three lines are all there is. Conditional probability, independence, Bayes' theorem, the law of large numbers, and the central limit theorem—everything that follows—is a logical consequence of these axioms.

Random variables

A random variable is often introduced as a "random value," but its exact definition is a function.

$$X : \Omega \to \mathbb{R}$$

Not every function qualifies: it must be measurable. For every Borel set $B \in \mathcal{B}(\mathbb{R})$, the preimage $X^{-1}(B) = \lbrace \omega : X(\omega) \in B \rbrace$ must belong to $\mathcal{F}$. This condition is what makes the question "what is the probability that $X$ falls inside $B$?" meaningful in the first place.

Distribution: the measure induced by a random variable

A random variable $X$ creates a new probability measure on the real line.

$$\mu_X(B) = P(X^{-1}(B)) = P(X \in B)$$

This $\mu_X$ is called the distribution (push-forward measure) of $X$. It is precisely what "Gaussian distribution" or "Beta distribution" refers to. We can forget what the original probability space $\Omega$ was and perform every calculation using only the measure $\mu_X$ on the real line.

Three ways to describe a distribution

Cumulative distribution function (CDF)

$$F_X(x) = P(X \le x)$$

• Monotonically nondecreasing and right-continuous
• $\lim_{x \to -\infty} F_X(x) = 0$, $\lim_{x \to \infty} F_X(x) = 1$
• Every real-valued random variable has a CDF. A CDF always exists even when there is no density or no moments. A distribution such as an alpha-stable distribution, whose density has no closed form, is still completely defined by its CDF.

Probability mass function (PMF, discrete)

When $X$ takes only countably many values, $p_X(x) = P(X = x)$ is sufficient. Bernoulli, binomial, and Poisson distributions belong here.

Probability density function (PDF, continuous)

If the CDF is absolutely continuous, there is a density $f_X$ satisfying

$$F_X(x) = \int_{-\infty}^{x} f_X(t)\, dt, \qquad f_X(x) = \frac{dF_X}{dx}(x)$$

Points to note:

• $f_X(x)$ is not a probability. It is possible for $f_X(x) > 1$ (for example, near the center of Beta(5,5)). Probability is always obtained through an integral.
• In the continuous case, $P(X = x) = 0$ for every individual point.

Quantile function

This is the generalized inverse of the CDF, $F_X^{-1}(u) = \inf \lbrace x : F_X(x) \ge u \rbrace$. When $U \sim \mathrm{Uniform}(0,1)$, the fact that $F_X^{-1}(U)$ has the same distribution as $X$—inverse transform sampling—is a fundamental principle of random-number generation.

Joint distributions, independence, and conditioning

When handling several random variables $X, Y$ together:

• Joint CDF: $F_{X,Y}(x,y) = P(X \le x, Y \le y)$
• Independence: $F_{X,Y}(x,y) = F_X(x) F_Y(y)$ for every $x, y$. If densities exist, this is equivalent to $f_{X,Y} = f_X f_Y$.
• Conditional density: $f_{X \mid Y}(x \mid y) = f_{X,Y}(x,y) / f_Y(y)$

Independence is a central assumption throughout this series. The "independent and identically distributed (i.i.d.)" condition in the central limit theorem uses this exact definition, and likelihood factorization in Bayesian inference comes from conditional independence.

Example: same distribution, different probability spaces

Let $\Omega = [0,1]$, let $P$ be the length measure (Lebesgue measure), and define $X(\omega) = \omega$. Then $X \sim \mathrm{Uniform}(0,1)$. On the other hand, define $X'(\omega) = \sum_i \omega_i 2^{-i}$ on $\Omega' = \lbrace H,T \rbrace^{\infty}$ (infinitely many coin tosses), and the same $\mathrm{Uniform}(0,1)$ distribution appears. The probability spaces differ, but the distributions are the same. This is why distribution theory is powerful: it abstracts away details of the underlying space and leaves only a measure on the real line.

What to remember from this page

1. Probability is an axiomatic system on $(\Omega, \mathcal{F}, P)$, and a distribution is the measure $\mu_X$ pushed onto the real line by a random variable.
2. A CDF always exists. A PDF may exist or may have no closed form (foreshadowing alpha-stable distributions).
3. A density value is not a probability. Probability is an integral.
4. Any distribution can be sampled using its quantile function and a uniform distribution.

Next step

• Expectation, Moments, and Characteristic Functions — how to extract numerical summaries from the measure $\mu_X$