07. Beta Distribution — A Probability Distribution over Probabilities

Distribution 2/3. The Beta distribution lives on $[0,1]$: it models proportions and probabilities themselves. Background: Gamma and Beta functions, Bayesian inference and conjugacy.

Definition

$X \sim \mathrm{Beta}(\alpha, \beta)$, $\alpha, \beta > 0$:

$$f(x) = \frac{x^{\alpha-1}(1-x)^{\beta-1}}{B(\alpha, \beta)}, \qquad x \in [0, 1]$$

The normalization constant is the Beta function:

$$B(\alpha, \beta) = \int_0^1 t^{\alpha-1}(1-t)^{\beta-1}\, dt = \frac{\Gamma(\alpha)\Gamma(\beta)}{\Gamma(\alpha+\beta)}$$

Reading the structure: the density is the product of "the pull toward $x$, $x^{\alpha-1}$" and "the pull toward $1-x$, $(1-x)^{\beta-1}$." $\alpha$ is the parameter that creates a preference near 1, and $\beta$ creates a preference near 0.

A map of shapes — what the parameters create

$(\alpha, \beta)$	Shape	Interpretation
$(1, 1)$	Flat (Uniform)	Complete ignorance
$(\alpha = \beta > 1)$	Symmetric bell centered at 0.5	Belief that it is "half and half"; sharper as the values grow
$(\alpha > \beta > 1)$	Peak skewed to the right	More weight on success
$(\alpha \lt 1,\ \beta \lt 1)$	U-shaped (diverges at both ends)	Belief in "one of the extremes" (either 0 or 1)
$(1, \beta > 1)$	Monotonically decreasing	Preference near 0
$(\alpha \to \infty$, fixed ratio$)$	Concentrated at one point	Limit of certainty

A single two-parameter family represents uniform, bell, U, and J shapes. It is effectively a "universal shape library" on $[0,1]$, and this expressiveness is the secret of its popularity as a prior.

Moments

They follow immediately from the Beta-type integral pattern.

$$\mathbb{E}[X] = \frac{\alpha}{\alpha+\beta}, \qquad \mathrm{Var}(X) = \frac{\alpha\beta}{(\alpha+\beta)^2(\alpha+\beta+1)}$$

• The mean is the proportion of success counts. With $\alpha = 7, \beta = 3$, the mean is 0.7.
• As $n_0 = \alpha + \beta$ (total pseudo-count) grows, the variance shrinks as $O(1/n_0)$—the "amount of evidence" tightens the variance.
• When $\alpha, \beta > 1$, the mode is $\frac{\alpha-1}{\alpha+\beta-2}$. A difference between mean and mode signals asymmetry.
• Because the support is bounded, moments of every order exist. This contrasts with the opposite extreme (alpha-stable) on the hierarchy of moments.

Derivation 1: conjugacy — the state variable of Bayesian inference

A summary of Beta–Bernoulli conjugacy:

$$\mathrm{Beta}(\alpha, \beta) \ +\ (\text{successes } k, \text{ failures } n-k) \ \longrightarrow\ \mathrm{Beta}(\alpha + k,\ \beta + n - k)$$

• The likelihood $\theta^k (1-\theta)^{n-k}$ and the density kernel have the same algebraic shape, so multiplying them preserves the shape. Conjugacy is another name for this "shape match."
• The posterior mean decomposes into a weighted average of the prior mean and sample proportion:

$$\frac{\alpha+k}{\alpha+\beta+n} = \frac{n_0}{n_0+n}\cdot\frac{\alpha}{n_0} + \frac{n}{n_0+n}\cdot\frac{k}{n}$$

The formula directly shows how the prior's influence disappears as data accumulates ($n \gg n_0$).

Derivation 2: order statistics — where Beta arises naturally

For the $k$th value $U_{(k)}$ obtained by sorting i.i.d. $U_1, \dots, U_n \sim \mathrm{Uniform}(0,1)$,

$$U_{(k)} \sim \mathrm{Beta}(k,\ n-k+1)$$

Intuition: for $U_{(k)}$ to lie near $x$, $k-1$ values must be on the left ($x^{k-1}$), $n-k$ values must be on the right ($(1-x)^{n-k}$), and the number of such combinations becomes the normalization constant. The Beta distribution was not "invented" as a prior; it emerges on its own from order within a uniform distribution. Quantile confidence intervals in nonparametric statistics stand on this fact.

Derivation 3: Gamma ratio representation

For independent $G_1 \sim \mathrm{Gamma}(\alpha), G_2 \sim \mathrm{Gamma}(\beta)$,

$$\frac{G_1}{G_1 + G_2} \sim \mathrm{Beta}(\alpha, \beta)$$

• This gives the interpretation of "the relative size of two parts" and is also the standard method for generating Beta random variables (two Gamma samples → ratio).
• Generalizing to $k$ parts yields the Dirichlet distribution $\mathrm{Dir}(\alpha_1, \dots, \alpha_k)$—a distribution over probability vectors (the simplex). Beta is Dirichlet with $k=2$. LDA topic models and conjugate priors for multinomial distributions all stand on this generalization.

Applications

Thompson sampling (multi-armed bandits)

For each slot machine $i$, maintain the belief about its success probability as $\mathrm{Beta}(\alpha_i, \beta_i)$.

1. Sample $\tilde{\theta}_i \sim \mathrm{Beta}(\alpha_i, \beta_i)$ from each arm.
2. Pull the arm with the largest $\tilde{\theta}_i$.
3. On success, $\alpha_i$ += 1; on failure, $\beta_i$ += 1.

An uncertain arm has high sample variance and occasionally takes the maximum (exploration), while a reliably good arm is selected consistently (exploitation). The exploration–exploitation balance emerges automatically from posterior variance without a separate hyperparameter. It is a standard algorithm for online advertising, recommender systems, and A/B test automation.

A/B testing

Giving the conversion rates $\theta_A, \theta_B$ their own Beta posteriors lets us estimate $P(\theta_B > \theta_A \mid D)$ directly with Monte Carlo (sample from both and compare). The ability to state the intuitive "probability that B is better" rather than a p-value is an advantage of Bayesian A/B testing.

Other applications

• Confidence intervals for proportions: Clopper–Pearson intervals are expressed using Beta quantiles.
• Project management (PERT): model task duration with a scaled Beta distribution from minimum, mode, and maximum estimates.
• Machine learning: priors for label smoothing and probability calibration.

What to remember from this page

1. Beta is a universal two-parameter shape family on $[0,1]$—it represents uniform, bell, U, and J shapes.
2. $\alpha, \beta$ are pseudo-counts: the mean is $\frac{\alpha}{\alpha+\beta}$, and a larger total count reduces variance.
3. Three independent derivations—Bernoulli conjugacy, uniform order statistics, and a Gamma ratio—reach the same distribution.
4. Thompson sampling obtains the exploration–exploitation balance for free by sampling from a Beta posterior.
5. Its multivariate generalization is Dirichlet, and Beta is the $k=2$ case.

Next step

• Alpha-Stable Distributions — opposite bounded support, the world where moments disappear