05. Bayesian Inference and Conjugate Priors — How Distributions Learn

Background 5/5. So far, distributions have been treated as "given." This page covers the procedure for updating beliefs about a distribution after observing data (Bayesian inference) and the special structure that gives this update a closed form (conjugacy). This is where the Beta distribution takes the stage as a "probability distribution over probabilities."

Bayes' theorem

It follows directly from the definition of conditional probability.

$$P(\theta \mid D) = \frac{P(D \mid \theta)\, P(\theta)}{P(D)}$$

The role of each term:

Term	Name	Meaning
$P(\theta)$	Prior	Belief about the parameter before observing data
$P(D \mid \theta)$	Likelihood	Plausibility of observing this data when the parameter is $\theta$
$P(\theta \mid D)$	Posterior	Belief updated after observing data
$P(D)$	Evidence	Normalization constant $\int P(D \mid \theta) P(\theta)\, d\theta$

The key shift in perspective: in frequentist statistics, $\theta$ is an unknown constant, while in Bayesian statistics, $\theta$ itself is a random variable. The notion of a "distribution over parameters" therefore makes sense, and inference is the process of tightening that distribution with data.

In practice, it is written as a proportionality that omits the normalization constant.

$$\underbrace{P(\theta \mid D)}_{\text{posterior}} \ \propto\ \underbrace{P(D \mid \theta)}_{\text{likelihood}} \times \underbrace{P(\theta)}_{\text{prior}}$$

The problem: the posterior is generally intractable

$P(D) = \int P(D \mid \theta) P(\theta) d\theta$ has no closed form for most combinations of likelihood and prior. Major branches of modern Bayesian statistics—MCMC and variational inference—are devoted to approximating this integral. But for certain combinations, this integral is solved for free. That is conjugacy.

Conjugate priors

A prior family $\mathcal{P}$ is conjugate to a likelihood $P(D \mid \theta)$ if

$$\text{prior} \in \mathcal{P} \implies \text{posterior} \in \mathcal{P}$$

The posterior remains in the same family as the prior, and updating ends with arithmetic on the parameters.

Representative example: Beta–Bernoulli

Suppose we estimate the probability $\theta \in [0,1]$ that a coin comes up heads. If $k$ heads are observed in $n$ tosses, the likelihood is

$$P(D \mid \theta) = \theta^{k} (1-\theta)^{n-k}$$

Choose $\mathrm{Beta}(\alpha, \beta)$ as the prior, so $P(\theta) \propto \theta^{\alpha-1}(1-\theta)^{\beta-1}$. Then

$$P(\theta \mid D) \ \propto\ \theta^{k}(1-\theta)^{n-k} \cdot \theta^{\alpha-1}(1-\theta)^{\beta-1} = \theta^{\alpha+k-1}(1-\theta)^{\beta+n-k-1}$$

The right-hand side is exactly the kernel of $\mathrm{Beta}(\alpha+k,\ \beta+n-k)$. Without performing a single integral,

$$\theta \mid D \ \sim\ \mathrm{Beta}(\alpha + k,\ \beta + n - k)$$

The update rule is "add the counts." $\alpha$ can be read as the number of successes observed so far plus the prior success count, while $\beta$ is the failure count. This is why $\alpha, \beta$ are called pseudo-counts. Starting from $\mathrm{Beta}(1,1)$ (= Uniform) makes an "observer with no bias," while starting from $\mathrm{Beta}(50,50)$ gives a "strong prior belief that the coin is fair."

Why those combinations fit — the exponential family

This is no coincidence. If the likelihood belongs to the exponential family,

$$p(x \mid \theta) = h(x) \exp\big( \eta(\theta)^{\top} T(x) - A(\theta) \big)$$

a conjugate prior always exists, and updating reduces to accumulating the sufficient statistic $T(x)$. Frequently used conjugate pairs include:

Likelihood	Conjugate prior	Posterior update
Bernoulli / binomial	Beta	Add success and failure counts
Poisson	Gamma	Add event counts and observation counts
Normal (known variance)	Normal	Precision-weighted mean
Normal (unknown mean and variance)	Normal-Inverse-Gamma	Update four parameters
Categorical / multinomial	Dirichlet	Add counts for each category

The third row is important: the Gaussian is conjugate to itself for its mean. From the perspective of conjugacy, Gaussian and Beta distributions play the same role on their respective stages (continuous location parameters versus probability parameters).

Posterior predictive distribution

The distribution of the next observation $x_{\text{new}}$ is obtained by integrating out parameter uncertainty.

$$P(x_{\text{new}} \mid D) = \int P(x_{\text{new}} \mid \theta)\, P(\theta \mid D)\, d\theta$$

For Beta–Bernoulli, the Beta function identity gives this integral in closed form:

$$P(x_{\text{new}} = 1 \mid D) = \mathbb{E}[\theta \mid D] = \frac{\alpha + k}{\alpha + \beta + n}$$

This formula contains Laplace's rule of succession as a special case (with $\alpha = \beta = 1$, it becomes $\frac{k+1}{n+2}$). Smoothing that "does not conclude the probability is zero even after zero observations" emerges automatically.

Sequential updating — the online nature of Bayesian inference

Thanks to conjugacy, even when data arrives as a stream, only a few parameters are needed as state.

$$\mathrm{Beta}(\alpha,\beta) \xrightarrow{\ \text{1 success}\ } \mathrm{Beta}(\alpha+1,\beta) \xrightarrow{\ \text{1 failure}\ } \mathrm{Beta}(\alpha+1,\beta+1) \to \cdots$$

• The final posterior is identical whether all data is inserted at once or one item at a time (exchangeability).
• This structure enables Thompson sampling in multi-armed bandits (see the Beta distribution page) and real-time monitoring of A/B tests.

Limitations of conjugacy

• Restricted expressiveness: real prior knowledge may not have a Beta shape. Conjugacy is a computational convenience, not an epistemological justification.
• In complex models (hierarchical models and neural networks), the conjugate structure breaks and MCMC or variational inference becomes necessary. Even within those algorithms, however, conjugate blocks remain useful performance levers (for example, full conditional distributions in Gibbs sampling).

What to remember from this page

1. Bayesian inference = prior × likelihood ∝ posterior. Its essence is the shift in perspective that promotes the parameter to a random variable.
2. Conjugacy = the property that the posterior remains in the prior's family. Updating reduces to arithmetic on sufficient statistics.
3. Beta–Bernoulli is the archetypal conjugate pair: $\mathrm{Beta}(\alpha,\beta) \to \mathrm{Beta}(\alpha+k, \beta+n-k)$, and its parameters are pseudo-counts.
4. The posterior predictive mean $\frac{\alpha+k}{\alpha+\beta+n}$ has automatic smoothing built in.

Next step

The background theory is complete. We now confront the three distributions directly.

• Gaussian Distribution
• Beta Distribution
• Alpha-Stable Distributions