15. Restricted Boltzmann Machine — Learning Distributions through Energy

Probabilistic Model 1/1. An RBM is a generative model that connects observed and latent variables in a bipartite graph and learns the data distribution through an energy function. The key is adjusting parameters to assign low energy to good data and high energy to poor reconstructions. Background: probability spaces, expectations and moments, and differentiation and optimization.

Goal of a generative model

A supervised model usually learns a function $f(x)$ that predicts the correct answer $y$ from an input $x$. A generative model instead tries to learn where data is abundant: the probability distribution $p(v)$.

Knowing the distribution makes two things possible:

1. Score how plausible observed data is.
2. Draw new samples from the distribution.

A Restricted Boltzmann Machine (RBM) solves this objective as an energy-based model. A state with lower energy has higher probability, and a state with higher energy has lower probability.

What is restricted in a Boltzmann Machine?

A general Boltzmann Machine is a probabilistic graph in which several nodes are connected. Its nodes are divided into two types:

• visible unit $v$ — the data actually observed
• hidden unit $h$ — an unobserved variable that explains latent factors in the data

The problem is that if every node is connected to every other node, calculating conditional distributions becomes too complex. The restriction in an RBM is simple:

Remove visible-visible and hidden-hidden connections, leaving only visible-hidden connections.

Once the graph becomes bipartite, units within the same layer are conditionally independent. Given $v$, every hidden unit can therefore be sampled in parallel; conversely, given $h$, every visible unit can be sampled in parallel.

Structure and parameters

In a Bernoulli RBM, each unit has value 0 or 1.

Symbol	Meaning	Dimension
$v$	visible vector	$d$
$h$	hidden vector	$n$
$b$	visible bias	$d$
$c$	hidden bias	$n$
$W$	visible-hidden weight matrix	$n \times d$

$b_j$ represents how easily visible unit $v_j$ turns on, and $c_i$ represents how easily hidden unit $h_i$ turns on. $W_{ij}$ determines how much the energy decreases when $v_j$ and $h_i$ are on together.

Energy and probability

The energy of an RBM is defined as

$$E(v,h) = -b^\top v - c^\top h - h^\top Wv.$$

Turning this energy into a probability gives the Boltzmann distribution:

$$p(v,h) = \frac{\exp(-E(v,h))}{Z}, \qquad Z = \sum_v \sum_h \exp(-E(v,h)).$$

$Z$ is the partition function. It is difficult to calculate exactly because every possible combination of $v,h$ must be summed. This normalization constant is precisely what makes RBM training difficult.

The probability of observed data $v$ is obtained by marginalizing every hidden state:

$$p(v) = \sum_h p(v,h).$$

In terms of free energy,

$$p(v)=\frac{\exp(-F(v))}{Z},$$

and the free energy of a Bernoulli RBM is

$$F(v) = -b^\top v - \sum_{i=1}^{n}\log\left(1+\exp(c_i+W_i v)\right).$$

Here $W_i$ is the $i$th row of $W$.

Conditional distributions — where the RBM becomes tractable

Thanks to the RBM's bipartite structure, hidden units are conditionally independent of one another given $v$:

$$p(h_i=1 \mid v)=\sigma(c_i+W_i v).$$

Conversely, visible units are conditionally independent of one another given $h$:

$$p(v_j=1 \mid h)=\sigma(b_j+W_{:j}^{\top}h).$$

Here $\sigma(x)=1/(1+e^{-x})$. These two expressions make RBMs practical. Although $p(v)$ itself is difficult, $p(h\mid v)$ and $p(v\mid h)$ are calculated directly as a sigmoid for each unit.

Training objective — lower the free energy of data

An RBM is trained to increase the likelihood of the data. For one observation $v$, the gradient of the negative log-likelihood has the structure

$$\frac{\partial}{\partial\theta}[-\log p(v)] = \frac{\partial F(v)}{\partial\theta} - \mathbb{E}_{\tilde v \sim p_{\text{model}}} \left[\frac{\partial F(\tilde v)}{\partial\theta}\right].$$

The first term is the positive phase. It lowers the free energy of real data $v$. The second term is the negative phase. It raises the free energy of samples $\tilde v$ produced by the model itself in relative terms, preventing probability mass from being spread everywhere.

The problem is that the expectation over the model distribution $p_{\text{model}}$ is difficult to obtain exactly. It is therefore approximated through sampling.

Contrastive Divergence

Contrastive Divergence (CD) approximates the negative phase with a short Gibbs chain.

1. Place data $v^{(0)}$ in the visible layer.
2. Sample $h^{(0)} \sim p(h \mid v^{(0)})$.
3. Sample $v^{(1)} \sim p(v \mid h^{(0)})$.
4. If needed, repeat the process $k$ times to obtain $v^{(k)}$.
5. Update in the direction that reduces the difference $F(v^{(0)}) - F(v^{(k)})$.

CD-1 makes only one round trip. Surprisingly, in many early applications, even this short approximation produced useful features. This is also why RBMs were important for pretraining Hinton's deep belief networks.

Caution about Gibbs sampling

Gibbs sampling in an RBM does not create candidates and accept or reject them as Metropolis-Hastings does. It samples directly and alternately from $p(h\mid v)$ and $p(v\mid h)$. The RBM's restriction is precisely the structure that makes this direct sampling possible.

Similar to an autoencoder, but with a different objective

One RBM step superficially resembles an autoencoder:

$$v \to h \to \tilde v.$$

The interpretation differs, however.

Model	Intermediate representation	Objective
Autoencoder	deterministic code	Minimize reconstruction error
RBM	stochastic hidden state	Increase likelihood of the data distribution

An RBM reconstruction is not a simple copy but a sample drawn from the model distribution. A well-trained RBM therefore forms low-energy regions where data lies rather than memorizing its inputs.

Where was it used?

RBMs are no longer the mainstream of modern generative models, but they played an important role in deep-learning history.

• Unsupervised feature learning — learn hidden representations without labels
• Deep Belief Network pretraining — initialize deep networks through layerwise RBM training
• Collaborative filtering — model binary user-item preference patterns
• Introductory example of an energy-based model — demonstrate the essentials of partition functions, free energy, and MCMC training in a small structure

VAE, GAN, and diffusion models now dominate generative modeling, but RBMs remain a good starting point for understanding the perspective of "modeling the distribution directly" and the basic grammar of energy-based learning.

What to remember from this page

1. An RBM is an energy-based generative model with a visible-hidden bipartite graph.
2. Thanks to the restricted structure, $p(h_i=1\mid v)$ and $p(v_j=1\mid h)$ factor into sigmoid conditional distributions.
3. $E(v,h)=-b^\top v-c^\top h-h^\top Wv$, and low-energy states have high probability.
4. The free energy $F(v)$ scores a visible configuration after marginalizing hidden states.
5. Contrastive Divergence approximates the likelihood gradient using the free-energy difference between real data and a short Gibbs reconstruction sample.

Reference

• Engineer's Mathematics Notes, Restricted Boltzmann Machine.

Full series index: Math:: index