12. Multivariable Differentiation — From Gradients to Jacobians and Hessians

Calculus 4/5. The moment the input becomes a vector, "one slope" is no longer enough. Because differentiation was defined as the "best linear approximation," the extension is natural: the linear approximation is merely promoted from a number to a vector (gradient), then from a vector to a matrix (Jacobian). Every derivative in deep learning is written in the language of this page.

Partial derivatives

For $f: \mathbb{R}^n \to \mathbb{R}$, differentiate only in the $x_i$ direction while holding the other variables fixed:

$$\frac{\partial f}{\partial x_i}(\mathbf{a}) = \lim_{h \to 0} \frac{f(\mathbf{a} + h\,\mathbf{e}_i) - f(\mathbf{a})}{h}$$

A partial derivative is only the slope of a cross-section along a coordinate axis. Unlike the single-variable case, a function may fail to be "differentiable" even when every partial derivative exists—there are functions that appear well-behaved along the axes but are discontinuous in a diagonal direction. The real multivariable definition therefore uses linear approximation.

Total differentiability = existence of a linear approximation

A function $f: \mathbb{R}^n \to \mathbb{R}^m$ is differentiable at $\mathbf{a}$ if a linear map (matrix) $J$ exists such that

$$f(\mathbf{a} + \mathbf{h}) = f(\mathbf{a}) + J\,\mathbf{h} + o(\|\mathbf{h}\|)$$

This $J \in \mathbb{R}^{m \times n}$ is the Jacobian matrix, whose entries are partial derivatives:

$$J_{ij} = \frac{\partial f_i}{\partial x_j} \qquad J = \begin{pmatrix} \frac{\partial f_1}{\partial x_1} & \cdots & \frac{\partial f_1}{\partial x_n} \\ \vdots & \ddots & \vdots \\ \frac{\partial f_m}{\partial x_1} & \cdots & \frac{\partial f_m}{\partial x_n} \end{pmatrix}$$

A sufficient condition: if every partial derivative exists and is continuous ($C^1$), the function is differentiable. Most functions used in practice fall into this category.

Gradient — when the output is scalar

When $m = 1$, the Jacobian is a row vector, and its transpose is called the gradient:

$$\nabla f = \left( \frac{\partial f}{\partial x_1}, \dots, \frac{\partial f}{\partial x_n} \right)^{\!\top} \qquad f(\mathbf{a} + \mathbf{h}) \approx f(\mathbf{a}) + \nabla f(\mathbf{a})^{\top} \mathbf{h}$$

Directional derivative and steepest ascent

The rate of change in the direction of a unit vector $\mathbf{u}$ (the directional derivative) is

$$D_{\mathbf{u}} f = \nabla f^{\top} \mathbf{u} = \|\nabla f\| \cos\theta$$

By Cauchy–Schwarz, this is maximized when $\mathbf{u}$ points in the direction of $\nabla f$. Therefore:

• The gradient points in the direction of the function's steepest increase, and its magnitude is the slope in that direction.
• It is always perpendicular to the level set.
• Moving in the $-\nabla f$ direction therefore gives the fastest local decrease—the reason gradient descent ($\mathbf{x} \leftarrow \mathbf{x} - \eta \nabla f$) exists.

Multivariable chain rule = product of Jacobians

For $f: \mathbb{R}^n \to \mathbb{R}^m$ and $g: \mathbb{R}^m \to \mathbb{R}^k$,

$$J_{g \circ f}(\mathbf{x}) = J_g\big(f(\mathbf{x})\big) \cdot J_f(\mathbf{x})$$

The "product of slopes" from one variable has merely been promoted to a matrix product; the content is identical ("the composition of linear approximations is the product of linear maps"). The gradient of a neural network $L = \ell(f_L(\cdots f_2(f_1(\mathbf{x}))))$ is exactly this product of layerwise Jacobians. The order of multiplication (left-to-right versus right-to-left) creates the distinction between forward- and reverse-mode automatic differentiation.

Written along scalar paths, it takes the familiar form. If $z = g(y_1, \dots, y_m)$ and $y_j = f_j(x)$,

$$\frac{dz}{dx} = \sum_{j=1}^{m} \frac{\partial z}{\partial y_j} \frac{d y_j}{dx}$$

"Add the contribution from every path"—this is the backpropagation rule that sums every gradient entering a node in a computational graph.

Hessian — second-order information

For $f: \mathbb{R}^n \to \mathbb{R}$, collect the second partial derivatives into a matrix:

$$H_{ij} = \frac{\partial^2 f}{\partial x_i\, \partial x_j}$$

• Schwarz's theorem: if the second partial derivatives are continuous, $H_{ij} = H_{ji}$—the Hessian is symmetric.
• Second-order Taylor expansion:

$$f(\mathbf{x} + \mathbf{h}) \approx f(\mathbf{x}) + \nabla f^{\top} \mathbf{h} + \frac{1}{2} \mathbf{h}^{\top} H\, \mathbf{h}$$

• Classification at a critical point ($\nabla f = 0$): if $H \succ 0$ (positive definite), it is a minimum; if $H \prec 0$, a maximum; and if the eigenvalues have mixed signs, a saddle point. It is known that high-dimensional loss landscapes contain overwhelmingly more saddle points than maxima or minima.
• If the ratio between the Hessian's eigenvalues (condition number) is large, the loss landscape becomes a narrow valley and gradient descent zigzags slowly—the problem that preconditioning, momentum, and Adam try to alleviate.
• Statistical connection: the negative expected Hessian of the log-likelihood is the Fisher information matrix, and it acts as the inverse covariance matrix of the posterior in the Laplace approximation.

Matrix differentiation cheatsheet

These identities recur in ML papers. ($\mathbf{a}$ and $\mathbf{b}$ are constant vectors, $A$ is a constant matrix, and denominator layout is used.)

$f(\mathbf{x})$	$\nabla f$	Note
$\mathbf{a}^{\top}\mathbf{x}$	$\mathbf{a}$	Linear term
$\mathbf{x}^{\top} A \mathbf{x}$	$(A + A^{\top})\,\mathbf{x}$	$2A\mathbf{x}$ when $A$ is symmetric
$\|\mathbf{x}\|^2 = \mathbf{x}^{\top}\mathbf{x}$	$2\mathbf{x}$	Special case above ($A = I$)
$\|A\mathbf{x} - \mathbf{b}\|^2$	$2A^{\top}(A\mathbf{x} - \mathbf{b})$	Least squares
$\ln \det A$	$\partial/\partial A = (A^{-1})^{\top}$	Gaussian log-likelihood

Example — normal equations for least squares: solving $\nabla \|A\mathbf{x} - \mathbf{b}\|^2 = 0$ gives

$$A^{\top} A\, \mathbf{x} = A^{\top} \mathbf{b}$$

The closed-form solution to linear regression emerges from one gradient calculation. The MLE of a multivariate Gaussian (estimating mean and covariance) is derived the same way, including the $\ln\det$ identity.

Summary

• Multivariable differentiation has the same definition as single-variable differentiation: the "best linear approximation." A scalar output gives a gradient; a vector output gives a Jacobian.
• Existence of partial derivatives ≠ differentiability. $C^1$ is safe.
• The gradient points in the direction of steepest ascent (perpendicular to level sets)—the basis of gradient descent.
• The chain rule is a product of Jacobian matrices, and choosing the multiplication order gives forward/reverse-mode automatic differentiation.
• The Hessian is second-order information that determines convexity, saddle points, condition numbers, and Fisher information.

Next: Differentiation, Optimization, and Automatic Differentiation