13. Differentiation, Optimization, and Automatic Differentiation — Machines That Learn from Gradients

Calculus 5/5. This page assembles the tools from the previous four parts: why the condition "derivative = 0" is the starting point of optimization, why gradient descent works (Taylor expansion), and how deep-learning frameworks calculate gradients without error (automatic differentiation = systematic application of the chain rule).

First-order condition — Fermat's theorem

If $f$ has an extremum at an interior point $x^\ast$ and is differentiable there, then

$$f'(x^\ast) = 0 \qquad \text{(multivariable: } \nabla f(\mathbf{x}^\ast) = \mathbf{0}\text{)}$$

Proof intuition: if $f'(x^\ast) > 0$, moving slightly right makes the value larger and moving left makes it smaller, so the point cannot be an extremum. As long as a gradient remains, there is room for improvement—the principle running through every optimization algorithm.

Caution: this is only a necessary condition. There are critical points that are not extrema (inflection or saddle points), such as $x = 0$ for $f(x) = x^3$; it does not detect extrema at nondifferentiable points such as $|x|$; and endpoints of an interval must be checked separately.

Second-order condition

At a critical point, the first-order term of the second-order Taylor expansion vanishes, so the second-order term determines its fate:

$$f(\mathbf{x}^\ast + \mathbf{h}) \approx f(\mathbf{x}^\ast) + \frac{1}{2}\, \mathbf{h}^{\top} H\, \mathbf{h}$$

Hessian $H$	Classification
$H \succ 0$ (all eigenvalues positive)	Minimum
$H \prec 0$ (all eigenvalues negative)	Maximum
Mixed eigenvalue signs	Saddle point
Semidefinite (includes zero eigenvalues)	Inconclusive—inspect higher-order terms

Convexity — the world where "local = global"

A function $f$ is convex if the line segment connecting any two points lies above its graph:

$$f\big(\lambda x + (1-\lambda) y\big) \le \lambda f(x) + (1-\lambda) f(y) \qquad (0 \le \lambda \le 1)$$

For differentiable functions, equivalent conditions are:

• First order: $f(y) \ge f(x) + \nabla f(x)^{\top}(y - x)$—the tangent (or tangent plane) always lies below the graph.
• Second order: $f'' \ge 0$ (multivariable: $H \succeq 0$ everywhere).

For a convex function, every critical point is a global minimum. Substituting $\nabla f(x^\ast) = 0$ into the first-order condition immediately gives $f(y) \ge f(x^\ast)$ for every $y$. Least squares, logistic regression (cross-entropy), and SVMs are convex and therefore carry a guarantee that "training works." Deep neural networks are non-convex and rely instead on empirical properties (escaping saddle points and overparameterization).

Gradient descent

$$\mathbf{x}_{t+1} = \mathbf{x}_t - \eta\, \nabla f(\mathbf{x}_t)$$

Why it works—a one-line proof with Taylor: if $f$ is $L$-smooth ($\|\nabla f(x) - \nabla f(y)\| \le L\|x-y\|$, Lipschitz control from the MVT), then

$$f(\mathbf{x}_{t+1}) \le f(\mathbf{x}_t) - \eta \Big(1 - \frac{L\eta}{2}\Big) \|\nabla f(\mathbf{x}_t)\|^2$$

When $\eta < 2/L$, the parenthesized term is positive, so the loss decreases at every step unless the gradient is zero. This one inequality contains both why an excessively large learning rate diverges (the second-order term defeats the first-order term) and why values near $\eta = 1/L$ are safe.

Every variant differs only in "how it processes the gradient":

• SGD: estimates $\nabla f$ without bias from a minibatch rather than the full dataset—the LLN guarantees convergence of the average.
• Momentum: uses an exponential moving average of gradients to cancel zigzags in narrow valleys.
• Adam: adjusts scale per coordinate using the second moment of the gradient—a cheap symptomatic treatment for the Hessian condition-number problem.

Newton's method — using second-order information

Directly use the step that minimizes the second-order expansion:

$$\mathbf{x}_{t+1} = \mathbf{x}_t - H^{-1} \nabla f(\mathbf{x}_t)$$

• Near the optimum it has quadratic convergence (the number of correct digits roughly doubles at each step), overwhelmingly faster than the linear convergence of gradient descent.
• Cost: computing and storing $H$ takes $O(n^2)$, and inversion takes $O(n^3)$. This is impossible with millions of parameters, so approximations such as quasi-Newton (L-BFGS) or diagonal approximations (the Adam family) are used.
• In statistics, applying Newton's method to the log-likelihood gives Fisher scoring, a classical numerical method for MLE.

Automatic differentiation

Comparison of three ways to obtain a gradient:

Method	Principle	Problem
Numerical differentiation	$\frac{f(x+h)-f(x-h)}{2h}$	Truncation/rounding-error trade-off; $2n$ evaluations for $n$ partial derivatives
Symbolic differentiation	Transform expressions into expressions	Expression swell; difficult control flow
Automatic differentiation	Apply the chain rule to values	Accurate to machine precision; differentiates the program as written

The core of automatic differentiation: every program is a composition of primitive operations ($+, \times, \exp, \dots$), so if the derivative of each operation is known, the chain rule can assemble the derivative of the whole exactly. It is not an approximation.

Forward mode vs reverse mode

For the Jacobian product $J_L \cdots J_2 J_1$ of $f: \mathbb{R}^n \to \mathbb{R}^m$, the choice is which end to multiply from:

• Forward mode: propagate $J \cdot \mathbf{v}$ (a Jacobian-vector product) from input to output. One pass obtains the derivative along one input direction → cost $\propto n$. Favorable when $n$ is small.
• Reverse mode: backpropagate $\mathbf{v}^{\top} J$ (a vector-Jacobian product) from output to input. One pass obtains the gradients of all inputs with respect to one output → cost $\propto m$.

Deep learning has $n =$ millions to billions of parameters and $m = 1$ (a scalar loss), so reverse mode is the only viable choice, and it is backpropagation. The full gradient of one loss is obtained at a constant multiple (roughly 2–3×) of the forward-pass cost. The price is memory: intermediate activations must be stored until the backward pass. Gradient checkpointing shifts this trade-off back toward recomputation.

Backpropagation once by hand

$f(x, y) = (x + y) \cdot x^2$ at $(x, y) = (2, 3)$:

Forward pass: $a = x + y = 5$, $b = x^2 = 4$, $f = ab = 20$.

Backward pass ($\bar{v} = \partial f / \partial v$):

$$\bar{f} = 1, \quad \bar{a} = \bar{f} \cdot b = 4, \quad \bar{b} = \bar{f} \cdot a = 5$$ $$\bar{x} = \bar{a} \cdot \frac{\partial a}{\partial x} + \bar{b} \cdot \frac{\partial b}{\partial x} = 4 \cdot 1 + 5 \cdot 2x = 24, \qquad \bar{y} = \bar{a} \cdot 1 = 4$$

The contributions from the two paths entering $x$ (the addition node and square node) are summed, which is the "sum over every path" rule of the multivariable chain rule. Check: $f = x^3 + x^2 y$, $\partial f/\partial x = 3x^2 + 2xy = 12 + 12 = 24$. ✓ (This is a check within the document; in an actual framework, gradient checking—comparison with numerical differentiation—plays this role.)

Handling nondifferentiable points in practice

Deep-learning functions are full of nondifferentiable points, including ReLU ($x=0$), $|x|$, and max-pooling boundaries. Practice still works because:

• A convex function always has a subgradient—a set of slopes of lines that pass below the graph instead of a tangent—and convergence theory holds with any member.
• Nondifferentiable points form a set of measure zero, so an input from a continuous distribution has probability zero of landing exactly on one.
• Frameworks conventionally fix one value (for example, ReLU'(0) = 0).

Summary

• The first-order condition $\nabla f = 0$ is necessary; the second-order condition (definiteness of the Hessian) distinguishes minima, maxima, and saddles.
• Convexity is the structure that guarantees "local minimum = global minimum," classified through equivalent first- and second-order conditions.
• Convergence of gradient descent follows from one Taylor inequality, which also reveals the learning-rate upper bound $2/L$.
• Automatic differentiation is not an approximation but exact assembly of the chain rule; in deep learning with a scalar loss, reverse mode (backpropagation) is the only economical choice.

Full series index: Math:: index