02. Calculus for Optimization — From Rates of Change to Second-Order Approximation

// BACKGROUND 2/6 — derivative, partial derivative, gradient, Hessian, Taylor expansion

In optimization, calculus is a tool for reading "how the function changes around the current position." Even without perfect knowledge of the entire function, local information at the current point can determine the next direction to move.

Derivative

The derivative of a one-dimensional function $f:\mathbb{R}\to\mathbb{R}$ is its instantaneous rate of change at one point.

$$f'(x)=\lim_{h\to 0}\frac{f(x+h)-f(x)}{h}.$$

From an optimization perspective, if $f'(x)$ is positive, the value increases when moving right; if it is negative, the value decreases when moving right. To minimize, moving opposite the slope is therefore natural.

Partial derivative

For a multivariable function, inspect the rate of change while varying only one coordinate:

$$\frac{\partial f}{\partial x_i}(x).$$

This is the rate of change when only $x_i$ moves and every other coordinate remains fixed. Think of it as a tool for inspecting sensitivity by coordinate.

Gradient

The gradient collects every partial derivative into a vector:

$$\nabla f(x)=\left(\frac{\partial f}{\partial x_1},\dots,\frac{\partial f}{\partial x_n}\right)^\top.$$

The important fact is that the gradient points in the direction of fastest increase. Gradient descent therefore moves as follows:

$$x_{k+1}=x_k-\alpha_k\nabla f(x_k).$$

Here $\alpha_k$ is the step size.

Hessian

The Hessian collects second derivatives into a matrix:

$$\nabla^2 f(x)= \begin{bmatrix} \frac{\partial^2 f}{\partial x_1^2} & \cdots & \frac{\partial^2 f}{\partial x_1\partial x_n}\\ \vdots & \ddots & \vdots\\ \frac{\partial^2 f}{\partial x_n\partial x_1} & \cdots & \frac{\partial^2 f}{\partial x_n^2} \end{bmatrix}.$$

If the gradient is "slope," the Hessian is "curvature." Newton's method uses the Hessian to minimize a second-order approximation of the function directly.

Taylor expansion

Taylor expansion approximates a complex function with a polynomial near the current point.

First-order approximation:

$$f(x+p)\approx f(x)+\nabla f(x)^\top p.$$

Second-order approximation:

$$f(x+p)\approx f(x)+\nabla f(x)^\top p+\frac{1}{2}p^\top \nabla^2 f(x)p.$$

Here $p$ gives the direction and magnitude of movement from the current point. Gradient descent fundamentally uses the first-order approximation, while Newton's method uses the second-order approximation.

One-page summary

Concept	Role in optimization
derivative	Reads the one-dimensional rate of change
partial derivative	Reads the sensitivity of one coordinate
gradient	Gives the direction of fastest increase
Hessian	Gives curvature and second-order structure
Taylor expansion	Creates an easy approximate problem around the current point

For existing in-depth explanations, see Limits and the Definition of the Derivative, Multivariable Differentiation, and Differentiation and Optimization.

Next: Linear Algebra for Optimization