11. Mean Value Theorem and Taylor Expansion — Imitating Functions with Polynomials

Calculus 3/5. The real utility of differentiation is "controlling the behavior of an entire function with one derivative." The mean value theorem is that control mechanism, and repeatedly applying it to reconstruct a function as a polynomial gives the Taylor expansion. Approximation theory, optimization, and the proof of the CLT all begin here.

Rolle's theorem → mean value theorem

Rolle's theorem: if $f$ is continuous on $[a,b]$, differentiable on $(a,b)$, and $f(a) = f(b)$, then some $c \in (a,b)$ satisfies $f'(c) = 0$. (If both ends have the same height, there is a horizontal tangent somewhere.)

Mean Value Theorem (MVT): under the same conditions, even when $f(a) \neq f(b)$, some $c \in (a,b)$ satisfies

$$f'(c) = \frac{f(b) - f(a)}{b - a}$$

This means there is a tangent somewhere with the same slope as the secant. The proof ends by subtracting the secant from $f$ and reducing the problem to Rolle's theorem.

Immediate consequences of the MVT

• Sign of the derivative ⟹ monotonicity: if $f' > 0$ on an interval, $f$ is strictly increasing. ($f(y) - f(x) = f'(c)(y-x) > 0$)
• $f' \equiv 0$ ⟹ constant: "functions with the same derivative differ by a constant" follows here, and the $+C$ in an indefinite integral is a consequence of this theorem.
• Lipschitz control: if $|f'| \le L$, then $|f(x) - f(y)| \le L|x - y|$. This single inequality controls every variation in function values with a derivative bound, and it is the standard $L$-smoothness assumption in convergence proofs for gradient descent.

L'Hôpital's rule

For limits of the form $\frac{0}{0}$ or $\frac{\infty}{\infty}$,

$$\lim_{x \to a} \frac{f(x)}{g(x)} = \lim_{x \to a} \frac{f'(x)}{g'(x)} \qquad \text{(when the limit on the right exists)}$$

This is a consequence of Cauchy's mean value theorem (the two-function version of the MVT). Once Taylor expansion below becomes familiar, however, most limits can be read immediately by "expanding and comparing the lowest-order terms" without L'Hôpital's rule.

Taylor's theorem

If $f$ is $n+1$ times differentiable, then near $a$,

$$f(x) = \underbrace{\sum_{k=0}^{n} \frac{f^{(k)}(a)}{k!}(x-a)^k}_{\text{Taylor polynomial } T_n(x)} + \underbrace{\frac{f^{(n+1)}(c)}{(n+1)!}(x-a)^{n+1}}_{\text{Lagrange remainder}}$$

($c$ is some point between $a$ and $x$.) When $n = 0$, this is exactly the MVT—Taylor's theorem is the higher-order extension of the MVT.

The key message: because the remainder is controlled by $(x-a)^{n+1}$, as $x$ approaches $a$, the polynomial approximation improves at the rate of its order.

$$f(x) = T_n(x) + O\big((x-a)^{n+1}\big)$$

Why the coefficient is $f^{(k)}(a)/k!$

$T_n$ is "the unique degree-$n$ polynomial whose derivatives through order $n$ all agree with $f$ at $a$." Since $\frac{d^k}{dx^k}(x-a)^k = k!$, division by $k!$ is required for the derivatives to match. The factorial from the Gamma function page again serves as a normalization constant here.

Five expansions to memorize

Around $x \to 0$:

$$e^x = 1 + x + \frac{x^2}{2!} + \frac{x^3}{3!} + \cdots$$ $$\sin x = x - \frac{x^3}{3!} + \frac{x^5}{5!} - \cdots \qquad \cos x = 1 - \frac{x^2}{2!} + \frac{x^4}{4!} - \cdots$$ $$\ln(1+x) = x - \frac{x^2}{2} + \frac{x^3}{3} - \cdots \qquad (|x| < 1)$$ $$(1+x)^\alpha = 1 + \alpha x + \frac{\alpha(\alpha-1)}{2!} x^2 + \cdots \qquad (|x| < 1)$$

Examples of use—moments when they actually appear in probability and ML:

• $\ln(1+x) \approx x - \frac{x^2}{2}$: log return ≈ simple return, and the origin of small-sample correction terms.
• $e^x \approx 1 + x$: compound interest → continuous compounding, $(1 + \lambda/n)^n \to e^\lambda$ (Poisson limit).
• The heart of the CLT proof (Part 4): expand the characteristic function to second order, $\varphi(t/\sqrt{n})^n = \big(1 - \frac{t^2}{2n} + o(1/n)\big)^n \to e^{-t^2/2}$. Without Taylor expansion, the proof of the CLT cannot begin.
• Linearizing sigmoid and softmax; analyzing local behavior of activation functions.

Second-order expansion — the language of optimization

Optimization most often uses the expansion through second order:

$$f(x + h) \approx f(x) + f'(x)\,h + \frac{1}{2} f''(x)\, h^2$$

• At a point where $f'(x) = 0$ (a critical point), the first-order term vanishes and the second-order term dominates—$f'' > 0$ gives a minimum, and $f'' < 0$ gives a maximum. This is the second-order condition in Part 5.
• Newton's method chooses the next step by matching through the second-order term: $h = -f'(x)/f''(x)$.
• In statistics, a second-order expansion of the log-likelihood leads to the Laplace approximation (approximating the posterior with a Gaussian) and Fisher information. The fact that "expanding the log density to second order near its maximum produces a Gaussian" is another route demonstrating the universality of the Gaussian.

Big-O / little-o

The standard vocabulary for expressing approximation precision (as $x \to a$):

Notation	Meaning	Example
$f = O(g)$	$\|f/g\|$ is bounded	$\sin x = x + O(x^3)$
$f = o(g)$	$f/g \to 0$	Differentiability: $f(a+h) - f(a) - f'(a)h = o(h)$
$f \sim g$	$f/g \to 1$	Stirling: $\Gamma(z) \sim \sqrt{2\pi/z}\,(z/e)^z$

The claim in Part 1 that the $O(h^2)$ central difference is better than the $O(h)$ forward difference follows immediately by subtracting two Taylor expansions: $f(x+h) - f(x-h) = 2f'(x)h + \frac{2 f'''(x)}{3!}h^3 + \cdots$

Pitfalls of Taylor series

The series as $n \to \infty$ does not always converge to the original function.

• Radius of convergence: $\ln(1+x)$ is valid only for $|x| < 1$. Using the expansion outside that radius is simply wrong.
• Smooth but non-analytic functions: for $f(x) = e^{-1/x^2}$ ($f(0)=0$), every derivative at the origin is 0, so its Taylor series is identically 0, but the function itself is not 0. "Even knowing every derivative may not reveal the function."
• In contrast, for objects with good properties such as characteristic functions, a local expansion determines the entire distribution—see Part 2 for the contrast with the moment problem.

Summary

• The MVT is a mechanism for "controlling changes in function values through derivatives"—monotonicity, Lipschitz bounds, and $+C$ all follow from it.
• Taylor's theorem is the higher-order extension of the MVT, and the remainder $O((x-a)^{n+1})$ guarantees approximation quality.
• The five expansions $e^x, \sin, \cos, \ln(1+x), (1+x)^\alpha$ are reused everywhere, from the CLT proof to numerical-stability tricks.
• Second-order expansion is a shared language of optimization (Newton, second-order conditions) and statistics (Laplace approximation, Fisher information).

Next: Multivariable Differentiation