09. Limits and the Definition of the Derivative — Inventing the Instantaneous Rate of Change

Calculus 1/5. Differentiation turns the observation that "a curve looks like a line when viewed closely enough" into mathematics. This page rigorously defines differentiation in the language of limits and explains why the second definition, "linear approximation," is more powerful.

Limits

The $\varepsilon$-$\delta$ definition rigorously expresses the intuition that a function $f$ "approaches" some value $L$ near a point $a$.

$$\lim_{x \to a} f(x) = L \quad\Longleftrightarrow\quad \forall \varepsilon > 0,\ \exists \delta > 0:\ 0 < |x - a| < \delta \implies |f(x) - L| < \varepsilon$$

How to read it: "No matter how small an error tolerance $\varepsilon$ is chosen, bringing the input sufficiently close to $a$ (within $\delta$) confines the output error inside it." As a game, the opponent names $\varepsilon$ and I answer with $\delta$. The limit exists only if every $\varepsilon$ can be answered.

Two points to note:

• Because of the condition $0 < |x - a|$, the value at $x = a$ is irrelevant to the limit. A limit may exist even when $f(a)$ is undefined.
• Both the left limit $\lim_{x \to a^-}$ and right limit $\lim_{x \to a^+}$ must exist and agree for the limit to exist.

Continuity

$$f \text{ is continuous at } a \quad\Longleftrightarrow\quad \lim_{x \to a} f(x) = f(a)$$

The limit and function value coincide. This is the rigorous version of "it can be drawn without a break."

Definition 1 of the derivative — limit of a difference quotient

$$f'(a) = \lim_{h \to 0} \frac{f(a+h) - f(a)}{h}$$

The fraction is the slope of the secant connecting the two points $(a, f(a))$ and $(a+h, f(a+h))$. Taking the limit $h \to 0$ makes the secant converge to the tangent, whose slope is the derivative.

Physically, the limit of average velocity (rate of change over an interval) is instantaneous velocity (instantaneous rate of change). Three notations are commonly used.

Notation	Name	Emphasis
$f'(x)$	Lagrange	Operation that maps a function to a new function
$\dfrac{df}{dx}$	Leibniz	Ratio of changes; strong for substitutions and chains
$\dot{x}$	Newton	Derivative with respect to time (physics)

Definition 2 of the derivative — best linear approximation

The same content can be written differently. A function $f$ is differentiable at $a$ if and only if some number $m$ exists such that

$$f(a + h) = f(a) + m\,h + o(h) \qquad (h \to 0)$$

Here $o(h)$ is an error that approaches zero faster than $h$, meaning $\lim_{h\to 0} o(h)/h = 0$. Then $m = f'(a)$.

This perspective is the real definition. Differentiation is "finding the linear function that best imitates a function locally," and the derivative is the slope of that linear function. Its advantages are:

• Generalization is immediate — for a multivariable function, "one slope" becomes a "linear map (matrix)," which is the Jacobian. The difference-quotient definition does not extend directly to several variables (what should we divide by?).
• The chain rule is natural — the chain rule follows from the one sentence, "the composition of linear approximations is the product of linear approximations."
• The zeroth step of Taylor expansion — adding second-, third-, and higher-order terms to a linear approximation yields the Taylor expansion.

Differentiable ⟹ continuous (the converse is false)

Differentiability implies continuity. The proof is one line:

$$\lim_{h \to 0} \big[ f(a+h) - f(a) \big] = \lim_{h \to 0} \frac{f(a+h) - f(a)}{h} \cdot h = f'(a) \cdot 0 = 0$$

The converse does not hold. Counterexamples come in several grades.

• $f(x) = |x|$ — continuous at the origin, but not differentiable because the left derivative $-1$ and right derivative $+1$ differ. This is the archetypal kink. ReLU $\max(0, x)$ in ML is exactly this situation; in practice, one subgradient value is chosen at the origin.
• $f(x) = x^{1/3}$ — the derivative diverges at the origin because the tangent is vertical.
• Weierstrass function — there are functions that are continuous everywhere but differentiable nowhere. The intuition that "a continuous function is generally differentiable" is wrong. Sample paths of Brownian motion also have this property (with probability 1).

Basic examples — calculate directly from the definition

$f(x) = x^2$:

$$f'(x) = \lim_{h \to 0} \frac{(x+h)^2 - x^2}{h} = \lim_{h \to 0} \frac{2xh + h^2}{h} = \lim_{h \to 0} (2x + h) = 2x$$

$f(x) = e^x$:

$$f'(x) = \lim_{h \to 0} \frac{e^{x+h} - e^x}{h} = e^x \lim_{h \to 0} \frac{e^h - 1}{h} = e^x$$

The final limit being 1 is the very definition of $e$. The property of being "a function equal to its own derivative" makes $e^x$ special, and it is why the exponential function is a fundamental building block throughout differential equations, probability (moment-generating functions), and analysis.

A taste of numerical differentiation — why the definition should not be coded literally

Translating the definition directly into code gives the forward difference:

$$f'(x) \approx \frac{f(x+h) - f(x)}{h}$$

The error of this approximation is $O(h)$, while the central difference

$$f'(x) \approx \frac{f(x+h) - f(x-h)}{2h}$$

has error $O(h^2)$ (proved with Taylor expansion in Part 3). But decreasing $h$ indefinitely amplifies floating-point rounding error by $1/h$, making the result worse. The trade-off between truncation error and rounding error produces an optimal $h$ (roughly $h \sim 10^{-8}$ in double precision), and this limitation is the fundamental reason for using automatic differentiation instead of numerical differentiation.

Summary

• A limit is an $\varepsilon$-$\delta$ game, and a derivative is the limit of a difference quotient.
• The better definition is the best linear approximation: $f(a+h) = f(a) + f'(a)h + o(h)$. Multivariable calculus, the chain rule, and Taylor expansion all emerge from this perspective.
• Differentiable ⟹ continuous, but the converse is false—kinks (ReLU), vertical tangents, and Weierstrass functions.
• Directly numericalizing the definition hits a precision limit, which is why automatic differentiation exists.

Next: Differentiation Rules and Techniques