10. Differentiation Rules and Techniques — Differentiating the World through Composition

Calculus 2/5. Every complex function is built from arithmetic operations and compositions of basic functions. A table of basic derivatives plus four combination rules is therefore enough to differentiate any function mechanically. This "mechanical" property is the seed that later becomes automatic differentiation.

Table of basic derivatives

$f(x)$	$f'(x)$	Note
$c$	$0$	Constant
$x^n$	$n x^{n-1}$	Holds for every real $n$
$e^x$	$e^x$	The unique fixed-point function
$a^x$	$a^x \ln a$	Reduce via $a^x = e^{x \ln a}$
$\ln x$	$1/x$	Derived with the inverse-function rule
$\sin x$	$\cos x$
$\cos x$	$-\sin x$
$\tan x$	$\sec^2 x$	Derived with the quotient rule
$\arcsin x$	$1/\sqrt{1-x^2}$	Inverse-function rule
$\arctan x$	$1/(1+x^2)$	Inverse-function rule

The four combination rules

1. Linearity

$$(af + bg)' = a f' + b g'$$

Differentiation is a linear operator. Together with the linearity of expectation, it illustrates the general mathematical principle that "linear things are easy to handle."

2. Product rule

$$(fg)' = f'g + fg'$$

Intuition: when the two sides of a rectangle with area $f \times g$ each grow slightly, the increase in area is "the sum of each side's contribution + a negligible corner piece ($f'g' h^2$)." For the proof, insert $f(x+h)g(x) - f(x+h)g(x) = 0$ into $f(x+h)g(x+h) - f(x)g(x)$ and separate the terms.

3. Quotient rule

$$\left( \frac{f}{g} \right)' = \frac{f'g - fg'}{g^2} \qquad (g \neq 0)$$

It does not need to be memorized separately; applying the product rule and chain rule to $f \cdot g^{-1}$ gives the same result.

4. Chain rule — the most important

$$(g \circ f)'(x) = g'\big(f(x)\big) \cdot f'(x) \qquad\text{or in Leibniz form}\qquad \frac{dz}{dx} = \frac{dz}{dy} \cdot \frac{dy}{dx}$$

Proof sketch (linear-approximation perspective): using Definition 2 from Part 1, it takes two lines.

$$f(x+h) = f(x) + f'(x)h + o(h), \qquad g(y+k) = g(y) + g'(y)k + o(k)$$

Substituting $k = f'(x)h + o(h)$ gives

$$g\big(f(x+h)\big) = g\big(f(x)\big) + g'\big(f(x)\big) f'(x)\, h + o(h)$$

That is, "the composition of linear approximations is the product of slopes." This one line is the entire chain rule. In multiple variables it extends to "the product of Jacobian matrices," and in computational graphs it becomes backpropagation. Training a deep neural network ultimately applies this rule millions of times.

Example: $\dfrac{d}{dx} e^{-x^2/2} = e^{-x^2/2} \cdot (-x)$—the fact that the derivative of a Gaussian density has the form "itself × $(-x)$" reappears in many places, including Stein's identity for the Gaussian distribution and the Ornstein–Uhlenbeck process.

Differentiating inverse functions

If $f$ has an inverse $f^{-1}$, then

$$\big(f^{-1}\big)'(y) = \frac{1}{f'\big(f^{-1}(y)\big)}$$

Derivation: differentiate both sides of $f(f^{-1}(y)) = y$ with respect to $y$ (chain rule) to get $f'(f^{-1}(y)) \cdot (f^{-1})'(y) = 1$.

Example — derivative of $\ln$: since $y = \ln x \iff x = e^y$,

$$(\ln x)' = \frac{1}{e^{\ln x}} = \frac{1}{x}$$

In probability, inverse-function differentiation is the heart of the change-of-variables formula. If $Y = g(X)$ and $g$ is monotonic,

$$f_Y(y) = f_X\big(g^{-1}(y)\big) \left| \frac{d}{dy} g^{-1}(y) \right|$$

Normalizing flows stand on this formula.

Implicit differentiation

Even when $y$ cannot be explicitly solved as a function of $x$, differentiating both sides of the relation yields $y'$.

Example — a circle: differentiating both sides of $x^2 + y^2 = 1$ with respect to $x$ gives $2x + 2y\,y' = 0$, so $y' = -x/y$. The tangent slope is obtained without solving for $y$ in terms of $x$.

This technique is used directly in implicit layers, where a layer is defined as the solution of an equation, and in hyperparameter differentiation (bilevel optimization based on the implicit function theorem).

Logarithmic differentiation

For a function with layers of products and powers, taking the logarithm first to turn them into sums makes differentiation overwhelmingly easier.

$$(\ln f)' = \frac{f'}{f} \quad\Longrightarrow\quad f' = f \cdot (\ln f)'$$

Example — $f(x) = x^x$: because $\ln f = x \ln x$, $(\ln f)' = \ln x + 1$, and therefore

$$(x^x)' = x^x (\ln x + 1)$$

Connection to statistics: differentiating the log-likelihood $\ell(\theta) = \sum_i \ln p(x_i \mid \theta)$ instead of the likelihood $L(\theta) = \prod_i p(x_i \mid \theta)$ directly is exactly logarithmic differentiation. The score function $\nabla_\theta \ln p$, cross-entropy loss, and REINFORCE's log-derivative trick $\nabla p = p \nabla \ln p$—half the reason logarithms appear everywhere in ML is "to turn products into sums and make differentiation easy."

Higher-order derivatives

Differentiating a derivative again gives the second derivative $f''$; repeating gives $f^{(n)}$.

• $f'$ — slope (first-order information): increasing/decreasing
• $f''$ — curvature (second-order information): convex/concave, above/below the tangent
• On an interval where $f'' > 0$, $f$ is convex—the condition leads to a "guaranteed minimum" in Optimization Part 5.

Example: $(e^{-x^2/2})'' = (x^2 - 1)e^{-x^2/2}$—the inflection points of the Gaussian bell are exactly $x = \pm 1$ (the standard-deviation points).

Summary

• A table of basic derivatives plus four rules (linearity, product, quotient, and chain) mechanically differentiates every elementary function.
• Chain rule = "the composition of linear approximations is the product of slopes." It is the mathematical core of deep-learning backpropagation.
• Inverse-function differentiation leads to the change-of-variables formula for random variables, while logarithmic differentiation leads to log-likelihoods and score functions.
• The second derivative determines convexity and connects to second-order conditions in optimization.

Next: Mean Value Theorem and Taylor Expansion