00. Math
// FOUNDATIONS — a space for rebuilding probability, differentiation, and optimization from axioms through algorithms
Optimization::
14 items
14. Delta Function — Mass Concentrated at a Point
Calculus 6/6. The Dirac delta is not an ordinary function but a linear operation that extracts the value at one point from a test function. Keep the engineering intuition of "an infinitely narrow spike with total mass 1," but handle calculations safely only inside integrals. Background: limits and differentiation, differentiation rules, and multivariable differentiation.
15. Restricted Boltzmann Machine — Learning Distributions through Energy
Probabilistic Model 1/1. An RBM is a generative model that connects observed and latent variables in a bipartite graph and learns the data distribution through an energy function. The key is adjusting parameters to assign low energy to good data and high energy to poor reconstructions. Background: probability spaces, expectations and moments, and differentiation and optimization.
01. Probability Spaces and Random Variables — Where Understanding Distributions Begins
Background 1/5. Every distribution in this series is ultimately "a measure induced by a random variable defined on a probability space." The goal of this page is to make that sentence completely interpretable.
02. Expectation, Moments, and Characteristic Functions — Numerical Summaries of Distributions
Background 2/5. This page organizes the tools for summarizing the infinite-dimensional object called a distribution with finitely many numbers (moments) and one function (the characteristic function). The characteristic function is the only practical means of defining alpha-stable distributions, so it receives particular attention.
03. Gamma and Beta Functions — A Factory for Normalization Constants
Background 3/5. A probability density is complete only after a normalization constant makes its total integral equal to 1. Both the Gaussian's $\sqrt{2\pi}$ and the Beta distribution's $B(\alpha,\beta)$ come from the two special functions on this page.
04. Law of Large Numbers, Central Limit Theorem, and Stability — Universal Laws of Sums
Background 4/5. The law of large numbers (LLN), central limit theorem (CLT), and stable distributions emerge in sequence from one question: "what shape does the sum of many independent factors take?" This page covers why Gaussians appear everywhere—and why the Gaussian is not the only answer.
05. Bayesian Inference and Conjugate Priors — How Distributions Learn
Background 5/5. So far, distributions have been treated as "given." This page covers the procedure for updating beliefs about a distribution after observing data (Bayesian inference) and the special structure that gives this update a closed form (conjugacy). This is where the Beta distribution takes the stage as a "probability distribution over probabilities."
06. Gaussian Distribution — The Distribution of Universality
Distribution 1/3. Understand the Gaussian (normal) distribution not as a "bell-shaped curve," but as the unique answer reached by three independent derivations. Background: probability spaces, moments and characteristic functions, and the CLT.
07. Beta Distribution — A Probability Distribution over Probabilities
Distribution 2/3. The Beta distribution lives on $0,1]$ [Gamma and Beta functions, Bayesian inference and conjugacy.
08. Alpha-Stable Distributions — The Universal Law of Heavy Tails
Distribution 3/3. Alpha-stable (α-stable, Lévy stable) distributions are a four-parameter family defined by the stability axiom that "their shape does not change under independent sums." They include Gaussian and Cauchy distributions as special cases and provide the CLT limits for infinite-variance data. Background: characteristic functions, stability and the generalized CLT.
09. Limits and the Definition of the Derivative — Inventing the Instantaneous Rate of Change
Calculus 1/5. Differentiation turns the observation that "a curve looks like a line when viewed closely enough" into mathematics. This page rigorously defines differentiation in the language of limits and explains why the second definition, "linear approximation," is more powerful.
10. Differentiation Rules and Techniques — Differentiating the World through Composition
Calculus 2/5. Every complex function is built from arithmetic operations and compositions of basic functions. A table of basic derivatives plus four combination rules is therefore enough to differentiate any function mechanically. This "mechanical" property is the seed that later becomes automatic differentiation.
11. Mean Value Theorem and Taylor Expansion — Imitating Functions with Polynomials
Calculus 3/5. The real utility of differentiation is "controlling the behavior of an entire function with one derivative." The mean value theorem is that control mechanism, and repeatedly applying it to reconstruct a function as a polynomial gives the Taylor expansion. Approximation theory, optimization, and the proof of the CLT all begin here.
12. Multivariable Differentiation — From Gradients to Jacobians and Hessians
Calculus 4/5. The moment the input becomes a vector, "one slope" is no longer enough. Because differentiation was defined as the "best linear approximation," the extension is natural: the linear approximation is merely promoted from a number to a vector (gradient), then from a vector to a matrix (Jacobian). Every derivative in deep learning is written in the language of this page.
13. Differentiation, Optimization, and Automatic Differentiation — Machines That Learn from Gradients
Calculus 5/5. This page assembles the tools from the previous four parts: why the condition "derivative = 0" is the starting point of optimization, why gradient descent works (Taylor expansion), and how deep-learning frameworks calculate gradients without error (automatic differentiation = systematic application of the chain rule).