Skip to main content

00. Math::

// FOUNDATIONS — a space for rebuilding probability, differentiation, and optimization from axioms through algorithms

These notes aim to understand mathematics not through "memorizing formulas," but by starting with why it has that shape. There are currently four groups: probability (five background articles + three advanced distribution articles), calculus (six articles), probabilistic models (one article), and optimization (six Background + seven Core articles). Read each series in order to connect definitions, properties, and applications into one flow.

The recommended learning order is below. The sidebar is sorted by recency, so follow this list if you are reading for the first time.

Background (5 articles)​

Distributions (3 advanced articles)​

  • Gaussian Distribution — definition, standardization, closure under linear combinations, maximum entropy, relationship to the CLT, and sampling
  • Beta Distribution — a distribution on [0,1], intuition for shape parameters, relationship to order statistics, Bernoulli conjugacy, and Thompson sampling
  • Alpha-Stable Distributions — definition of stability, characteristic-function parameterization, Gaussian/Cauchy/Lévy special cases, heavy tails, and the generalized CLT

Calculus (6 articles)​

  • Limits and the Definition of the Derivative — ε-δ limits, difference quotients, differentiation as the best linear approximation, differentiability and continuity, and limitations of numerical differentiation
  • Differentiation Rules and Techniques — linearity, product/quotient/chain rules, inverse-function differentiation and change of variables, implicit differentiation, logarithmic differentiation, and the score function
  • Mean Value Theorem and Taylor Expansion — MVT and Lipschitz control, Taylor's theorem and the remainder, five essential expansions, and Big-O notation
  • Multivariable Differentiation — partial derivatives, the gradient and steepest ascent, Jacobians and the matrix chain rule, Hessians, and a matrix-calculus cheatsheet
  • Differentiation, Optimization, and Automatic Differentiation — first- and second-order optimality conditions, convexity, a convergence proof for gradient descent, Newton's method, forward/reverse-mode automatic differentiation, and backpropagation
  • Delta Function — the Dirac delta as a generalized function, the sifting property, delta sequences, change of variables, derivatives of the delta, and Fourier/convolution perspectives

Probabilistic Models (1 article)​

  • Restricted Boltzmann Machine — visible/hidden bipartite graphs, the energy function, free energy, sigmoid conditional distributions, and Contrastive Divergence training

Optimization (13 articles)​

  • Optimization index — a map that connects problem formulation, background mathematics, optimality conditions, algorithms, and expensive optimization in one flow
  • 6 Background articles — mathematical notation, calculus, linear algebra, numerical analysis, probability and statistics, and engineering modeling
  • 7 Core articles — decision variables/objective/constraints/feasible region, gradient/Jacobian/Hessian, optimality conditions, line search, Newton/BFGS, coordinate descent/CG, and DOE/LHS/surrogate-based optimization

Perspective of this series​

  • Distributions are consequences of axioms — all three distributions are derived as "the unique (or natural) answer satisfying certain requirements." Gaussian distributions arise from the CLT and maximum entropy, Beta distributions from conjugacy and order statistics, and alpha-stable distributions from the stability axiom.
  • The tool is the characteristic function — to handle distributions whose density has no closed form (alpha-stable distributions), the characteristic function rather than the MGF becomes the fundamental tool.
  • Watch the tail — the decisive difference among the three distributions is tail thickness. Which moments exist determines practical applicability.
  • Differentiation is linear approximation — the Calculus series defines differentiation as "finding the best linear approximation" and uses that single perspective to connect the chain rule → Taylor expansion → gradient/Jacobian/Hessian → gradient descent and backpropagation in a straight line.
  • Read generalized functions through their action — the delta function is not a pointwise value but an action that extracts values from test functions. This perspective safely connects impulses, Green's functions, and Fourier analysis.
  • Probabilistic models create energy landscapes — an RBM learns a distribution by lowering energy where data lies and comparing it with the energy of model samples.
  • Optimization begins with modeling — decision variables, the objective function, constraints, and the feasible region must be fixed before choosing an algorithm.
  • The series converge — the CLT proof is a Taylor expansion of a characteristic function; the Laplace approximation is the moment when a second-order expansion of the log density produces a Gaussian; SGD stands on the LLN; and RBMs and expensive optimization stand on sampling and surrogate modeling.