00. Optimization
// PROBLEMTOALGORITHM — a map for rereading an optimization problem from its definition through algorithm selection
01. Mathematical Notation — The Minimum Grammar of Optimization
// BACKGROUND 1/6 — read sets, intervals, vectors, functions, mappings, summations, and norms in a consistent way
02. Calculus for Optimization — From Rates of Change to Second-Order Approximation
// BACKGROUND 2/6 — derivative, partial derivative, gradient, Hessian, Taylor expansion
03. Linear Algebra for Optimization — Directions, Curvature, and Linear Systems
// BACKGROUND 3/6 — vector, matrix, inner product, linear independence, eigenvalue, positive definiteness, linear systems
04. Numerical Analysis for Optimization — Making Paper Methods Work in Code
// BACKGROUND 4/6 — approximation error, floating-point arithmetic, iterative method, convergence, condition number
05. Probability and Statistics Background — Uncertain Evaluations and Data-Driven Objectives
// BACKGROUND 5/6 — random variable, distribution, expectation, variance, sampling, regression
06. Engineering Modeling — Turning Real Problems into Optimization Problems
// BACKGROUND 6/6 — input-output model, simulation, ODE/PDE, objective definition, constraint definition
07. Problem Formulation — Variables, Objectives, Constraints, and the Feasible Region
// CORE 1/7 — decision variables, objective function, constraints, feasible region
08. Mathematical Foundations — Gradient, Convexity, Norm, and Decomposition
// CORE 2/7 — gradient/Jacobian/Hessian, Taylor approximation, convexity, norm, inner product, orthogonality, matrix decomposition
09. Optimality Conditions — Identifying Where to Stop
// CORE 3/7 — first-order condition, second-order condition, positive definiteness
10. Gradient Descent and Line Search — Separating Direction from Step Length
// CORE 4/7 — gradient descent, exact line search, backtracking, Armijo, Wolfe, momentum, Nesterov
11. Newton and Quasi-Newton Methods — Using Curvature while Controlling Cost
// CORE 5/7 — Newton, quasi-Newton, DFP, BFGS
12. Coordinate Descent and Conjugate Gradient — Coordinates and Conjugate Directions
// CORE 6/7 — coordinate descent, conjugate gradient, preconditioning
13. Expensive Optimization — Optimization when Evaluations Are Costly
// CORE 7/7 — DOE, Latin Hypercube Sampling, surrogate modeling, surrogate-based optimization