00. Optimization::
// PROBLEM_TO_ALGORITHM— a map for rereading an optimization problem from its definition through algorithm selection
Optimization is the process of expressing "what can be changed, what counts as good, and what must be respected" in equations, then exploiting that structure to find a solution. This section connects problem formulation, gradient descent, Newton and quasi-Newton methods, conjugate gradient, and surrogate-based optimization in one sequence.
If you are reading for the first time, align the language with the six Background articles before reading the seven Core articles. If you already know calculus and linear algebra, you can begin with Problem Formulation.
Background (6 articles)
- Mathematical Notation — set, interval, vector, function, mapping, summation, norm
- Calculus for Optimization — derivative, partial derivative, gradient, Hessian, Taylor expansion
- Linear Algebra for Optimization — vector, matrix, inner product, linear independence, eigenvalue, positive definiteness, linear systems
- Numerical Analysis for Optimization — approximation error, floating-point arithmetic, iterative method, convergence, condition number
- Probability and Statistics Background — random variable, distribution, expectation, variance, sampling, regression
- Engineering Modeling — input-output model, simulation, ODE/PDE, objective definition, constraint definition
Core optimization (7 articles on problems and algorithms)
- Problem Formulation — decision variables, objective function, constraints, feasible region
- Mathematical Foundations — gradient/Jacobian/Hessian, Taylor approximation, convexity, norm, inner product, orthogonality, matrix decomposition
- Optimality Conditions — first-order condition, second-order condition, positive definiteness
- Gradient Descent and Line Search — exact line search, backtracking, Armijo, Wolfe, momentum, Nesterov
- Newton and Quasi-Newton Methods — Newton, DFP, BFGS
- Coordinate Descent and Conjugate Gradient — coordinate descent, conjugate gradient, preconditioning
- Expensive Optimization — DOE, Latin Hypercube Sampling, surrogate modeling, surrogate-based optimization
Coverage checklist
Every requested topic appears explicitly at the locations below.
| Area | Included topics | Location |
|---|---|---|
| Problem formulation | decision variables, objective function, constraints, feasible region | 07 |
| Mathematical foundations | gradient/Jacobian/Hessian, Taylor approximation, convexity, norm/inner product, orthogonality, matrix decomposition | 08 |
| Optimality conditions | first-order, second-order, positive definiteness | 09 |
| Optimization algorithms | gradient descent, line search, momentum, Nesterov, Newton, DFP, BFGS, coordinate descent, conjugate gradient, preconditioning | 10–12 |
| Expensive optimization | DOE, LHS, surrogate modeling, surrogate-based optimization | 13 |
| Background | notation, calculus, linear algebra, numerical analysis, probability/statistics, engineering modeling | 01–06 |
Reading perspective
- Formulation is half the problem — if variables and constraints are chosen poorly, even a good algorithm only solves the wrong problem quickly.
- Derivatives are local information — the gradient tells you which direction to move, and the Hessian tells you about curvature.
- Linear algebra is geometry — norms, inner products, orthogonality, and eigenvalues form the language for explaining algorithmic convergence.
- Numerical analysis is practicality — without floating point, condition numbers, and convergence, a method that works on paper breaks in code.
- Expensive optimization is a budget problem — when one objective evaluation is costly, samples must be chosen more intelligently and a surrogate must be built.