A Paper Roadmap for Understanding Surrogate Gradients
Surrogate gradients are a detour connecting spiking neural networks, or SNNs, with the training tools of modern deep learning. The forward pass uses real spikes, while the backward pass uses a differentiable fake gradient. That sounds simple in one sentence, but properly understanding the idea requires three pieces of background.
First, why are spiking neurons discontinuous dynamical systems? Second, why does backpropagation require differentiability? Third, how were SNNs trained before surrogate gradients, and what prevented those methods from progressing?






