source: arxiv statistics ml: backpropagation-free trunk training via the split forward gradients
level: research
training deep neural networks with backpropagation requires storing intermediate activations, which uses a lot of memory. forward-mode methods avoid this by not needing a backward pass, but their gradient estimates get noisier as more parameters are trained. a new approach called split forward gradient splits the network at an intermediate layer. it computes the output head gradient exactly and estimates only the trunk gradient using a jacobian-vector product. this reduces noise and removes the need for a backward pass through the trunk.
experiments on wikitext-103 showed a practical problem with naive forward-gradient training. when the trunk was trained with standard adam settings, it performed worse than keeping a randomly initialized trunk frozen. the issue seems to be that adam updates every noisy, under-determined trunk coordinate too aggressively. the fix was simple: using a much smaller learning rate for the trunk reversed the performance drop. this suggests that careful tuning of optimization hyperparameters is critical for forward-mode methods.
the method comes with an adam-style convergence guarantee, meaning it can still benefit from adaptive learning rates. by reducing estimator variance, split forward gradient makes forward-mode training more practical for large models. it opens a path to training deep networks with lower memory requirements, which is important for scaling up models on limited hardware. the work highlights that forward gradients are not just a theoretical curiosity but can be made to work with the right adjustments.
why it matters: this method could allow training larger models on hardware with limited memory by avoiding the need to store activations for backpropagation.
source: arxiv statistics ml: backpropagation-free trunk training via the split forward gradients