source: arxiv statistics ml: born discrete, made smooth: variational formulation of shallow neural networks
level: research
neural network training is usually a messy, non-convex optimization problem solved with stochastic gradient descent. this paper proposes a different approach: instead of optimizing discrete parameters, it frames shallow network training as a variational problem over a continuous density of parameters. the authors define a family of functionals on weighted sobolev spaces that are λ-convex, meaning they have a single global minimum and are stable under perturbations.
unlike mean-field or wasserstein gradient flow methods, which often struggle with regularity and discretization, this formulation inherits elliptic regularity from the sobolev setting. the optimal parameter density turns out to be almost c³ smooth, a surprising result that gives strong theoretical guarantees. the key insight is that the variational problem can be solved by a single linear system, completely avoiding iterative optimization.
the work focuses on shallow networks with one hidden layer and a specific activation function, but it opens a path to more principled training methods. by turning a discrete, heuristic process into a well-posed continuous problem, the approach could lead to faster and more reliable training, as well as better understanding of network behavior. the linear system solution is a major practical advantage, though extending the idea to deep networks remains an open challenge.
why it matters: this could lead to faster, more reliable neural network training by replacing iterative optimization with a direct linear solve, and provides new theoretical tools for understanding network solutions.
source: arxiv statistics ml: born discrete, made smooth: variational formulation of shallow neural networks