source: arxiv statistics ml: riemannian stochastic optimization for sufficient dimension reduction
level: research
sufficient dimension reduction (sdr) makes high-dimensional regression easier by projecting covariates onto a low-dimensional subspace that keeps the conditional mean of the response. existing gradient-based methods either work in the full space and struggle with high dimensions, or localize in the reduced space but have high per-iteration costs. the authors show that minimizers of the population minimum average variance estimation (mave) risk approximate the same grassmannian target as the outer product of gradients (opg). they reformulate the empirical criterion as a smooth maximization on the stiefel manifold with a closed-form riemannian gradient.
the resulting algorithm, smave, combines sparse projected-space nearest-neighbor localization with riemannian stochastic gradient ascent. a simplified version has almost-sure convergence and a non-asymptotic rate matching standard stochastic gradient methods. this approach avoids the curse of dimensionality by operating directly on the manifold of low-dimensional subspaces, using efficient riemannian optimization techniques.
the method is designed for regression problems where the response depends only on a few linear combinations of predictors. by estimating the central mean subspace, smave can reduce the number of variables needed for prediction or visualization. the use of stochastic gradients and sparse localization makes it scalable to large datasets, while the riemannian framework ensures the solution stays on the stiefel manifold.
why it matters: it provides a scalable way to find low-dimensional structure in high-dimensional regression, which is common in data science and machine learning applications.
source: arxiv statistics ml: riemannian stochastic optimization for sufficient dimension reduction