source: hugging face blog: discoformer: one transformer for density and score, across distributions
level: technical
many problems in machine learning and science require recovering a distribution from data points. this means estimating the density, which shows where points cluster, and the score, the gradient of log-density that points toward more probable regions. diffusion models and bayesian sampling rely on the score, but current methods force a trade-off. kernel density estimation works without training but fails in high dimensions. neural score-matching models stay accurate in high dimensions but need retraining for each new distribution.
discoformer is a transformer that takes a set of data points and outputs both density and score in one pass. it uses cross-attention to evaluate at any query point, not just data locations. the model has a shared backbone with two heads, one for density and one for score. because score is the gradient of log-density, any mismatch between the heads provides a consistency loss. at inference, the model can adapt to out-of-distribution data by taking gradient steps on this loss without ground truth.
the transformer architecture generalizes kernel density estimation. a single attention head can approximate a gaussian kernel, so one cross-attention block reproduces kde. discoformer learns multiple scales and adapts them to data. trained on gaussian mixture models, which are universal approximators with closed-form targets, the model beats kde in density and score estimation. in 100 dimensions, it reduces score error by 6.5 times and density error by over 37 times. it also generalizes to unseen distributions like laplace and student-t.
why it matters: a pretrained score estimator that works in high dimensions without retraining can reduce costs across generative modeling, bayesian inference, and scientific computing.
source: hugging face blog: discoformer: one transformer for density and score, across distributions