level: research
researchers compared two ways of getting varied answers from language models: running one model many times with high temperature, and running many different models once each at zero temperature. they used a random-matrix test to separate real signal from sampling noise. across five model families and three benchmarks, a single model showed at most one dimension of structured variation above noise. in contrast, an ensemble of 24 models showed four clear dimensions of variation.
the study tested whether repeated sampling from one model reveals what the model does not know. self-consistency, which uses majority voting across multiple runs, gives good per-question uncertainty estimates. however, the variation across questions from one model is shallow. it does not capture the cross-question structure that a diverse ensemble provides. a matched-difficulty bernoulli null model rarely produced more than one significant dimension, confirming the ensemble's diversity is not just due to difficulty.
the findings suggest that temperature scaling alone cannot substitute for true model diversity. while self-consistency is useful for single-question confidence, it fails to uncover deeper epistemic uncertainty that spans multiple questions. this limits the ability to detect when a model systematically misunderstands a topic. the dimensionality gap highlights a fundamental difference between within-model and across-model variation.
why it matters: practitioners relying on repeated sampling for uncertainty estimation may miss systematic model failures that only diverse ensembles can reveal.