source: arxiv machine learning: bayesian wind tunnels for model selection
level: research
prior work showed transformers can do exact bayesian filtering inside a fixed hypothesis class. this new study asks if they can also select the correct hypothesis class from data. the researchers built model-selection bayesian wind tunnels, which are controlled settings where ground-truth posteriors over hypothesis classes are known in closed form. they used fixed-point-free involutions, a type of function where applying it twice returns the original input. a 2.8 million parameter transformer reached 0.01-bit entropy agreement with the bayesian optimum across three random seeds. this held for both integer tokens and opaque symbols whose meanings changed every episode.
the method extends to non-nested comparisons, like involutions versus 3-cycles, where neither class is a subset of the other. in these cases, the class-posterior mean absolute error stayed under 0.001. this shows genuine model selection that goes beyond simple biases toward smaller or subset models. the transformer learned to distinguish hypothesis classes without relying on fixed token meanings, suggesting it captured the underlying relational structure.
the work also identifies a sharp perceptual access condition. when the discriminative information is available, the transformer matches the bayesian ideal. this points to a clear boundary for when neural networks can perform principled model selection. the findings help clarify how transformers handle uncertainty across different model structures, not just within a single model.
why it matters: understanding when transformers can do bayesian model selection guides the design of ai systems that need to choose between competing explanations of data, such as in scientific discovery or adaptive learning.
source: arxiv machine learning: bayesian wind tunnels for model selection