level: research
modern machine learning and ai models, especially large language models, are increasingly used to generate scientific hypotheses and mechanistic explanations from observational data. this paper argues that in the high-dimensional proxy regimes where modern ml excels, mechanistic learning is generally underdetermined. many incompatible mechanisms can produce essentially the same observational relationships on the support of the data. therefore, predictive success and coherent explanations are not enough to show that a model has discovered the true mechanism.
this underdetermination becomes uniquely hazardous with large language models. llms tend to collapse large equivalence classes of explanations into a single fluent narrative. this can create a false sense of certainty and hide the fact that many different explanations are equally consistent with the data. the paper warns that this can mislead scientists into thinking they have found a causal mechanism when they have only found one of many possible stories that fit the observations.
the paper proposes concrete standards for mechanistic ml. these norms are necessary if llm-centered workflows are to support science rather than merely simulate it. the authors argue that researchers should prioritize identifying the structure of the problem, such as invariances and causal relationships, rather than focusing on complex models that may overfit or produce plausible but incorrect explanations. this shift in focus could lead to more reliable scientific discoveries.
why it matters: for ai and data science, this highlights the risk of trusting llm-generated explanations without checking if multiple mechanisms could produce the same data, which is crucial for building reliable models in science and industry.