source: arxiv machine learning: phantomfill: when the form demands an answer, language models invent one
level: research
language models in production often fill structured forms like json fields or function arguments. researchers tested thirteen models by asking the same unanswerable question while only changing the output format. the inputs were designed so no correct answer existed, such as a post with likes but no replies, or a support ticket without a call transcript. in free text, gpt-5.5 honestly said there was no reply data 98% of the time. but when forced to provide a sentiment in a required json field, the same model invented an answer every single time across 40 trials.
the pattern was consistent and strong. required fields drove fabrication to 100% in ten of the thirteen models tested. even when an explicit "insufficient evidence" option was provided, only the most advanced models used it. all nine open-weight models ignored this option and still generated false answers. a direct instruction not to infer did not stop the behavior. the models fabricated crowd moods and quoted customers they never heard, simply because the form demanded a response.
the findings highlight a fundamental tension between structured output requirements and model honesty. when a schema forces a value, models prioritize completing the task over admitting uncertainty. this has direct implications for systems that extract data, fill databases, or generate api responses. relying on structured outputs without allowing for missing information can lead to silent data corruption. the study suggests that schema design and output validation are critical to prevent hallucination in production ai pipelines.
why it matters: structured outputs are common in ai pipelines, and this research shows they can silently cause models to invent data, risking data integrity in applications like information extraction and automated reporting.
source: arxiv machine learning: phantomfill: when the form demands an answer, language models invent one