level: research
prompt wrappers, which differ only in formatting, can change model scores enough to flip leaderboard conclusions. researchers studied this variance using a token-controlled protocol and introduced two metrics: the format sensitivity index (fsi), measuring accuracy range from wrapper choice, and the parseability sensitivity index (psi), measuring the range in answer parseability. the study ran 140,000 generations across 7 question-answering tasks, 5 wrapper families, and 4 instruct models ranging from 7b to 72b parameters.
results show that mean fsi varies by over 30 times across models, largely due to compliance failures. a fixed-effects regression found that parseability strongly predicts accuracy even after controlling for task, model, and wrapper. this means that how well a model follows output formatting instructions directly impacts its performance scores, and different wrappers can cause wide swings in both accuracy and the ability to extract answers.
the authors argue that reporting accuracy without accounting for wrapper variance and compliance is statistically fragile. they recommend that benchmarks include sensitivity metrics to give a more complete picture of model reliability. for practitioners, this means that small prompt engineering choices can have outsized effects on results, and relying on a single prompt format may lead to misleading conclusions about model capabilities.
why it matters: for ai and data science, this research shows that leaderboard rankings can be unreliable if they ignore prompt formatting sensitivity, so practitioners should test multiple prompt styles to get true performance estimates.