source: arxiv statistics ml: confidencebench: evaluating confidence calibration in large language models
level: research
large language models often produce fluent but incorrect answers. in high-stakes settings, it is not enough for a model to be accurate. it must also recognize when it is likely to be wrong. confidencebench is a new benchmark that evaluates this ability. it uses the brier score, a proper scoring rule that rewards honest probability estimates. the benchmark tests 15 frontier models by asking them to give verbal confidence ratings. this method does not need access to internal model probabilities, so it works for both closed and open models.
the benchmark includes 200 private multiple-choice questions. they cover four areas: spatial reasoning, high-precision math, word lookup, and questions that cannot be answered. each model is tested three times to check consistency. the questions are designed to be challenging and to expose gaps in a model's self-assessment. by keeping the questions private, the benchmark prevents models from memorizing answers from training data.
claude opus 4.6 and gemini 3.1 pro preview performed best. both achieved a brier score of 0.103. this score is much better than a calibrated random baseline. the results show that some models can give useful confidence estimates. however, there is still room for improvement. the benchmark provides a standard way to compare how well different models express uncertainty.
why it matters: reliable confidence scores help users decide when to trust a model's output, reducing risks in applications like medical advice or legal analysis.
source: arxiv statistics ml: confidencebench: evaluating confidence calibration in large language models