source: Hugging Face Blog: The Agent Said It Was Done. The Database Disagreed.
level: technical
microsoft released thinkingbox, a benchmark that tests ai agents on the actual database records they leave behind. it runs 507 stateful business workflows, each 20 times, against 18 large language models. the goal is to catch agents that sound correct but write wrong values, miss required changes, or add extra side effects. the benchmark is now available through hugging face, with code and tasks for others to run.
across 121,680 valid trials, 79,853 attempts failed executable checks. of those failures, 67.24% still ended cleanly and used a state-changing tool, yet 77.61% had wrong field values, 43.30% had unintended extra effects, and 25.36% missed required effects. only three models kept most of their single-attempt score over 20 repeats: gpt-6 astra retained 78%, claude opus 5.5 and claude opus 5 each retained 71%. kimi-k3 solved the most tasks at least once (93.89%) but passed all 20 attempts on only 13.41% of tasks.
the benchmark also prices consistency. gpt-5.4 was cheapest per dependable task at $6.80, but only 128 tasks passed all 20 runs. claude opus 5.5 and gpt-6 astra reached 241 and 231 dependable tasks at $7.80 and $7.45 respectively. failure analysis shows 79.9% of failures are tool handling issues, not reasoning errors. this suggests retry and error recovery matter more than raw model intelligence for reliable agents.
why it matters: for ai and data science teams deploying agents that touch real records, this benchmark shows single-run accuracy is misleading and consistency plus backend state checks are essential.
source: Hugging Face Blog: The Agent Said It Was Done. The Database Disagreed.