level: research
ai agents can now handle short, clear tasks, but most terminal benchmarks only check final results and ignore partial progress. this gives sparse feedback and misses how agents work through problems. the new long-horizon-terminal-bench addresses this by offering 46 tasks across nine areas like experiment reproduction, software engineering, multimodal analysis, interactive games, and scientific computing. each task has a reference solution or simulation engine and is split into smaller graded subtasks.
the benchmark provides dense intermediate rewards and partial credit. this means evaluation captures not just whether an agent finishes, but how far it gets and what steps it completes. the setup builds on the terminal-bench style but extends it for tasks that can take much longer than a few minutes. by breaking tasks into fine-grained pieces, the benchmark gives a clearer picture of agent capabilities over extended periods.
the tasks cover diverse domains, pushing agents to maintain performance over long horizons. the graded subtask structure allows researchers to see where agents succeed or fail in detail. this helps identify strengths and weaknesses in planning, tool use, and sustained reasoning. the benchmark aims to move evaluation beyond simple pass/fail metrics, offering a richer signal for developing more capable and reliable ai agents.
why it matters: it provides a way to measure ai agent progress on complex, multi-step tasks, helping developers build systems that can handle real-world long-running jobs.