source: arxiv artificial intelligence: data and evaluation closed-loop for model capability enhancement
level: technical
large language model pre-training relies on data to shape capabilities and evaluation to measure them, but the two processes are disconnected. data is fed in prospectively, while evaluation scores come out retrospectively, mixing samples, prompts, and scoring rules into a single noisy number. when a model fails, engineers must guess which data to change, because benchmarks and per-sample correctness do not map directly to data sources, domains, or quality labels. this gap forces intuition-based fixes instead of systematic improvement.
the paper introduces the capability slice, a group of evaluation samples that share background condition, task type, solving operation, and output constraint. this unit is precise enough to pinpoint a specific weakness but stable enough to survive aggregation, unlike a coarse benchmark name or a noisy single sample. around this slice, the authors build an evaluation taxonomy and a data taxonomy, then link them through a closed-loop process. when a slice underperforms, the system traces back to the responsible data subsets and suggests targeted fixes.
the approach turns model improvement from a reactive, ad-hoc task into a structured feedback loop. by connecting evaluation signals directly to data adjustments, it reduces reliance on engineer intuition and speeds up pre-training cycles. the method is designed to work with existing training pipelines and can be applied to various model scales and domains. it offers a practical way to make capability enhancement more data-driven and less error-prone.
why it matters: this gives ai engineers a systematic way to fix model weaknesses by linking test failures directly to training data changes, reducing guesswork in pre-training.
source: arxiv artificial intelligence: data and evaluation closed-loop for model capability enhancement