source: arxiv machine learning: dataprep-bench: benchmarking llms as training data preparators
level: research
dataprep-bench is the first unified benchmark for evaluating large language models as training data preparators. it covers two main tasks: data construction, where models turn raw sources into supervised training data, and data quality evaluation, where models predict how useful a dataset will be for downstream training. the benchmark uses a shared protocol across six domains and multiple base models. for construction, methods get the same raw sources and are scored by fine-tuning a base model on their outputs together with dolly-15k.
the benchmark includes 20 data construction methods and 15 data quality evaluation methods. it tests both single-turn and multi-turn agent workflows. results show that llm-based data construction often beats human-curated data, but only when the raw sources are rich. simple filtering and rewriting can match or exceed complex agent pipelines. for data quality evaluation, llm judges that look at actual data points outperform perplexity-based metrics, but they still struggle with subtle quality differences.
the study finds that data construction and evaluation capabilities are not strongly correlated. a model good at building data may not be good at judging it. the benchmark also reveals that multi-agent collaboration helps more for construction than for evaluation. all code and data are open source. the work provides a practical way to compare data preparation methods and highlights gaps in current llm abilities.
why it matters: better training data preparation can improve model performance without scaling model size, making ai development more efficient.
source: arxiv machine learning: dataprep-bench: benchmarking llms as training data preparators