source: Hugging Face Blog: AutoSynthData: Generating Training Data for Enterprise Agents
level: technical
servicenow coreai built autosynthdata to turn capability gaps into training data for enterprise agents. it evaluates a target model in an environment, identifies tasks it struggles with, and uses a stronger teacher to characterize solvable failures. the system then generates new tasks that exercise those weaknesses, validates them, and uses accepted samples for post-training. as the model improves, the curriculum shifts toward remaining difficulties.
in enterpriseops gym experiments, autosynthdata generated 2,000 synthetic training samples in about 18 hours for the hybrid domain. fine-tuning gemma-4-26b-a4b-it on this dataset improved mean pass@1 by 7.2 percentage points, a 35% relative gain, and raised verifier success from 63.01% to 68.55%. it closed 59% of the original pass@1 gap between gemma and the reference model. the generator did not receive original evaluation tasks, only sanitized capability specification cards.
the pipeline includes sample-level verification and repair, plus batch-level review to ensure coverage and diversity. each candidate must pass positive and negative verification gates, and a critic diagnoses failures for targeted repair. the approach treats synthetic data generation as a search for tasks near the model's capability boundary. the authors plan to test the moving difficulty-calibrated frontier beyond supervised fine-tuning, including reinforcement learning.
why it matters: this method lets enterprises create targeted training data from their own agent failures, improving model performance in specific environments without manual data collection.
source: Hugging Face Blog: AutoSynthData: Generating Training Data for Enterprise Agents