source: hugging face blog: scarfbench: benchmarking ai agents for enterprise java framework migration
level: technical
scarfbench is an open benchmark for testing ai agents on cross-framework migration tasks in enterprise java. it covers migrations across spring, jakarta ee, and quarkus. unlike traditional benchmarks that compare generated code to reference implementations, scarfbench checks whether migrated applications actually build, deploy, and preserve behavior. the benchmark includes 34 applications, 102 framework implementations, and 204 migration tasks, with about 151,000 lines of code and over 1,300 expert-written tests.
evaluations of frontier coding agents show that framework migration remains difficult. even the strongest agents achieve less than 10% behavioral success. compile success is higher than deploy success, which is higher than behavioral success, so build success alone overestimates migration quality. agents are often overconfident: one agent reported successful builds for 29 out of 30 applications, but only 22 actually built. migration difficulty varies by target framework, with jakarta ee proving especially challenging.
analysis reveals that migration is iterative, with agents frequently revisiting configuration, web, database, and service layers. configuration dominates migration effort. agents also struggle with environmental issues like docker cache problems, port connectivity, and maven tooling. the main challenge is not translating java code but managing dependencies across configuration, infrastructure, and runtime environments. scarfbench provides a standardized way to measure progress toward autonomous application modernization.
why it matters: it shows that current ai agents cannot reliably modernize enterprise applications, highlighting the need for better validation and architectural reasoning in ai-assisted software engineering.
source: hugging face blog: scarfbench: benchmarking ai agents for enterprise java framework migration