source: hugging face blog: native-speed vllm transformers modeling backend
level: technical
the transformers library has long been the standard for implementing machine learning models, with over 450 architectures. last year, it became a modeling backend for vllm, letting model authors run their code inside vllm without porting. now, that integration is as fast as custom vllm implementations. tests on qwen3 models—4b dense, 32b dense, and 235b mixture-of-experts—show the transformers backend meets or beats native throughput on the same hardware.
the speedup comes from dynamic runtime optimizations. the backend uses torch.fx to analyze the model graph and find patterns that can be fused. it then rewrites parts of the source code using abstract syntax trees. this enables fused operations like mergedcolumnparallellinear and qkvparallellinear, which map to optimized vllm kernels. the process also infers tensor and pipeline parallel plans automatically. the modified models remain compilable with torch.compile and cuda graphs, just like dedicated vllm implementations.
using the backend requires only a flag: --model-impl transformers. it works with standard parallelism options, so existing serving setups need no changes. models that use linear attention are not yet supported, and custom hub models may not work if they are not written compliantly. the same model code can be used for training and inference, removing the need for separate optimized ports. a detailed technical post on the optimization methods is planned.
why it matters: model developers can get production-grade inference speed without writing custom vllm code, saving time and reducing duplication.
source: hugging face blog: native-speed vllm transformers modeling backend