source: pytorch blog: miles: a pytorch-native stack for large-scale llm rl post-training
level: technical
miles is radixark's open source framework for large-scale llm rl post-training. it composes sglang for high-throughput rollout generation, nvidia megatron-lm for scalable distributed training, ray for cluster orchestration, and pytorch as the common numerical layer. the core trainer is intentionally small, with user-supplied python modules handling rollout logic, reward computation, loss functions, sample filtering, and metrics. this design lets teams adapt the system to new algorithms and production constraints without forking the framework.
the framework addresses the distributed systems challenges of modern rl post-training. it coordinates rollout workers and trainers, keeps policies synchronized, and handles moe routing alignment across the rollout/training boundary. miles supports both disaggregated and colocated execution, uses ray actors for process placement and supervision, and provides fast weight synchronization over dedicated nccl/rdma channels. a fully asynchronous mode decouples rollout from training, allowing continuous sample streaming.
miles provides unified low-precision recipes (bf16, fp8, mxfp8, int4-qat) that span training and rollout, ensuring numerical consistency. it supports lora in both paths for parameter-efficient post-training. built-in fault tolerance, observability via ray and pytorch profiler, and broad model support (including deepseek-v4, kimi k2.5, glm-5, qwen3.5) make it suitable for frontier-scale runs. extension points for custom rollout, rewards, losses, and model specs enable diverse workflows from rlhf to agentic tasks.
why it matters: it simplifies building and scaling rl post-training for large language models, reducing engineering overhead for ai teams working on alignment and reasoning improvements.
source: pytorch blog: miles: a pytorch-native stack for large-scale llm rl post-training