Topology-Aware Transfers Speed Up Disaggregated GPU Inference

source: arXiv Machine Learning: Topology-Aware Data Movement for Disaggregated GPU Inference

level: technical

Disaggregated large language model inference splits prefill and decode across separate GPU pools, requiring KV cache transfers of 2.6 GB per request for a 70B model. Existing systems like DistServe, Splitwise, and Mooncake use uniform RDMA, ignoring that inter-GPU bandwidth varies by up to 72 times depending on physical location. A new topology-aware orchestrator discovers the interconnect hierarchy at startup and selects the best transport for each transfer.

The orchestrator employs three mechanisms: pipelined layer-by-layer transfer overlapping transmission with ongoing prefill, hiding 60 to 85 percent of latency; NVLink domain-aware placement for Mixture-of-Experts models; and adaptive routing that avoids congested links. At production scale, aggregate bandwidth demands exceed 100 GB/s, making naive uniform RDMA impractical. The system leverages NVLink’s 900 GB/s within a domain, InfiniBand’s 50 GB/s across nodes, and TCP’s 12.5 GB/s across data centers.

Disaggregated inference improves resource utilization by letting prefill and decode scale independently, but the KV cache transfer bottleneck has limited adoption. Prior work focused on scheduling or compression, not on exploiting physical topology. This approach reduces tail latency and increases throughput without model changes. It is especially relevant for large-scale deployments where network topology is heterogeneous and bandwidth is a constrained resource, directly improving cost efficiency and user experience.

why it matters: Efficient KV cache transfer is critical for cost-effective, low-latency large model serving in disaggregated GPU clusters, directly impacting cloud AI service scalability.

source: arXiv Machine Learning: Topology-Aware Data Movement for Disaggregated GPU Inference