source: hugging face blog: model routing is simple. until it isn’t.
level: technical
building a router into an ai agent seems straightforward: send easy tasks to cheap models and hard ones to expensive models. but in practice, model selection is not just a classification problem. it becomes a systems optimization challenge. three factors make it surprisingly difficult: real costs go beyond sticker prices, task complexity is not always visible upfront, and latency depends on infrastructure, not just model speed.
cost is more than token pricing. in tests with 417 tasks, gpt-4.1 cost nearly double claude sonnet despite lower per-token rates. the reason was caching. agent workloads reuse context heavily, and sonnet's lower cache-read pricing gave it a big advantage. a router that only looks at pricing sheets will make wrong decisions. actual cost depends on how the model, workload, and serving infrastructure interact.
complexity and latency also defy simple rules. a task that seems easy can trigger hidden steps like retrieval or compliance checks. a router must juggle cost, quality, latency, and governance rules at once. latency is not just about model size; routing overhead, hardware, and cache warmth matter more. routing at every step adds flexibility but also delay. treating routing as an optimization problem—balancing cost, accuracy, and latency—yielded better results. a lightweight optimizer found operating points that cut cost by 21% and latency by 9% with only a 4% accuracy drop, outperforming a difficulty-based router.
why it matters: ignoring caching, infrastructure, and multi-objective tradeoffs leads to routers that waste money and slow down systems.
source: hugging face blog: model routing is simple. until it isn’t.