level: research
conditional computation methods like mixture-of-experts, mixture-of-depths, and kv-cache quantization each target one part of a transformer. they are usually designed separately. this paper argues that attention resolution, expert choice, and cache precision are linked. a token that needs full attention might also need high-precision caching, no matter which expert handles it. treating these decisions together could avoid wasted computation.
the proposed system, triroute, uses one small controller shared across all layers. for every token at every layer, it outputs three things at once: an attention mode (skip, local, or full), a sparse set of feedforward experts (including a null expert that skips the block), and a kv-cache bit-width. the controller is trained end-to-end with a gumbel-softmax relaxation that handles the mix of discrete choices. this lets the model learn a joint policy that balances quality and cost.
experiments show triroute matches or beats standalone methods while using fewer total resources. the joint routing finds efficient combinations that separate controllers miss. for example, it can assign full attention and high cache precision to rare tokens while using cheaper settings for common ones. the approach works across different model sizes and tasks, suggesting that unified routing is a practical way to make large language models more efficient at inference time.
why it matters: unified routing can lower the cost of running large language models by making smarter per-token compute decisions, which matters for deploying ai at scale.