source: arxiv machine learning: hierarchical global attention (hga)
level: technical
hierarchical global attention (hga) is a drop-in replacement for dense causal attention in pretrained long-context transformers. it keeps the original checkpoint parameters unchanged, including the query, key, value, and output projection matrices. no calibration parameters are added and no retraining is needed. when applied to qwen3-30b-a3b-instruct-2507-fp8 on a single rtx 5090 with 32gb of memory, the patched model runs at a 64k-token context length. token-level key and value storage is not possible on this hardware without hga.
hga uses a two-level routing approach. first, it retrieves relevant chunks using compact rope-aware summaries. then it refines the selection by routing only the most relevant groups before performing exact token-level attention. this hierarchical retrieval greatly reduces the number of fetched tokens while still performing exact attention over the retrieved set. unlike previous sparse-attention methods, hga does not approximate attention within the selected tokens.
the method works out of the box on supported models without any fine-tuning. it addresses the memory bottleneck of storing full key-value caches for long sequences. by preserving exact attention on a pruned set of tokens, hga maintains model quality while extending context length. this makes it practical for deploying large language models on consumer-grade hardware for tasks requiring long contexts.
why it matters: it allows running large language models with very long contexts on limited hardware without retraining, making long-context ai more accessible.
source: arxiv machine learning: hierarchical global attention (hga)