ratequant tunes kv cache precision for llms
a new method allocates different bit-widths to attention heads in kv cache quantization, avoiding distortion model mismatch to improve large language model serving efficiency.
ai newssummaries filed under machine learning
a new method allocates different bit-widths to attention heads in kv cache quantization, avoiding distortion model mismatch to improve large language model serving efficiency.
ai newsemo is a mixture-of-experts model trained so that experts self-organize into task-specific groups, allowing strong performance with only a small subset of experts.
ai news