signed symmetric quantization for few-bit integers

source: arxiv machine learning: signed symmetric quantization for few-bit integers

level: technical

standard symmetric quantization for signed integers fixes the scale to be positive, which assigns the extra negative representable value to the negative tail. this can force clipping of positive outliers, causing non-trivial error at few-bit precision. asymmetric quantization solves this by adding a zero point to shift the grid, but it incurs a runtime penalty. for example, on an amd epyc turin cpu, 4-bit symmetric formats use up to 9% less memory and achieve up to 2.45 times higher throughput than asymmetric ones.

signed symmetric quantization is a third option that keeps the runtime benefits of symmetric quantization. it allows the scale to be negative, flipping the grid so the extra representable value falls on the positive side. this matches the data range better without needing a zero point. the method works by simply negating the scale and the quantized values, which is mathematically equivalent to using a negative scale.

experiments show that signed symmetric quantization reduces error compared to standard symmetric quantization, especially at 4 bits and below. it performs similarly to asymmetric quantization in accuracy but without the speed and memory costs. the technique is easy to implement in existing inference engines by just allowing negative scales. it is particularly useful for large language model inference where low-bit quantization is critical for performance.

why it matters: it enables faster and more memory-efficient low-bit inference for large language models without sacrificing accuracy.

source: arxiv machine learning: signed symmetric quantization for few-bit integers