Seminar: Graduate Seminar

ECE Women Community

HERA: A Hardware-Software Engine for Ratio-Outlier Aware Quantization for KV Cache

Date: November,23,2026 Start Time: 10:00 - 11:00
Location: 506, New Zisapel Building
Add to:
Lecturer: Maayan Ella

The aggressive scaling of Large Language Models (LLMs) has shifted the primary inference bottleneck from computation to the memory bandwidth and capacity of the KV cache, creating a severe “memory wall” that limits throughput and drives high energy consumption. While Post-Training Quantization (PTQ) offers a direct remedy, aggressive uniform quantization (INT4 and below) leads to severe accuracy degradation due to emergent activation outliers that are critical to the attention mechanism, while existing alternatives suffer from hardware inefficiencies, runtime overheads, or permanent context loss. To resolve this tension, we introduce HERA—a hardware-software co-designed framework that dynamically identifies and preserves top-magnitude outliers within fixed-size data tiles at runtime, drastically reducing the dynamic range and enabling low-bit uniform quantization of the remaining data with minimal reconstruction error. HERA’s software phase profiles value distributions offline and uses a greedy optimization algorithm to assign a customized outlier preservation ratio to each layer and attention head individually, enabling the system to meet a user-defined global compression target without requiring model retraining.
At runtime, the HERA hardware engine operates as a highly efficient, pipelined, fixed-function block directly on the critical memory path (between the attention compute units and the HBM controller), consisting of three specialized units: a tournament-tree comparator for dynamic outlier extraction during writes, a multi-lane binary search engine for fast quantization mapping, and a single-pass value restoration unit during reads. Evaluated on Llama-2 and Llama-3 architectures (from 7B to 13B parameters) across diverse zero-shot benchmarks and synthesized in a commercial 16nm FinFET process, HERA successfully compressed the KV cache to 50%, 30%, and 24% of its original size. This reduction yielded corresponding energy savings of 48%, 68%, and 74%, while incurring negligible average accuracy drops of only 0.24%, 1.57%, and 3.59% respectively, with negligible silicon area overhead. HERA thus offers a practical and deployable solution that allows production systems to dynamically scale batch sizes and context lengths within existing hardware constraints.

M.Sc. student under the supervision of Prof. Uri Weiser.

 

All Seminars