Seminar: Graduate Seminar

ECE Women Community

SIGMA-KV: Sign-bit Guided KV-Cache Selection for Efficient Long-Context Transformer Inference

Date: November,23,2026 Start Time: 15:00 - 16:00
Location: 1061, Meyer Building
Add to:
Lecturer: Muhammad Khalaily

Large language models (LLMs) are becoming a dominant workload for modern computing systems, yet their long-context inference efficiency is increasingly constrained by the memory wall. During autoregressive decoding, the growing key–value (KV) cache places substantial pressure on memory bandwidth and storage capacity, making KV accesses a major contributor to decoding latency and energy consumption. Ex-isting KV-selection methods reduce memory traffic but often incur costly per-query scoring or additional storage overhead, limiting their overall efficiency. We introduce SIGMA-KV, a training-free hardware–software co-design that combines fast, query-dependent KV selection with substantial memory-traffic reduction and minimal storage overhead, while keeping the KV cache in GPU memory without CPU offloading. SIGMA-KV exploits the observation that a small set of consistently dominant query/key dimensions provides an effective basis for identifying relevant KV entries. Offline profiling identifies these dimensions, whose signs form compact tags that serve jointly as retrieval keys and KV-cache allocation keys. During decoding, lightweight Hamming-distance matching selects relevant tag-based buckets, eliminating per-key scoring and ranking. A dedicated microarchi- tecture accelerates tag generation, matching, and bucket retrieval, with token-selection cost bounded by the tag space rather than the number of cached tokens. Across LLaMA and Mistral models, SIGMA-KV reduces KV-cache traffic by up to 93%, matching or exceeding state-of-the-art traffic reductions while maintaining task accuracy close to full attention. Ramulator-based evaluation with HBM3 and HBM3e shows up to 2.3× lower memory- read latency and 2.8× lower memory-read energy than HashAt- tention. SIGMA-KV also outperforms DDR5-backed MagicPIG and achieves memory-read latency and energy comparable to a hypothetical MagicPIG configuration with all data resident in GPU memory. At 129K tokens, SIGMA-KV incurs less than 1% additional memory overhead, compared with approximately 4% for HashAttention and 117% for DDR5-backed MagicPIG.

M.Sc. student under the supervision of Prof. Uri Weiser and Prof. Freddy Gabbay.

 

All Seminars