SIGMA-KV: Sign-bit Guided KV-Cache Selection for Efficient Long-Context Transformer Inference

As LLMs process increasingly longer contexts, KV- cache accesses grow linearly with sequence length, creating a major memory and bandwidth bottleneck. While prior KV-selection methods reduce memory traffic, they often require costly per-query scoring or produce irregular KV retrieval patterns that result in scattered memory accesses and reduced spatial locality. We introduce SIGMA-KV, a training-free hardware-software co-design for KV-cache selection that reduces KV traffic while reorganizing KV retrieval around contiguous memory regions, improving spatial locality and hardware efficiency.
SIGMA-KV leverages our empirical observation that attention scores are largely determined by a small set of consistently dominant query/key dimensions. Using offline profiling, SIGMA-KV identifies these dimensions and organizes KV entries into compact tag-based buckets. During inference, queries generate lightweight sign-bit tags that directly guide retrieval of relevant KV regions through Hamming-distance matching, eliminating expensive token-level scoring. Unlike prior methods, SIGMA-KV reorganizes the KV layout for contiguous memory accesses, significantly improving spatial locality and hardware efficiency. Across LLaMA and Mistral models, SIGMA-KV reduces KV-cache traffic by up to 93%, lowers KV cache transfer time by up to 14×, and reduces memory-access energy by∼90% with only marginal accuracy degradation.

M.Sc. student under the supervision of Prof. Uri Weiser and Prof. Freddy Gabbay.