Characterizing KV-Cache Performance in Large Language Model Inference: Context Length, Attention Architecture, and GPU Memory Residency
Abstract
Large language models (LLMs) commonly employ key–value (KV) caching to avoid recomputing previous attention states during autoregressive inference. However, its measured performance benefit depends on workload characteristics and the memory environment. This study presents a controlled systems characterization of KV-cache performance across input context length, grouped-query attention configurations, and GPU-memory residency. Experiments use Qwen2.5-3B and SmolLM3-3B under verified fully resident conditions with fixed 128-token generation, prompt lengths from 128 to 2048 tokens, randomized cache-enabled/cache-disabled execution, GPU telemetry, and repeated paired validation. KV-cache speedup increases from 1.34× to 8.31× for Qwen2.5-3B and from 1.28× to 8.62× for SmolLM3-3B. At 2048 prompt tokens, 30 randomized paired trials confirm mean speedups of 8.10× and 8.51×, corresponding to runtime reductions of 87.65% and 88.25%, while physical GPU-memory utilization remains below 61%. Despite SmolLM3-3B requiring twice the theoretical KV-cache storage per token, both models show comparable context-dependent scaling in the fully resident regime. A separate Mistral-7B case study further demonstrates that memory pressure can substantially alter apparent speedup. These results establish KV-cache speedup as workload- and system-dependent and demonstrate that meaningful benchmarking requires reporting context length, cache configuration, model and cache footprints, physical…
Read the paper
Related studies
- Diarylethenes for Memories and Switches
- Long-Term Potentiation and Memory
- Social Memory Studies: From “Collective Memory” to the Historical Sociology of Mnemonic Practices
- The Cognitive Neuroscience of Working Memory
- Memory aging and brain maintenance
- Effector and Memory CTL Differentiation
Go deeper in Evidence Hub
Evidence Hub adds a study data sheet (design, sample size, evidence level, key numbers), an AI summary and a chat grounded in the paper's text.