KV-Cache in LLM Inference

The key-value (KV) cache stores attention keys and values already computed for earlier tokens. During autoregressive decoding, it avoids recomputing those projections at every step—trading compute for accelerator memory and memory bandwidth.

Quick reference

Question	Answer
What is stored?	key and value vectors for cached tokens, per layer
What is not stored?	a permanent query vector for every past token
What grows memory?	live tokens, concurrent sequences, layers, KV heads, head dimension, and bytes per element
What improves reuse?	prefix caching when identical eligible prefixes can safely share blocks
What releases memory?	request completion, eviction, cancellation, or cache-policy action

The problem it solves

Transformers use attention: every output token attends to every previous token. Without caching, generating a 1000-token response requires recomputing attention for tokens 1–999 when generating token 1000.

Without KV-cache:
  Token 1:   1 attention computation
  Token 2:   2 attention computations
  Token N:   N attention computations
  Total:     O(N²) computations ← quadratic!

With KV-cache during token-by-token decode:
  Token 1:   1 computation → store K,V in cache
  Token 2:   1 computation → append K,V to cache
  Token N:   1 computation → append K,V to cache
  Each step reuses prior K,V projections; attention still reads and scores the cached sequence.

The cache removes repeated K/V projection work for old tokens. It does not make attention independent of context length; decode cost and memory traffic still grow as the cached sequence grows.

What gets cached

For each transformer layer, every token produces three vectors: Query (Q), Key (K), and Value (V). During generation, Q changes every step (new token), but K and V for previous tokens are static — they never change.

import torch

def attention_with_kv_cache(query, key_cache, value_cache, new_key, new_value):
    """
    query:       [1, d_head]      ← current token only
    key_cache:   [seq_len, d_head] ← all previous K vectors
    value_cache: [seq_len, d_head] ← all previous V vectors
    """
    # Append new K, V to cache
    key_cache   = torch.cat([key_cache,   new_key.unsqueeze(0)],   dim=0)
    value_cache = torch.cat([value_cache, new_value.unsqueeze(0)], dim=0)

    # Attention over full cached sequence
    scores = torch.matmul(query, key_cache.T)          # [1, seq_len]
    scores = scores / (query.shape[-1] ** 0.5)
    weights = torch.softmax(scores, dim=-1)
    output  = torch.matmul(weights, value_cache)       # [1, d_head]

    return output, key_cache, value_cache

Memory cost of KV-cache

This is where storage becomes critical. KV-cache size for a single request:

def kv_cache_bytes(num_layers, num_heads, head_dim, seq_len, dtype_bytes=2):
    """
    num_layers:  e.g. 32 (Llama-2-7B has 32 layers)
    num_heads:   e.g. 32
    head_dim:    e.g. 128
    seq_len:     max context length, e.g. 4096
    dtype_bytes: 2 for float16/bfloat16
    """
    per_token = num_layers * num_heads * head_dim * dtype_bytes * 2  # K + V
    total     = per_token * seq_len
    return total

# Llama-2-7B, 4096 token context, bfloat16
size = kv_cache_bytes(32, 32, 128, 4096, 2)
print(f"KV-cache per request: {size / (1024 ** 3):.2f} GiB")
# Output: KV-cache per request: 2.00 GiB

This cache alone consumes 2 GiB per fully occupied request. Real concurrency must also leave room for weights, activations, runtime workspaces, allocator fragmentation, and operational reserve.

Paged KV-cache management

Paged cache managers divide KV memory into blocks and map a sequence to non-contiguous blocks. This reduces waste from reserving one maximum-size contiguous region per request and makes block reuse, eviction, and scheduling more flexible.

Traditional KV-cache:           PagedAttention (vLLM):
┌────────────────────┐          ┌──────┐ ┌──────┐ ┌──────┐
│ Request A (2048 tok)│          │Page 1│ │Page 3│ │Page 5│  ← Request A
│ [pre-allocated]    │          └──────┘ └──────┘ └──────┘
│ [wasted if shorter]│          ┌──────┐ ┌──────┐
├────────────────────┤          │Page 2│ │Page 4│           ← Request B
│ Request B (512 tok) │          └──────┘ └──────┘
│ [padded to max]    │
└────────────────────┘          Pages allocated on demand — no waste

Real throughput gains depend on model architecture, sequence distribution, batching, kernel choice, cache precision, and hardware. Benchmark the target workload rather than treating a published speedup as universal.

Lifecycle and pressure

prefill prompt → allocate/write KV blocks
decode token   → read prior KV + append one token
batch changes  → scheduler remaps active sequences
request ends   → release blocks

Long prompts make prefill compute-heavy; large live-token populations make decode memory capacity and bandwidth critical. Continuous batching improves utilization but means the peak aggregate live-token count—not one request’s context limit—is the key capacity variable.

Operational signals

• KV blocks used/free and allocation failures;
• active, queued, and preempted sequences;
• aggregate live tokens and prefix-cache hit rate;
• prefill time, inter-token latency, and tokens per second;
• cache eviction/recompute rate;
• GPU memory headroom and memory-bandwidth utilization.

Failure and design questions

• Does tensor parallelism shard KV heads for this architecture and runtime?
• Is cache quantization acceptable for quality and kernel support?
• Can prefixes be shared without crossing tenant or authorization boundaries?
• What happens when the block pool is exhausted: queue, preempt, recompute, or reject?
• Are reported limits based on configured context, reserved tokens, or actual live tokens?

Key takeaways

Concept	Detail
What’s cached	K and V vectors per layer, per token
Memory grows	Linearly with sequence length
GPU memory limit	Defines max concurrent requests
vLLM innovation	Paged KV-cache — eliminates fragmentation
Quantization effect	fewer bytes per element; concurrency gain is workload- and runtime-dependent

Use the GPU Memory Planner for a full serving budget and the StorageCraft CLI for a transparent KV baseline.