Every LLM response starts with a wait. The model reads your whole prompt before it writes a single word, and on long prompts that wait dominates the user experience. The KV cache is what makes the rest of generation fast, and managing it well is what makes the first token fast too.
This guide walks through the three ideas in order: what KV cache is, how KV cache orchestration works, and where KV cache should live.
What is KV cache?
KV cache (key-value cache) is the stored set of attention keys and values a transformer computes for every token in a prompt. It lets the model generate each new token without reprocessing everything that came before.
Prefill and decode: the two phases of LLM inference
Every request to an LLM runs in two phases:
- Prefill (read the prompt, build KV). The model processes all prompt tokens in parallel. For each token and each layer, it computes a key vector and a value vector and writes them to the KV cache. Prefill is compute-heavy.
- Decode (write the answer). The model generates one token at a time. Each step computes keys and values only for the newest token and reads all earlier ones from the KV cache. Decode is memory-bandwidth-heavy.
Without a KV cache, each decode step would recompute attention for the entire sequence so far. With it, that work is done once in prefill and reused at every step.
KV cache and time to first token (TTFT)
Time to first token (TTFT) is the delay between sending a request and receiving the first output token. For long prompts, TTFT is dominated by prefill, because the model must build the KV for every prompt token before decode can start.
That is where reuse pays off. Many requests share a long common prefix: a system prompt, a retrieved document, an agent's tool definitions, or the earlier turns of a chat. If the KV for that prefix is already cached, the request can skip most of prefill:
Request type | What happens before the first token | TTFT |
|---|---|---|
Cold request (no cache) | Prefill recomputes KV for the whole prompt | Longest |
Warm request (prefix hit) | Load the cached prefix KV, then prefill only the new suffix | Shorter; decode starts earlier |
This is called prefix caching: matching the beginning of a new prompt against KV that was computed earlier, and reusing it.
Open-source serving engines already build this in. The vLLM documentation on Automatic Prefix Caching describes it as caching "the KV cache of existing queries, so that a new query can directly reuse the KV cache if it shares the same prefix," and names two typical workloads: repeated questions over one long document, and multi-round conversations.
Why KV cache matters
KV cache is both the biggest lever on LLM latency and one of the biggest consumers of GPU memory.
1. Reused KV turns expensive GPU compute into cheap IO. Because TTFT is dominated by prefill, every reused KV block is prefill work the GPU does not repeat. The result is lower latency and more concurrent streams per GPU.
2. KV grows with context length and concurrency, and quickly outgrows GPU HBM. KV cache size scales linearly with the number of tokens held and the number of requests in flight:
\text{KV bytes per token} = 2 \times \text{layers} \times \text{KV heads} \times \text{head dim} \times \text{bytes per value}Take Llama 3 70B. Meta's Llama 3 paper lists 80 layers, a model dimension of 8,192, 64 attention heads and 8 key/value heads, which gives a head dimension of 128. At FP16, that is about 320 KB of KV per token, so one 128K-token context needs roughly 40 GB of KV before counting any other user. (Our arithmetic; exact sizes depend on precision and serving engine.)
GPU high-bandwidth memory (HBM) also has to hold the model weights, so it cannot keep every useful prefix for every user. The cache has to extend beyond the GPU.
How KV memory is managed directly limits throughput. The PagedAttention paper (Kwon et al., SOSP 2023), which introduced vLLM, found that KV memory "can be significantly wasted by fragmentation and redundant duplication," limiting batch size; managing it in pages improved throughput 2–4× at the same latency.
What is KV cache orchestration?
KV cache orchestration is traffic control for the model's working memory. It decides which KV to keep, moves KV between GPU memory and lower-cost memory and storage tiers, and hands KV from prefill to decode. It does three jobs.
1. Offload and load
Offloading moves KV out of GPU memory before it is evicted; loading brings it back when a later request hits the same prefix. KV moves across a tiered hierarchy:
Tier | Medium | Where it sits |
|---|---|---|
G1 | GPU HBM | On the GPU |
G2 | DRAM | Local to the server |
G3 | Local NVMe SSD | Local to the server |
G3.5 | KV appliance (a dedicated KV cache pool) | External, shared |
G4 | Shared storage | External, shared |
Each step down holds more KV at lower cost per byte, and takes longer to reach.
The G1–G4 naming matches the NVIDIA Dynamo KV Block Manager (KVBM) design, which defines G1 as a GPU-resident pool, G2 as CPU pinned memory, G3 as local NVMe SSD and G4 as remote or cloud storage. KVBM "offloads" blocks down this hierarchy and "onboards" them back to the GPU. G3.5 is TuringData's dedicated, shared KV tier between local disk and general storage.
2. Transfer from prefill to decode (P→D)
In disaggregated serving, prefill and decode run on separate servers so each can be sized for its own bottleneck. The decode server needs the KV that prefill built, so orchestration hands it across, via a GPU interconnect or the network.
The case for splitting the phases is well documented. DistServe (Zhong et al., OSDI 2024) showed that colocating prefill and decode causes "strong prefill-decoding interferences." By assigning them to different GPUs, it served up to 7.4× more requests, or met 12.6× tighter latency targets, than the systems it was compared against. The trade-off is that KV must now move between machines.
3. Optimization: the policy behind every move
Every offload, load and transfer answers three questions:
- Which KV to keep
- Where it lives
- How long it stays
Why KV cache orchestration matters
- Eviction can mean recomputation. If needed KV is evicted and no reusable copy remains, the system must recompute the missing context. Offloading preserves it for potential reuse.
- KV transfer can delay generation. In disaggregated serving, decode cannot start without the relevant KV state. Transfer often adds to TTFT; how much depends on the serving and streaming design.
Where should KV cache live?
Keep active KV close to compute, and keep reusable context within reach. KV cache can always be rebuilt, but rebuilding is not free: it costs a full prefill. That gives KV storage three jobs.
1. Keep active KV close
KV for requests that are generating right now stays in GPU HBM (G1). Each decode step reads it to produce the next token, so it must sit in the fastest memory available.
2. Retain reusable KV
KV that may be needed again (a shared system prompt, a popular document, an idle chat session) does not need to occupy HBM. It can be retained across memory and storage tiers:
- Local tiers: host DRAM (G2) and local NVMe (G3) on the same server.
- Shared tier: a KV cache pool shared across GPU nodes (G3.5), such as TuringData ContextCube.
3. Retrieve when worthwhile
When a new request matches a cached prefix, load the KV instead of re-running prefill:
- Prefix hit: load KV, prefill the suffix only, and decode starts earlier.
- Prefix miss: prefill processes prefix and suffix from scratch.
Retrieval is worthwhile when loading the KV is faster than recomputing it, which becomes more likely as prompts get longer.
Why KV placement matters
- Storage keeps reusable KV alive. When GPU HBM fills, reusable KV may be evicted unless it is retained in another tier. If it is, a later request can load it instead of running prefill again.
- A shared pool multiplies reuse. With a local cache, a prefix computed on one node only helps that node. With a shared pool, participating GPU nodes can reuse compatible cached KV from another node, reducing repeated prefill across the cluster.
How TuringData helps
TuringData builds the layer that keeps KV cache reusable beyond a single GPU.
- TuringData Cache Fabric orchestrates KV across tiers, so warm requests hit a cached prefix, load the KV and prefill only the suffix.
- TuringData ContextCube is a G3.5 shared KV cache pool across GPU nodes. It retains reusable KV outside HBM and makes it available to every participating node in the cluster.
Together they target the two costs this guide describes: repeated prefill, which inflates TTFT, and HBM capacity, which limits how much context and concurrency each GPU can serve.
Glossary
Term | Definition |
|---|---|
KV cache | Stored attention keys and values for tokens already processed |
Prefill | Inference phase that reads the prompt and builds KV |
Decode | Inference phase that generates output one token at a time |
TTFT | Time to first token: delay before the first output token appears |
Prefix caching | Reusing cached KV for the matching start of a new prompt |
HBM | High-bandwidth memory on the GPU (tier G1) |
Offloading | Moving KV from GPU memory to a lower tier |
Disaggregated serving | Running prefill and decode on separate servers |
KV cache pool | Shared store of reusable KV across GPU nodes (tier G3.5) |
Sources
- Kwon et al., Efficient Memory Management for Large Language Model Serving with PagedAttention, SOSP 2023
- Zhong et al., DistServe: Disaggregating Prefill and Decoding for Goodput-optimized LLM Serving, OSDI 2024
- Llama Team, Meta, The Llama 3 Herd of Models, 2024
- vLLM, Automatic Prefix Caching
- NVIDIA Dynamo, KVBM Design