The inference Bottleneck: KV Cache

← Back to blog

Most of the AI storage conversation is about training: feeding GPUs with data, writing checkpoints, handling millions of small files. That matters. But as more AI spend shifts to inference, a different bottleneck decides how fast and how cheaply you can serve tokens. It isn't the data. It's the context.

Why inference keeps doing the same work twice

Every time an LLM reads a prompt, it builds a key-value (KV) cache: the model's working memory of the context so far. Reuse that cache and the model can skip straight to generating. Lose it and the model has to rebuild it from scratch, which is called prefill.

The problem is space. On a typical GPU, model weights take 60–70% of HBM. The KV cache gets the 20–30% that's left, shared by every concurrent user. A single 100K-token context needs roughly 8 GB of KV cache. RAG pipelines, coding agents and multi-turn chat re-read that context on every turn.

When HBM fills up, the cache is evicted. The next request pays for a full prefill again. Users wait longer for the first token, and the GPU spends its time recomputing instead of serving.

Give the KV cache more room, closer to the GPU

TuringData Cache Fabric runs as a container on the GPU servers you already have. It plugs into vLLM and SGLang with one connector setting and tiers the KV cache across three levels:

  1. GPU HBM for the hottest blocks
  2. DRAM to extend capacity on the node
  3. Local NVMe SSD, striped across several drives, for large and persistent capacity

When a request arrives with a prefix that's already cached, Cache Fabric loads it instead of recomputing it. The model only prefills the new part. Model output doesn't change.

What we measured

On a single node with 8x NVIDIA H200 running GLM-5.2-FP8 on vLLM 0.23.0, with a 100K-token context and 3 concurrent streams (GuideLLM 0.7.3):

Metric

Native vLLM

vLLM + Cache Fabric

Change

Time to first token

36.7 s

2.1 s

-94%

Time per output token

0.548 s

0.044 s

-92%

Token throughput

5,816 tok/s

67,833 tok/s

11x

When one node isn't enough

Node-local caching has limits. Capacity stops at the node. A prefix cached on node A is rebuilt on node B. And when an endpoint restarts or scales to zero, its cache is gone.

TuringData ContextCube adds a shared, persistent KV cache pool for the whole cluster, based on the NVIDIA CMX tiered-cache design. Each unit delivers 120 GB/s of KV bandwidth over RDMA through BlueField-3 DPUs, so host CPUs stay free for serving. Compared with using an all-flash storage cluster as the shared tier, it cuts TCO by 67% and power by 70%.

The takeaway

Storage for training keeps GPUs fed. Memory for inference keeps them from repeating themselves. If your inference costs are climbing with context length, the fastest token is the one you don't have to recompute.

Book a PoC with our team. We'll test on your own model, workload and hardware, with a target of time to first token at or under 2 s and at least 1.5x your baseline throughput.

Find out how much GPU utilization you're leaving on the table. Schedule a free 30-minute architecture review with our experts.