Most of the AI storage conversation is about training: feeding GPUs with data, writing checkpoints, handling millions of small files. That matters. But as more AI spend shifts to inference, a different bottleneck decides how fast and how cheaply you can serve tokens. It isn't the data. It's the context.
Why inference keeps doing the same work twice
Every time an LLM reads a prompt, it builds a key-value (KV) cache: the model's working memory of the context so far. Reuse that cache and the model can skip straight to generating. Lose it and the model has to rebuild it from scratch, which is called prefill.
The problem is space. On a typical GPU, model weights take 60–70% of HBM. The KV cache gets the 20–30% that's left, shared by every concurrent user. A single 100K-token context needs roughly 8 GB of KV cache. RAG pipelines, coding agents and multi-turn chat re-read that context on every turn.
When HBM fills up, the cache is evicted. The next request pays for a full prefill again. Users wait longer for the first token, and the GPU spends its time recomputing instead of serving.
Give the KV cache more room, closer to the GPU
TuringData Cache Fabric runs as a container on the GPU servers you already have. It plugs into vLLM and SGLang with one connector setting and tiers the KV cache across three levels:
- GPU HBM for the hottest blocks
- DRAM to extend capacity on the node
- Local NVMe SSD, striped across several drives, for large and persistent capacity
When a request arrives with a prefix that's already cached, Cache Fabric loads it instead of recomputing it. The model only prefills the new part. Model output doesn't change.
What we measured
On a single node with 8x NVIDIA H200 running GLM-5.2-FP8 on vLLM 0.23.0, with a 100K-token context and 3 concurrent streams (GuideLLM 0.7.3):
Metric | Native vLLM | vLLM + Cache Fabric | Change |
|---|---|---|---|
Time to first token | 36.7 s | 2.1 s | -94% |
Time per output token | 0.548 s | 0.044 s | -92% |
Token throughput | 5,816 tok/s | 67,833 tok/s | 11x |
When one node isn't enough
Node-local caching has limits. Capacity stops at the node. A prefix cached on node A is rebuilt on node B. And when an endpoint restarts or scales to zero, its cache is gone.
TuringData ContextCube adds a shared, persistent KV cache pool for the whole cluster, based on the NVIDIA CMX tiered-cache design. Each unit delivers 120 GB/s of KV bandwidth over RDMA through BlueField-3 DPUs, so host CPUs stay free for serving. Compared with using an all-flash storage cluster as the shared tier, it cuts TCO by 67% and power by 70%.
The takeaway
Storage for training keeps GPUs fed. Memory for inference keeps them from repeating themselves. If your inference costs are climbing with context length, the fastest token is the one you don't have to recompute.
Book a PoC with our team. We'll test on your own model, workload and hardware, with a target of time to first token at or under 2 s and at least 1.5x your baseline throughput.