Imagine a neocloud provider onboarding three customers. The first wants to train a model on a large dataset. The second needs to save and restore checkpoints during a long-running job. The third is launching an inference service that repeatedly processes lengthy documents and conversation histories.
All three need GPUs. But once their workloads begin, they place very different demands on the infrastructure around those GPUs.
Training can slow when data arrives too late. Checkpoint writes can compete with other jobs. Inference can spend time rebuilding context that has already been computed.
For the provider, these issues become part of the customer experience—and part of the challenge of building a competitive AI cloud service.
The pressure is to turn infrastructure investment into a service customers value. Gartner’s October 2026 analysis identifies cost pressure and insufficient business differentiation as major challenges facing neocloud providers. Its outlook emphasises the need to earn sustainable returns from AI infrastructure and associated platforms.
The data layer is one area where operators can make a practical difference. TuringData helps address two distinct requirements: high performance storage for training and inference, and KV cache tiering and sharing for context reuse.
When training waits for data, address the path between storage and compute. Return to the first customer. Their dataset is available, and the job has its GPU allocation, but progress falls short of expectations. The operator needs to identify the constraint. Large sequential reads, small-file activity, metadata operations, network congestion and application preprocessing require different investigations.
NVIDIA’s DGX SuperPOD guidance makes this distinction: storage requirements vary with the model, dataset and workload. Multimodal training can place substantial demands on sustained I/O, while other workloads depend more heavily on local caching.
TuringData Platform provides software-defined, scale-out storage for both training and inference. Parallel I/O, distributed metadata and RDMA networking support different parts of the data-access path. Unified file and object capabilities help operators manage access and movement across storage environments.
For neocloud providers, the opportunity is to reduce storage-related waiting under representative customer load. That means evaluating the application’s progress alongside bandwidth, metadata performance and I/O latency.
TuringData Flash provides an integrated deployment option, combining TuringData Platform software with validated NVMe hardware and high-speed networking. Starting with three storage nodes, it offers a packaged path to deploying storage for training, checkpointing and inference data access.
Operators can therefore choose software deployment or an appliance according to their infrastructure strategy and operational requirements.
When checkpointing interrupts progress, improve the save-and-recovery workflow. The second customer’s concern is preserving work—and recovering it when needed.
Checkpoint operations can create substantial bursts of I/O. NVIDIA notes that checkpoint files can reach terabytes and that synchronous checkpoint writes can block training progress.
TuringData Platform and TuringData Flash provide storage capabilities for parallel checkpoint reads and writes. Where storage is the constraint, improving that path can shorten the storage portion of saving or restoring training state.
For a provider, the useful measures are durable save time, restore integrity and time until the application resumes training. Framework behavior, checkpoint strategy and concurrent workloads remain part of that evaluation.
As more customers arrive, shared storage becomes an operational challenge. A growing service must accommodate different data-access patterns and storage allocations without turning every request into a custom engineering project.
TuringData Platform provides directory-level QoS and quotas to help operators manage shared storage allocation. Its tiering capabilities can retain frequently accessed data on NVMe while moving colder data to object storage.
These controls give providers tools for shaping storage services as demand changes. Performance allocation, access control and tenant isolation should each be validated within the deployment’s wider architecture.
When inference repeatedly processes the same context, make reuse part of the service design. The third customer introduces a different problem.
Their application sends requests containing repeated instructions, document prefixes or conversation history. When compatible cached context is unavailable, the model may process those prefixes again before returning its first token.
Prefix caching allows matching prompt prefixes to reuse previously computed attention keys and values. The benefit targets prompt processing, or prefill; it does not directly remove the computation needed to generate each new output token.
TuringData Cache Fabric runs as software on GPU servers and manages KV cache across GPU HBM, host DRAM and local NVMe, with optional external shared storage.
For a neocloud offering managed inference, this provides a way to retain reusable KV data beyond GPU-resident cache capacity. When a matching prefix returns, cached blocks can be retrieved instead of recomputed.
The opportunity is particularly relevant to workloads with repeated, compatible context. Faster retrieval can reduce first-token delay and reduce repeated prefill work when it outweighs the overhead of moving the cache data—a condition NVIDIA also highlights in its KV offloading guidance.
When inference grows across servers, give reusable context a shared home. Local cache helps when useful context is available on the node handling the request. A larger serving cluster must also consider what happens when the next request reaches another node.
TuringData ContextCube provides a dedicated, shared and persistent KV cache appliance. Participating inference nodes retrieve eligible cached prefixes through RDMA, using an architecture powered by NVIDIA BlueField-3 DPUs.
This gives operators a way to extend context availability beyond individual GPU servers. ContextCube supports compatible software including TuringData Cache Fabric, LMCache and Mooncake, with inference engines such as vLLM and SGLang.
Its value depends on the actual serving architecture: matching cache representations, request routing, reuse frequency, transfer performance and tenant boundaries all matter.
TuringData Cache Fabric can begin with local tiers without shared storage. TuringData ContextCube does not require TuringData Platform. Operators can evaluate storage, local cache tiering and shared caching independently.
The three customers at the start of this story need different capabilities. TuringData gives the operator deployment choices for addressing their data requirements: storage that supports training and inference, local KV cache tiering, and a shared cache pool where cross-node reuse is useful.
For neocloud providers, TuringData's solutions address the data constraints that can stand between available GPU capacity and a better customer workload experience. And of course, these capabilities also apply across enterprise and research AI environments.
Share your pain points. Let’s identify the bottleneck, agree on what success looks like, and work out the solutions together.