Jensen Huang put it plainly at GTC Taipei: tokens are now profitable units of revenue. The real question is how you maximize them — and the answer isn't more GPUs.
Tokens are the New Profitable Unit of Revenue
At NVIDIA GTC Taipei, Jensen Huang made a declaration that reframed how the industry thinks about AI infrastructure: tokens are now profitable units of revenue, each token represents revenue, and AI companies want to build more tokens and more AI factories.
Every token generated is a unit of output delivered, a task completed, a dollar earned. If tokens are revenue, then the ability to produce more tokens per unit of hardware directly translates to more revenue per dollar of investment. The race is on — but it may not be the race you think it is.
More GPUs Do Not Necessarily Mean More Tokens
The instinctive response to "produce more tokens" is "buy more GPUs." More compute, more throughput, more revenue. Simple math.
But the math doesn't hold when you look at what's actually happening inside your AI factory. The real bottleneck isn't compute — it's memory. Specifically, it's KV cache. And until that problem is solved, adding GPUs is like expanding a restaurant's kitchen when the real problem is the dining room is full.
Agentic AI has Arrived — and Memory Is Becoming the Next AI Infrastructure Layer
The AI landscape has shifted decisively toward agentic workloads. Reasoning models, long-context inference, and AI-to-AI workflows are now production realities.
AI agents don't just answer questions — they plan, execute, and iterate over extended sessions, calling tools, reading files, and building up context with every step.
Jensen put his finger on exactly this challenge at GTC Taipei. AI agents, like humans, need both short-term working memory and long-term memory to function effectively. In transformer-based models, that memory is the KV (key-value) cache — the data structure that stores the model's attention states so it doesn't have to recompute them for every new token.
As agents work around the clock — autonomously processing tasks, calling APIs, reasoning across long chains of thought — their context windows grow rapidly. This causes the KV cache to swell at a breakneck pace, quickly suffocating the GPU’s high-bandwidth memory (HBM).
When HBM overflows, AI factories hit a wall:
- Out-of-Memory Crashes: Systems fail mid-task, killing autonomous workflows.
- Context Truncation: To avoid crashes, systems force-erase older context, rendering the Agent "amnesic" and erratic.
- Massive Compute Waste: Systems are forced to drop and recalculate the KV Cache repeatedly, locking GPUs in a costly, infinite loop of "Prefill" cycles.
The result is:
- Time to First Token (TTFT) spikes dramatically — users wait longer for responses to begin.
- Token throughput collapses — the number of tokens your system can generate per second falls sharply, directly reducing the revenue your AI factory can produce.
- GPU utilization becomes inefficient — expensive HBM is consumed managing cache state rather than performing computation. You're paying for GPUs to do storage work.
The ability to efficiently manage and orchestrate memory is becoming a key determinant of token production in AI factories.
Brute-force GPU scaling is expensive, inefficient, and addresses the symptom rather than the core bottleneck in the system.
Building a Unified Memory Layer with TuringData Cache Fabric
TuringData Cache Fabric is purpose-built KV cache platform for modern AI factories. Rather than constraining KV cache to the limits of GPU HBM, it builds a multi-tier memory hierarchy that extends KV cache across the full storage stack — and does so intelligently.
TuringData Cache Fabric creates a unified multi-tier memory architecture spanning GPU HBM, CPU DRAM, local NVMe SSDs and remote shared storage. Through intelligent KV Cache offloading, multi-tier cache management, predictive prefetching, and fast cache restoration, it helps maximize GPU memory efficiency while maintaining inference performance.
Its goal is not simply to store KV Cache, but to ensure that valuable GPU memory remains available for active computation.With a unified memory layer, organizations can:
- Support longer context windows
- Run more concurrent AI agents
- Improve GPU utilization
- Reduce TTFT
- Increase token throughput
Most importantly, these gains can often be achieved without proportionally increasing GPU investments.
Real-World Benchmark: Same GPUs, More Tokens, Better Revenue
In benchmark testing with real-world inference workloads, TuringData Cache Fabric delivered:
- 91% Reduction in TTFT
- 90% Reduction in TPOT (Time Per Output Token)
- 8× Increase in token throughput
As illustrated in the figure above, in a 400Gbps network environment, an 8-GPU server increased throughput from 14,269 tokens/s to 103,050 tokens/s with TuringData Cache Fabric enabled.
Using GPT-5.5's published output token pricing ($3 per million tokens) as a reference, this translates into approximately $23,000 more token-generating capacity per day, or over $8.4 million per year from the same GPU infrastructure operating continuously.
From GPU ROI to Token ROI
AI infrastructure is entering a new era. The defining question is no longer: “How many GPUs do we have?” Instead, it becomes: “How many tokens can each GPU generate?”
When tokens become units of revenue, token production efficiency becomes a core business metric. In the age of Agentic AI, revenue growth is determined not only by compute capacity, but also by the efficiency of the memory system supporting it.
The future competition among AI factories will not be defined solely by GPUs. It will also be defined by memory storage infrastructure.
This is the challenge TuringData Cache Fabric is built to solve: enabling AI factories to generate more tokens, unlock higher GPU efficiency, and maximize value from existing infrastructure investments.