Supermicro, IBM, and NVIDIA have teamed up to launch a new KV cache solution designed to boost AI inference performance. The joint offering combines Supermicro’s Petascale storage servers, IBM Storage Scale software, and NVIDIA’s Dynamo inference software and Spectrum-X Ethernet switches. Performance testing shows that the solution significantly reduces the Time-to-First-Token (TTFT) for long-context windows, a critical metric for AI workloads.
The KV cache solution helps enterprises improve GPU utilization, throughput, and infrastructure scalability. By storing previously computed key-value states in a persistent cache, the system avoids redundant computations, leading to faster and more consistent AI responses. Testing with a 120-billion-parameter LLM demonstrated up to 56x faster TTFT at 130,000-token prompts, with only a 0.377-second increase in TTFT as input length grew, compared to 29.55 seconds without the cache.
Charles Liang, president and CEO of Supermicro, emphasized the importance of removing bottlenecks to improve GPU efficiency as AI workloads expand from training to inference. The solution also enhances enterprise resilience by providing high availability of KV cache content and non-disruptive scaling of cache capacity. Supermicro, IBM, and NVIDIA jointly validated the architecture across high-concurrency and noisy-neighbor scenarios.
Sam Werner, GM of IBM Storage, highlighted the collaboration’s role in making enterprise AI inference more scalable, reliable, and cost-effective. Shruti Koparkar, NVIDIA AI Inference Product Lead, noted that the solution helps maximize GPU throughput for agentic AI and long-context applications. Supermicro’s accelerated deployment services can reduce solution deployment time from weeks to days.