Nvidia Shifts Focus to Storage as Key to Unlocking AI Performance

Nvidia is shifting its focus to storage as the next frontier for AI performance. The company argues that future gains will depend not only on faster processors but also on how efficiently systems can deliver data to GPUs.

Nvidia's efforts build on technology that has been in development for several years, including cuFile and SCADA. CuFile removes the CPU from the main data path, allowing data to move directly between storage and GPU memory using Direct Memory Access. However, with smaller transfers, the fixed cost per request becomes significant, potentially overwhelming the CPU even when underlying storage hardware still has capacity available.

SCADA addresses this limitation by shifting more of the storage-control process to the GPU. The GPU can construct and complete storage requests itself, allowing it to keep a large number of operations active simultaneously. Nvidia sees cuFile as useful for large data transfers, while SCADA is intended to improve performance when AI workloads require many small reads.

The need for this approach is linked to the way modern AI applications access data. Enterprise SSDs have traditionally been optimized around 4KB random reads, but AI inference can operate on much smaller pieces of information, including embeddings that may be only a few hundred bytes and key-value cache blocks that can remain below 1KB.