How do we feed the cluster?
Storage must deliver data in the pattern the application reads it. A large dataset and a high headline throughput number do not tell the whole story.
The checkpoint burst is the hard part
Reads come in different shapes
Millions of small files can create opening, lookup, and authorization work. A benchmark reading a few large files may miss that cost.
Caching and prefetching can help. Test an empty cache as well as a warm one, and keep durable data separate from disposable cached copies.
Writes can arrive in bursts
A checkpoint saves progress so a job can resume. Writing a file is not enough; the application must restore a consistent checkpoint successfully.
Sixteen fictional servers each writing 8 GB produce 128 GB together. Their average demand may look small while a synchronized burst is large.
Supported direct paths such as GPUDirect Storage still require compatible hardware, software, and storage.
Try it yourself
Those servers checkpoint every ten minutes. Calculate the average rate and the rate needed to finish the burst in ten seconds.
Check your answer
About 0.213 GB/s average, versus 12.8 GB/s during the burst, excluding overhead. Provisioning from the average alone misses the difficult part.