GPUs are expensive. A GPU sitting idle because it is waiting for data is expensive hardware doing nothing.
That is why AI teams ask for fast object storage.
But “fast” does not mean one large number on a storage benchmark. It means the next batch of data reaches the GPU before the GPU needs it.
The normal path looks like this:
Object storage → network → CPU memory or local NVMe → GPU memory
The GPU is at the end of that chain. Any slow part can make it wait.
Fast means many GPUs can read at once
One server reading one large object is an easy test. A real training job is different.
Hundreds or thousands of GPUs may start at the same time. They all need training data. They all load model files. They may all restore a checkpoint after a failure.
The question is not, “How fast can this one client read?”
It is, “Can the whole cluster keep reading without leaving GPUs idle?”
The first byte matters
Throughput is how quickly data moves after a request starts. Time to first byte is how long the request waits before data starts moving.
Both matter.
Slow first-byte time hurts when a job starts or opens a new file. It hurts even more when thousands of workers do it together. Storage may look healthy overall, but the job can still spend too much time waiting for requests to begin.
This is why connection reuse, retries, parallel requests, and keeping compute close to storage matter. Small delays add up quickly at GPU scale.
Small files create a big problem
Imagine a training set with millions of images. Each image is a separate object.
The GPU does not care that each image is small. The data loader still has to open it, make a request, wait for a response, and decode it. Do that millions of times and the overhead becomes the problem.
The storage service may be working fine. The GPU may be working fine. The training job is still slow because the path between them is doing too much work.
The practical fix is to put many samples into larger files called shards, then read those files in order. AWS recommends shard sizes roughly between 100 MB and 1 GB for many training workloads.
This is not a storage trick. It is a way to stop wasting GPU time.
Cache data near the GPU
Object storage is still the right home for the durable copy of training data, models, and checkpoints. It scales well and it is built to keep data safe.
But a training job often reads the same data again in the next epoch. Going back across the network for every read is unnecessary.
The usual answer is local NVMe storage. Read the data once from object storage, keep a local copy close to the GPU, and use it again. Prefetch the next data before the GPU asks for it.
Object storage keeps the truth. Local NVMe keeps the job moving.
Do not let checkpoints stop the job
Training jobs save checkpoints so they can restart after a failure. Those checkpoints can be huge.
If every GPU waits while the checkpoint is written all the way to object storage, the whole cluster pauses. That is a costly pause.
The better approach is to move the state out of GPU memory quickly, then write it in the background. Local SSD can take the first copy. Peer storage can help. Object storage keeps the durable copy.
The other half of the problem is restore time. A checkpoint is not very useful if a job takes hours to come back after a failure.
What I would watch
I would not stop at storage availability or total bytes served. I would watch:
- Time to first byte, especially p95 and p99
- Total read throughput across the cluster
- Request errors, throttling, and retries
- Local-cache hit rate
- Time the data loader spends waiting
- GPU utilization and idle time
- Checkpoint write and restore time
There are newer ways to speed up the path. NVIDIA’s GPUDirect Storage for Objects can use RDMA with compatible S3 storage to avoid part of the normal CPU network path. It is useful, but it is not magic. The network, software, GPUs, and storage all need to support it.
The simple idea is this:
Fast object storage is not about serving bytes quickly. It is about keeping GPUs busy.
Sources