What makes a balanced server?
A balanced server supplies the GPUs at the rate the workload needs. There is no fixed rule saying every GPU needs the same amount of CPU, RAM, or network.
Demand exceeds the shared supply
Start with one workload
Measure input bytes, preparation work, GPU time, memory, and output. Then multiply only the demand that really happens together.
For an invented example, eight GPUs consuming 2 GB/s each need 16 GB/s. A shared path delivering 10 GB/s cannot sustain that forever without useful data reuse.
A cache can help. It does not make an insufficient source infinitely fast.
Include the busy moments
Model loading, checkpoints, and simultaneous job starts can create bursts above the usual rate.
Test these as well as steady operation. Leave room for temporary memory allocations and the other work the server must do.
Try it yourself
Four GPUs each need 3 GB/s. Their common path supplies 8 GB/s. Is it enough?
Check your answer
Not for a sustained 12 GB/s demand. Check whether the data is reused, peaks overlap, or preparation changes the bytes transferred. Label those assumptions rather than silently relying on them.