How do GPUs talk to each other?
When GPUs share a job, they also exchange data. That communication can matter as much as the calculation.
Two communication paths
GPU ↔ NVLink / NVSwitch ↔ GPU
GPU path ↔ NIC ↔ network ↔ NIC ↔ GPU path
Two connection boundaries
NVLink connects supported NVIDIA GPUs. NVSwitch provides switching within supported NVLink systems. Capabilities depend on the generation and design.
Communication between servers typically adds network adapters and network switches. People often call these boundaries scale-up and scale-out.
Neither name guarantees performance or automatically combines memory for your application.
Count communication time
In an invented sequential step, calculation takes 10 ms and communication takes another 10 ms.
Halving calculation time makes the step 15 ms, not 10 ms. The unchanged communication now accounts for more of the total.
Real workloads may overlap the two, so measure their actual behavior.
Try it yourself
System A takes 8 ms to calculate and 2 ms to communicate. System B takes 8 ms and 8 ms. Assume no overlap.
Check your answer
A takes 10 ms per step; B takes 16 ms. The GPUs could be identical. Their connections and placement can explain the difference.