How does work spread across GPUs?
A job can spread across GPUs in several ways. The choice changes what they need to exchange.
What gets divided
Same model replicas · different inputs
Parts of a mathematical operation across devices
Stage 1 → stage 2 → stage 3 on different devices
Three patterns
Data parallelism: replicas process different input data, then combine relevant results such as gradients.
Tensor parallelism: devices split parts of the mathematical work and exchange intermediate values.
Pipeline parallelism: devices handle successive stages. An unbalanced stage can leave others waiting.
Real systems may combine these patterns.
Learn one exchange
An all-reduce combines values and returns the combined result to every participant. If four participants contribute 2, 3, 5, and 7 to a sum, each receives 17.
That coordination has a cost. Keep the work and quality target constant when comparing different GPU counts.
Try it yourself
A job takes 120 seconds on two GPUs and 80 seconds on four. What did doubling the count achieve?
Check your answer
Speedup is 120 ÷ 80 = 1.5×. Relative to ideal doubling, efficiency is 75%. Communication, imbalance, and work that cannot run in parallel are possible reasons, not automatically a faulty device.