Why does a GPU wait?
A powerful GPU can spend its time waiting. Before buying a faster one, find out where the job actually spends its time.
Where six seconds go
Follow the slowest stage
For a separate GPU, data may travel from storage to CPU preparation, then GPU memory, then calculation. A delay anywhere can hold up the result.
The diagram uses an invented job with no overlap. Four seconds go into preparation, one into transfer, and one into calculation.
Halving calculation time saves only half a second. Reducing preparation from four seconds to two saves two seconds.
Three different measurements
Capacity is how much fits. Bandwidth is how much moves per second. Latency is how long an operation takes.
A large memory capacity does not guarantee a fast data supply. Also watch units: eight bits make one byte.
Try it yourself
Why does doubling calculation speed not halve this six-second job?
Check your answer
Only one second was calculation. The new total is 5.5 seconds. Real applications may overlap stages; then the slowest stage can set the steady pace instead of their times simply adding up.