What is inside a GPU?
Inside a GPU are processing groups, memory, and connections. More arithmetic units help only when the rest of that system can keep them supplied.
Inside the GPU
Meet the SM
NVIDIA calls a major processing group a streaming multiprocessor, or SM. A GPU contains multiple SMs. Each has execution units and nearby resources for working data.
An advertised CUDA core is not a complete, independent CPU. Instructions and working state are managed within larger groups.
Trace one addition
To add two numbers, the GPU must get them, perform the addition, and put the answer somewhere. If fetching the numbers is slow, more arithmetic capacity alone may not help.
The GPU chip, its card or module, and the server containing it are also different boundaries. Do not use those names interchangeably.
Try it yourself
In the diagram, mark two places where the inputs could be delayed.
Check your answer
The CPU-to-GPU connection and memory access are two possibilities. The point is to follow data movement as well as calculation. A core count cannot describe both.