Why are there different kinds of memory?
A GPU uses several places to hold data. The useful distinction is who uses each place and how much it can hold.
See the connection
Who can use which memory
A simplified ownership map. These boxes show scope, not relative capacity or physical size.
Reference: NVIDIA memory guidance.
Learn four names
Registers hold a thread’s working values. Shared memory lets threads in a block cooperate. Caches keep reusable copies of data. Device memory, often HBM or GDDR, holds larger arrays.
CUDA shared memory is not the same idea as an Apple Mac sharing system memory between CPU and GPU.
Reuse saves trips
If many calculations need the same small table, keeping it nearby can reduce repeated memory traffic.
But nearby resources are limited. A block using more shared memory can leave room for fewer blocks on an SM. Registers can impose another limit.
Occupancy describes resident warps relative to a hardware limit. It is not a direct measure of useful work, and higher is not always faster.
Try it yourself
A fictional SM has 64 KiB available for shared-memory allocations. How many blocks fit if each needs 32 KiB? What about 16 KiB?
Check your answer
Two or four, considering only this limit. Other limits may reduce those counts. You still need a timed workload to decide which arrangement performs better.