GPU Session 7: What is a warp?

A block is divided into warps for execution. On NVIDIA GPUs, a warp contains 32 threads. Our 128-thread block therefore contains four warps.

A warp lives inside a block

Same example as Session 6: 128 threads in block 2. Each numbered square below is one thread, not a core.

Block 2 · 128 threads4 warps × 32 threads

Warp 0Threads 0–31

Warp 1Threads 32–63

Warp 2Threads 64–95

Warp 3Threads 96–127

Thread 5 belongs to warp 0 of block 2 and handles item 261. Warp numbers here are local to this block. A warp never combines threads from two blocks.

Where the hardware fits

One SMStreaming multiprocessor · hardware

Block 2
4 warpsAnother block
If resources allow

Warp scheduler(s)Select ready warp instructions

Execution unitsPerform the work

Illustrative placement, not a chip floor plan. An ordinary block runs on one SM; an SM can host multiple blocks. Warps share execution resources over time.

When one warp takes two paths

Path A

Threads 0–15 active16–31 inactive

Path B

0–15 inactiveThreads 16–31 active

Simplified divergence example, not a timing guarantee. Inactive threads do not contribute to that path. Correct data exchange still needs appropriate synchronization.

Same recipe with different numbers

The threads can run the same instruction on different data. Imagine applying a discount to 32 different prices.

If half need formula A and half need formula B, the warp takes different paths with the appropriate threads active. This is divergence. It can leave some execution lanes inactive while a path runs.

That is not automatically a bug or a prediction that the whole program becomes twice as slow.

What the SM does

The SM’s schedulers select ready warp instructions. While one warp waits, another may make progress.

More threads do not create unlimited capacity. Registers, shared memory, and hardware limits constrain how much work can reside there.

Do not assume threads can exchange data safely just because they share a warp. Use the synchronization the algorithm requires.

Try it yourself

For 64 items, compare alternating A/B formulas with 32 A items followed by 32 B items.

Check your answer

Alternating items mix paths in both warps. Grouping makes each warp follow one path in this example. Rearranging data also costs time, so measure before calling it an improvement.

Sources and further reading

execution model