01
GPUs Explained
What a GPU does, why AI uses it, and how memory, networks, switches, and CUDA bring the whole system together.
Download episode 01 (M4A)Read transcript
Lightly edited for readability. Timestamps are approximate.
I’ve been trying to understand GPUs beyond just hearing that AI needs more of them. What are they actually doing? What's inside those racks and why does NVIDIA have such a head start?
Welcome to As I Understand It. I am Sridhar.
Here is how I understand it now. GPU stands for Graphics Processing Unit. These chips became popular for rendering video games, which involves doing lots of similar calculations across an image. The key idea is parallel processing, doing many calculations at the same time. And a lot of AI needs something similar. Huge amounts of multiplication and addition organized into collections of numbers called matrices.
So, hardware developed for graphics became useful for training neural networks. The connection helped it click for me. Gaming and AI look completely different, but some of the computing they need overlaps. A CPU still matters. It handles general tasks and coordinates work. The GPU takes on calculations that benefit from parallel processing.
Now, when people say they are buying GPUs, I used to picture individual graphics cards. But in a data center, there is a whole system around them. Inside a GPU are processing units and fast memory. That memory holds the numbers the processor needs. Getting those numbers to the processor quickly is a big part of performance.
Then you have GPUs, CPUs, storage and networking packaged into servers. Or compute trays that slide into a rack. For example, NVIDIA's GB200 NVL72 puts 72 GPUs in one rack. Connect racks together and you have a cluster. You may also hear SU, meaning scalable unit, a repeatable group of systems.
Its size depends on the design. So, I picture it by zooming out, chip, server, rack and cluster. Here is the part I had not appreciated. All those GPUs need to talk to each other. Sometimes a model is too large for one GPU's memory. Sometimes you are splitting the training work across many GPUs. Either way, information has to move.
Broadly, the front-end network connects the systems to users and supporting services. The back-end fabric carries communication between the machines doing the AI work. Storage and management can have separate networks too. In that NVIDIA rack example, NVLink and NVSwitch connect the GPUs. Between the racks, systems can use InfiniBand or Ethernet.
The switch directs data between connected devices. Why does that matter? Because a fast GPU waiting for data isn't getting much done. Imagine a training job where different GPUs process different batches of examples. Each calculates how the model should change. Then they combine those calculations so their copies of the model stay aligned.
They compute, exchange results, update and repeat. That helped me understand why twice as many GPUs doesn't automatically mean twice as fast; the communication has to keep up. Which brings me to NVIDIA. I had heard that CUDA was a big advantage, but I didn't really understand why. CUDA stands for Compute Unified Device Architecture.
It's NVIDIA's platform for programming its GPUs to do the work beyond graphics. Over time, NVIDIA built tools and libraries that developers could reuse instead of starting from scratch. Then in 2012, AlexNet used NVIDIA GPUs and CUDA to make a major advance in image recognition. More researchers adopted the tools.
More software grew around them. More developers learned how to get good performance from NVIDIA hardware. So a competing chip has to offer enough value to make moving that software worthwhile. And there are alternatives, right? AMD offers GPUs and its ROCm software platform. Google builds TPUs. Amazon builds Trainium. Those last two are purpose-built AI accelerators.
What I have learned is that the competition involves the whole system. Now, when I hear about thousands of GPUs going into a data center, I picture the memory, the networks, switches, software, power and cooling around them. And I have a better question than “how many GPUs?” How much useful work can they actually do together?
Sources explained
Plain-language summaries, key takeaways, and links to the original references. Select a title to read more.
Supporting references verified after recording.
NVIDIA CUDA Programming Guide
How does software organize work for a GPU?
CUDA provides a programming model and software platform for using NVIDIA GPUs. Its introductory material explains the concepts without requiring readers to start by writing code.
Key takeaways
- Applications can divide work between CPUs and GPUs.
- GPU programs organize parallel work into groups of threads.
- Understanding execution and memory helps explain performance, even when using higher-level libraries.
Connection to the episode: This supplies the software side of the explanation: useful GPU performance depends on how work is organized, alongside the hardware.
Read first: Part 1, especially Programming Model. The coding chapters are optional for this episode.
Read the original sourceNVIDIA DGX SuperPOD GB200 Reference Architecture — Network Fabrics
What connections does a large GPU system need?
This architecture separates communication into different functions: connecting GPUs, connecting compute systems, accessing storage, and managing equipment.
Key takeaways
- NVLink connects GPUs within the rack-scale system.
- The compute fabric connects systems across the cluster.
- Storage and management traffic have their own requirements.
- Logical network separation does not necessarily mean a separate physical network for every function.
Connection to the episode: It explains why connecting GPUs together involves several kinds of communication.
Read first: The opening network list, Multi-node NVLink Fabric, and Compute Fabric diagrams.
Read the original source