As I Understand It — Episode 01: GPUs Explained Audio: https://sridharrajarao.com/audio/podcast/01-gpus-explained.m4a Transcript Lightly edited for readability. Timestamps are approximate. [00:00] I’ve been trying to understand GPUs beyond just hearing that AI needs more of them. What are they actually doing? What's inside those racks and why does NVIDIA have such a head start? [00:15] Welcome to As I Understand It. I am Sridhar. [00:21] Here is how I understand it now. GPU stands for Graphics Processing Unit. These chips became popular for rendering video games, which involves doing lots of similar calculations across an image. The key idea is parallel processing, doing many calculations at the same time. And a lot of AI needs something similar. Huge amounts of multiplication and addition organized into collections of numbers called matrices. [00:58] So, hardware developed for graphics became useful for training neural networks. The connection helped it click for me. Gaming and AI look completely different, but some of the computing they need overlaps. A CPU still matters. It handles general tasks and coordinates work. The GPU takes on calculations that benefit from parallel processing. [01:32] Now, when people say they are buying GPUs, I used to picture individual graphics cards. But in a data center, there is a whole system around them. Inside a GPU are processing units and fast memory. That memory holds the numbers the processor needs. Getting those numbers to the processor quickly is a big part of performance. [02:06] Then you have GPUs, CPUs, storage and networking packaged into servers. Or compute trays that slide into a rack. For example, NVIDIA's GB200 NVL72 puts 72 GPUs in one rack. Connect racks together and you have a cluster. You may also hear SU, meaning scalable unit, a repeatable group of systems. [02:41] Its size depends on the design. So, I picture it by zooming out, chip, server, rack and cluster. Here is the part I had not appreciated. All those GPUs need to talk to each other. Sometimes a model is too large for one GPU's memory. Sometimes you are splitting the training work across many GPUs. Either way, information has to move. [03:18] Broadly, the front-end network connects the systems to users and supporting services. The back-end fabric carries communication between the machines doing the AI work. Storage and management can have separate networks too. In that NVIDIA rack example, NVLink and NVSwitch connect the GPUs. Between the racks, systems can use InfiniBand or Ethernet. [03:52] The switch directs data between connected devices. Why does that matter? Because a fast GPU waiting for data isn't getting much done. Imagine a training job where different GPUs process different batches of examples. Each calculates how the model should change. Then they combine those calculations so their copies of the model stay aligned. [04:26] They compute, exchange results, update and repeat. That helped me understand why twice as many GPUs doesn't automatically mean twice as fast; the communication has to keep up. Which brings me to NVIDIA. I had heard that CUDA was a big advantage, but I didn't really understand why. CUDA stands for Compute Unified Device Architecture. [05:01] It's NVIDIA's platform for programming its GPUs to do the work beyond graphics. Over time, NVIDIA built tools and libraries that developers could reuse instead of starting from scratch. Then in 2012, AlexNet used NVIDIA GPUs and CUDA to make a major advance in image recognition. More researchers adopted the tools. [05:35] More software grew around them. More developers learned how to get good performance from NVIDIA hardware. So a competing chip has to offer enough value to make moving that software worthwhile. And there are alternatives, right? AMD offers GPUs and its ROCm software platform. Google builds TPUs. Amazon builds Trainium. Those last two are purpose-built AI accelerators. [06:09] What I have learned is that the competition involves the whole system. Now, when I hear about thousands of GPUs going into a data center, I picture the memory, the networks, switches, software, power and cooling around them. And I have a better question than “how many GPUs?” How much useful work can they actually do together?