As I Understand It — Episode 03: Inside an AI Data Center Audio: https://sridharrajarao.com/audio/podcast/03-inside-an-ai-data-center.m4a Transcript Lightly edited for readability. Timestamps are approximate. [00:00] When I started learning about AI data centers, I thought, okay, you buy a lot of GPUs, put them in a building, connect them together and start running AI. Then I looked at some actual designs. And I realized buying the GPUs is only the beginning. Welcome back to As I Understand It. I'm Sridhar. We have spoken about GPUs and their names. This time I wanted to understand the place that holds all of them. What is actually inside an AI data center? [00:40] Let's start with the part that surprised me. I looked at a design from Vertiv, a company that builds power and cooling infrastructure. It had 8 racks for computing and 10 more for networking and storage. That's one design. So the numbers will vary. But it made something clear to me. All those GPUs need a lot of support. Inside the compute racks, you have servers or trays containing GPUs, CPUs, memory and connections. Now imagine a large AI training job spreads across thousands of GPUs. Each GPU does part of the calculation. But they also need to exchange results and coordinate. And that's where I had to change my thinking. I was thinking about how fast a GPU can calculate. But what happens if it finishes its calculation and then waits for information from another GPU? You've paid for all the computing power and now it's waiting. [01:57] So the network becomes part of the performance. There are fast connections between GPUs, networks connecting racks and connections bringing data from storage. There is also a management network so engineers can operate the equipment. And storage does more than hold the training data. It also saves checkpoints. The way I understand a checkpoint is simple. Save your progress. If a long training job fails, you want to continue from a recent save. [02:36] Meta describes thousands of GPUs saving and loading checkpoints together. So even “save my work” becomes quite an engineering problem at this scale. Then there is the electricity. Power has to be transformed, protected and distributed to the racks. Backup systems help keep equipment running through interruptions. And because GPUs can work in step with each other, their power demand can change together too. The electrical system has to cope with that. Now think about heat. [03:18] All that equipment is consuming electricity. And most of it ends up as heat that needs to go somewhere. In direct-to-chip liquid cooling, coolant flows through the cold plates attached to hot components. It carries the heat through the pipes to equipment that transfers it into the building's cooling system. Eventually that heat has to leave the building. Some components still need air cooling as well. So now when I picture an AI data center, I can follow what's happening. Data comes in. GPUs work together. Storage saves progress. [04:09] Electricity keeps everything running. Cooling carries the heat away. Software coordinates the jobs. And people monitor and maintain the whole system. That's what stayed with me. A GPU's performance depends on whether the rest of the building can keep up with it. And whenever I hear that AI runs in the cloud, I now picture something very physical. Cables, pipes, pumps, power equipment, and people keeping it all working. That's an AI data center, as I understand it.