Ryan Peterman · Ryan Peterman · Transcript & summary by WhipScribe
A Turing Award-winning textbook author explains in under ten minutes why AI ended up on graphics chips and then on purpose-built ones.
Tap a chapter to open that moment in the transcript; the link to it is copied for you.
“At the heart of neural networking is a matrix multiply.” David Patterson · at 5:22 — open in the transcript →
“It was like 30 times better at inference than the contemporary GPU and 80 times better than the CPU.” David Patterson · at 6:56 — open in the transcript →
In a nine-minute clip from Ryan Peterman's channel, Turing Award winner David Patterson explains the three chip families behind modern computing as a story of specialization. CPUs stay general-purpose, with dozens to a hundred cores built much like those of two decades ago. GPUs grew out of graphics, where heavy hardware multithreading hides slow memory and 16- or 32-bit floating point is enough, which made them cheap sources of fast arithmetic.
Patterson credits two turns for moving that arithmetic into AI. Jensen Huang funded CUDA in 2006 so the graphics hardware could be programmed for other work, and the 2012 AlexNet result, trained on a GPU, convinced the field to adopt both neural networks and GPUs. Google then started from a clean slate with the TPU: a large matrix-multiply unit, software-managed memory in place of hardware caches, and the brain float 16 format that favours range over precision.
The payoff, in Patterson's account, was a chip roughly 30 times better at inference than a contemporary GPU, which pushed Intel, NVIDIA and the hyperscalers to respond. With clock speeds stuck near 2005 levels, he argues, CPUs keep their role for operating systems and compilers while the money flows to AI accelerators.
Turing Award-winning computer architect and textbook co-author, who here explains how CPUs, GPUs and Google's TPU came to be built the way they are.
David Patterson frames the three as rising levels of specialization. A CPU is general-purpose, with 50 to 100 cores much like designs of 20 years ago. A GPU is a heavily multithreaded design built for graphics, with modest 16- and 32-bit floating point. Google's TPU is dedicated to machine learning, built around one large matrix-multiply unit with the general-purpose machinery stripped out.
David Patterson says GPUs offered far better single-precision floating-point performance per dollar than CPUs, at prices in the hundreds of dollars. Researchers first disguised their problems as image generation to use them, then CUDA made them programmable. The 2012 AlexNet entry, trained on a GPU by a University of Toronto researcher, crushed an image-recognition competition, and within a few years the field had switched to neural networks running on GPUs.
David Patterson says NVIDIA's founder and CEO Jensen Huang liked the idea of people using cheap graphics chips for other computing, and in 2006 funded a proprietary C-like language, CUDA, to make the multithreaded GPU easier to program. Huang then courted scientific users such as Department of Energy labs and built domain-specific libraries so GPUs could break out of graphics.
David Patterson says Google designed the TPU from a clean slate with no graphics legacy. It centred on a giant matrix-multiply unit, dropped hardware caches because machine-learning memory accesses are predictable enough for software to schedule, and used a narrow floating-point format. The result was about 30 times better at inference than a contemporary GPU and 80 times better than a CPU.
Brain float 16 is the floating-point format Google introduced with the TPU, named after the Google Brain research group. David Patterson says it was the first format to give the exponent more bits than the fraction. Scientific computing mostly uses 64-bit numbers with a 50-plus-bit fraction for precision, but machine learning needs range far more than precision, so Google made that trade.
David Patterson says that if CPUs could still get dramatically faster, GPUs would have stayed a niche. Clock speeds have barely improved since two-to-three-gigahertz chips in 2005, so specialization took over. CPUs remain the right tool for operating systems, compilers and similar work, even as most industry investment now goes to AI accelerators.
Our own transcript of the public episode, with a timestamp about every 90 seconds. Tap a timestamp or the link button on a paragraph to copy a link that opens this page there.
0:00What's the high-level differences between a CPU, a GPU, and a TPU? The CPU has got to be this general-purpose thing. And basically, even today, they have these general-purpose cores that each core is very similar to what the instruction sets used to be like 20 years before that. It's not a surprising design, and it's just got lots of cores and modern ones today could have a hundred, you know, 50 or a hundred cores in there. So that's, that's the feature. For graphics, what they decided to do, which to do the graphics job, is they need to get a lot of performance on their memory system. So they went to multi-threading. So they have hardware threads that you would send a memory request out and the hardware
0:46would switch over to do something else while the memory is coming out. So it's this highly threaded architecture and that's what they developed and could run well for graphics. And graphics, it turns out, doesn't need a powerful floating point. It doesn't need powerful floating point. It doesn't need wide floating points. So 32-bit floating point is plenty for graphics, even 16-bit. So they were pushing graphics with 16- and 32-bit floating point in this multi-threaded kind of architecture. And because it was kind of its own unique thing, it has its own set of terminology all to itself. It wasn't kind of out of the main branch of computer architecture. So when you look at our textbooks, we have kind of like a Rosetta stone,
1:33which says, here's the terms that NVIDIA uses to describe DP2s. This is what it means in kind of normal, well, normal in the mainstream processor design. So they were this niche product that happened to do pretty fast single precision floating point and even half precision floating point. And they were pretty cheap. They were like hundreds of dollars. So when people, so kind of, but some people were thinking, boy, for some applications, if I could turn my program into like pretend that it was creating an image, if I could turn my problem into image generation, I could use these pretty cheap GPUs, which had very good floating point performance per dollar compared to anything else. And so people started playing around with that. And the founder of CEO, Jensen Wang, really liked that idea. So in 2006,
2:35he funded an effort to create a programming language that would kind of handle this multi-threaded hardware architecture that's for graphics to make it easier to program. And so that led to the invention of CUDA, which I can't remember its acronym. But it's really a proprietary programming language for the multi-threaded GPU architecture to make it kind of easier to program. It was a C-like language, but you can't just compile C programs and run it. But it was C-like language, but you can't just compile C programs and run it. But it was C-like. But people actually liked it. You know, it was certainly much better than trying to turn your program into an image generation. But people liked it.
3:13But Jensen Wang, his vision was try to get these teenagers in the Department of Energy labs, like fluid floating and some of the things like, you know, some. And he would build special purpose libraries that could handle that domain. And then he used his GPUs to do that kind of special purpose computing, but breaking out of graphics. But that's kind of the heritage there. Now for the TPU program, and so what's happened, people started right from the very beginning, people started using GPUs because they had much better single precision floating point performance than the CPUs. They had a lot more processors on them. So they had the potentially much more performance. So they had potentially much more performance. So one of the breakthrough moments was in 2012 when within the machine learning community,
4:11this neural networking piece, which had a few advocates, but a lot of people didn't believe in it, they went into a competition to see who could do the best image recognition. And in 2012, the so-called AlexNet beat all the competitions. It was this historic moment in machine learning and neural networking. And once they did it, and the guy who did that had taken a coup de grace at the University of Toronto to learn how to use it. And so he says, well, as long as I'm doing this, I'll do it on a GPU. And so he was able to explore a lot more space on this cheap, fast GPU. So he entered it in the competition 2012. And he was the only one using neural networks. And he crushed the competition. And within a few years, everybody switched over.
4:57And not only were they doing neural networks, but they're using GPUs. So GPUs have this heritage of being a graphics engine, but more programmable and started getting used for machine learning. When Google came along, they decided, you know, it was a clean slate. They didn't care about graphics. So at the heart of neural networking is a matrix multiply. That's the thing. So they designed a processor for at the heart of neural networking is a matrix multiply. That's the thing. So they designed a processor for at the time, microprocessor at the time, had a giant matrix multiplying unit. That was the main thing. And then they threw a bunch of stuff out that they didn't need. So a lot of general purpose computing is,
5:39what's on the chip is maybe three levels of caches to try and have, so you don't spend all your time going to the relatively slow memory. Well, for machine learning, they knew when their memory accesses were, so they could schedule that. So they didn't, a hardware cache didn't make any sense. They would just have a memory, the software understood and would transfer in time. So those are some of the innovations. Also, they innovated on the floating point format. You didn't need, you know, hot scientific computing cares a lot about precision. They have, you know, most of it's done in 64-bit floating point where the exponent is, you know, less than 10 bits and most of it is the precision, you know, the fraction can be 50-some bits. Well, what they
6:23realized for machine learning and AI, they don't need all that precision. They needed the range. So Google did the first floating point format where the exponent was bigger than the fraction. That was a radical idea, that so-called brain float 16, the part of Google that was doing this was the brain research group. So they brought out this architecture. They had this narrow floating point format. They had big matrix multiply unit. It only had one processor in it, unlike the other ones. It was just dedicated for machine learning. And it just kind of blew the doors off of everybody in the field. It was like 30 times better at inference than the contemporary GPU and 80 times better than the CPU by having this dedicated one.
7:07So when Google made this announcement at their annual retreat a year or so after they had deployed it inside, it just shook everybody up. Intel started buying companies. companies, NVIDIA started modifying the design for it to be much more handsome machine learning. And then a bunch of other competitors started, hyperscalers started doing their own efforts. So I think that was the watershed. I think the TPU announcement was the watershed moment there. So it's almost like levels of specialization, like the CPU is the most general. could build 100 terahertz microprocessors, that's what we'd do. GPUs would still be a niche. If we could do that, it raises all boats. That would be fantastic if we could do that. But that's long in
8:13the history. We haven't been able to do that for 20 years. And so there were probably two or three gigahertz microprocessors in 2005, and that's kind of where they barely improve today. So we can't do that. So there's still areas where it's important to have CPUs. They're the right, you know, you need operating systems, compilers and things. They're the right solution. But things have been specialized. And then now in terms of industry terms, huge emphasis is being put into the AI accelerators or just AI in general is where the money is being invested. And so the question for any processor designers was, how does my processor design fit into this AI universe? And what should I do? How should I change it for this important application?