Part 3 ended with a chip full of switches and nothing else. Here is the question that bothered me for a long time: if all you have is on and off, how do you add 3 and 5? There are no numbers in the machine. There is no "3" anywhere. There is a wire that either carries a voltage or does not. And yet the machine on your desk adds billions of numbers a second and never gets one wrong.
The answer comes in three steps. Agree on a way to write numbers using only on and off. Build a few tiny circuits that combine ons and offs in useful ways. Then wire those circuits together in the same pattern you learned for adding in primary school. By the end of this part you will have watched a working adder built from nothing but gates, and you will understand the one economic fact that explains why AI chips keep using fewer and fewer bits per number.
Counting with two fingers
We write numbers in tens, almost certainly because we have ten fingers. It is the usual explanation and a good one, the word digit means both a finger and a numeral, but it is a reasonable hypothesis rather than a proven fact, and plenty of cultures counted otherwise. What matters here is that ten is a choice, not a law. Change the choice and everything still works. A machine made of switches has, in effect, two fingers. Its digits go 0 and 1, and when you run out you carry one to the next column, which is worth twice as much. This is binary, and it is the same positional system you already know with a different base.
Side note: humans have used other bases too. The Sumerians and Babylonians counted in base 60, and we still live with it every day: 60 seconds in a minute, 60 minutes in an hour, and 360 degrees in a circle are all leftovers of theirs. The Maya used base 20, and a trace survives in French, where 80 is quatre-vingts, "four twenties". Base 12 is scattered through English, in the dozen, the 12 inches of a foot, and the 12 hours on a clock, and one theory is that people counted the three bones of each of the four fingers with the thumb, giving twelve per hand. None of these is more "natural" than another. Each is just a number of symbols before you carry, and computers carry after two.
So the columns of a binary number are worth 1, 2, 4, 8, 16, and so on, doubling each time. The number 5 is one four plus one one, written 0101. The number 3 is 0011. Add them the way you were taught, column by column from the right, carrying when a column overflows, and you get 1000, which is one eight. Three plus five is eight. Each column is a bit, and four of them can hold any number from 0 to 15. Eight bits (a byte) reach 255, and 32 bits reach a little over four billion. A 32-bit number is just 32 wires, each on or off.
Binary place values. Click any bit to flip it. Every column is worth double the one to its right, and the number is simply the lit columns added up.
Counting upward works the same way, and there is an old machine that shows it beautifully: the mileometer in a car, a row of numbered wheels. It is the clearest picture I know of what a "base" really is, and it works for any base at all.
Counting is a row of wheels, like a car's mileometer. Press +1 and the rightmost wheel turns. When a wheel passes its last digit it clicks back to 0 and nudges its neighbour up by one, and that click is all "carrying" is. A base-10 wheel rolls over every ten. A binary wheel has only two faces, 0 and 1, so it rolls over every two, which is why binary needs more wheels for the same number.
Where does the "3" live, then? Nowhere and everywhere. It lives in the agreement that wires two and one are on and the rest are off. The machine does not know it is a three. It only knows which wires are hot. Every number, letter, pixel, and neural network weight in a computer is a convention like this, laid over a pattern of switches.
Gates, built from transistors
The next step is to build circuits that take some bits in and put a bit out. These are logic gates, and there are only a few of them, each made from a small handful of transistors. Before drawing any, I want to be exact about what a transistor is doing inside one, because the drawings are easy to misread.
Recall from Part 3 that a transistor has three connections. Two are the ends of the channel, the source and the drain, the path that current flows along when the switch is closed. The third is the gate, a plate that sits on top of the channel behind a film of insulator. I will call it the control gate from here on, because "gate" is about to mean something else. Put a voltage on the plate and the channel below it opens or closes. Nothing flows through the plate itself. It is insulated from the channel and takes almost no current at all. It is a lever, not a pipe. Everything in this section follows from that one distinction. An input such as A is a voltage that we apply to control gates, and it is one of the same two levels every wire uses. The current that lights the output flows somewhere else entirely, along the channels, and A never touches it.
Chips use two flavours of transistor. An n-type conducts when its control gate is at 1 and stops when it is at 0. A p-type is the mirror image: it conducts when its control gate is at 0 and stops at 1. In the drawings the p-type carries a small circle on its control wire, the same "invert this" bubble you will meet on the gate symbols later. Here is the smallest gate of all, a NOT, drawn the way every gate in this section will be drawn, with each part labelled. It is worth reading slowly before going on.
Start with the two long horizontal bars. They are the chip's power supply, the battery from Part 1 in modern dress, and they define what the two digits mean. The dark one at the bottom is the ground rail, held at zero volts, and "0" means "at the ground rail's voltage". The green one at the top is the power rail, held at the supply voltage, a little under one volt on a modern GPU and five volts on the chips of the 1980s, and "1" means "at the power rail's voltage". It is worth being exact here, because it is easy to slip into thinking that 0 is the absence of a voltage and 1 is its presence. It is not. Both are definite levels that a wire is actively held at, and neither is special. Zero volts is only "zero" because we chose the ground rail as the level everything else is measured from. Each digit also comes with a tolerance. Anything within a band near the ground rail's voltage reads as 0, anything within a band near the power rail's reads as 1, and the gap between the two bands is a no-man's land that a healthy signal crosses quickly on its way from one to the other and never rests in. Part 1's side note said this in passing. Here is where it starts to matter. The rails themselves are not inputs and nobody sets them. They are simply there, the way the mains socket is simply there, and every gate on the chip is wired between the two. A gate's whole job is to connect its output wire to one of the rails, depending on the inputs, and the lamp on the right reports which.
The input comes in from the left. It is a single wire, drawn thin and blue, running from the labelled terminal to the control gate of every transistor that A is meant to work, here the p-type and the n-type both. That answers the question I found most confusing when I first met these diagrams: one input has to work two transistors, so how do I "set" both of them? By setting one wire. Pressing the button to make A = 1 puts the power rail's voltage on wire A, and every control gate hanging off that wire feels it at the same instant, because they are all the same wire. Two transistors on the same input wire are two switches worked by one lever. The thick wires are different. Those are the ones current can flow along, from a rail, through a transistor's channel, to the output. When a transistor conducts, its rail's colour spreads through it, green from the top or dark from the bottom, and a wire joined to neither rail stays pale.
One thing the drawing must not suggest: thin does not mean low voltage. The thin wire carries exactly the same two levels as every other wire on the chip, the ground rail's voltage for a 0 and the power rail's for a 1. What separates the two kinds of wire is what happens at the far end, not the voltage on them. A thick wire is part of a path that current can flow along, from a rail through a channel to the output. A thin wire ends at an insulated plate, so it delivers its voltage to the plate and almost nothing flows along it. On the chip both are the same metal, and one wire often does both jobs in turn. The AND demo further down shows a gate's output wire, drawn thick, running straight on to become the next stage's control wire, drawn thin, without its voltage changing at all. The thickness is my convention for what a wire is doing, not a difference in the wire. In the figure A is 1, so the n-type is on, the ground rail's dark colour runs up through it to the output, and the lamp reads 0.
Here is the same gate with A set to 0. Nothing has moved except the input. The p-type is now the one that conducts, the power rail's green runs down through it to the output, and the lamp reads 1.
One more rule, and it is the whole secret of the design. Every gate is a pull-up team of p-types between the power rail and the output, and a pull-down team of n-types between the output and ground, arranged so that for any input exactly one team conducts. The output is therefore always firmly attached to one rail or the other, never left dangling, and because there is never an open path straight from power to ground, almost no current flows except at the instant of switching. That balance is what the C in CMOS (complementary metal-oxide-semiconductor) means, and it is the reason a chip with 76 billion switches does not melt.
NOT
This is the gate from the figure above, and the one to stare at until the others make sense. One p-type on top, one n-type below, both control gates on the same input wire, and the output taken from the point between them. Set A to 0: the p-type conducts (it wants a 0) and the n-type does not, so the output is joined to the power rail and reads 1. Set A to 1 and the two swap roles, the output is joined to ground and reads 0. One in, the opposite out. Toggle A and watch the rail's colour flow to the lamp.
NOT from two transistors. There is only one input, and both control gates hang off it. The second button takes the n-type away, to show what it was doing.
Side note: why not just the p-type? Cover the n-type with your thumb and the gate seems to work with one transistor. A at 0, the p-type conducts, the output reads 1. A at 1, the p-type stops, and the output is joined to nothing. That is the catch, and the "remove the n-type" button above shows it. A wire joined to nothing does not read 0. It reads whatever charge happened to be left on it, drifting as that charge leaks away and twitching with every neighbouring wire's activity, and sooner or later it wanders into the no-man's land between the two bands, where the next gate along has to guess whether it is looking at a 0 or a 1. To get a firm 0 the output has to be actively pulled to the ground rail, and that is the n-type's whole job. Early chips did without the second transistor by using a resistor (or a transistor wired to behave like one) to do the pulling instead, which gives a firm answer but at a price: whenever the switching transistor is on, current runs straight from the power rail through the resistor to ground, all the time, doing nothing useful except heating the chip. The NMOS chips of the 1970s were built that way, and they ran hot and could not scale. The complementary pair costs one more transistor and wastes almost nothing, which is why nearly every chip since the mid-1980s has been built from pairs like this one.
NAND
With two inputs there is a choice to make: wire the pull-down transistors in series, one after the other, or in parallel, side by side. Series first. The output can only be dragged down to 0 if current can get all the way through both n-types to the ground rail, which needs both A and B to be 1. Any other combination breaks the chain somewhere, and then the pull-up team, two p-types in parallel, takes over: either one of them conducting is enough to join the output to the power rail. So the output reads 0 exactly when A and B are both 1, and 1 otherwise. That is NOT (A AND B), and the gate is called NAND. Try all four combinations, and notice that the two teams are mirror images, series below and parallel above, so that one team is always conducting while the other is always broken.
NAND from four transistors. Each input wire runs to two control gates, one p-type and one n-type. The small arcs are where a control wire hops over another wire without touching it.
AND
A plain AND, which is 1 only when both inputs are 1, is a NAND with a NOT hung on its output. There is no cheaper way to build it from these transistors. The natural single-stage gate comes out inverted, and un-inverting it costs two more transistors. In the drawing the NAND's output wire runs straight into the control gates of a NOT stage, and both stages share the same two rails, exactly as they would on a real chip. Six transistors in all.
AND as a NAND stage followed by a NOT stage. The dot in the middle is the NAND's output, which becomes the input voltage for the NOT.
NOR
Now the other choice: pull-down transistors in parallel. Either n-type conducting on its own is enough to join the output to ground, so the output reads 0 whenever A or B is 1. It reads 1 only when both are 0, which is exactly when the pull-up team, two p-types now in series, conducts all the way through. The output is NOT (A OR B), and the gate is NOR. Set it against the NAND above: the two are each other turned upside down.
NOR from four transistors. Series on top this time, parallel below.
OR
Just as AND is a NAND plus a NOT, OR is a NOR plus a NOT. Same pattern, same six transistors.
OR as a NOR stage followed by a NOT stage.
XOR
XOR ("exclusive or") is 1 when the inputs differ. It is the odd one out, because no single pull-up and pull-down arrangement produces it, so it is usually assembled from gates we already have. One way to say "A and B differ" is "at least one of them is 1, but not both", which is (A OR B) AND (A NAND B): three gates and about a dozen transistors. Chip designers have cleverer versions with as few as six, but this one is the easiest to read. I have drawn it with gate symbols rather than transistors, since every box in it is a circuit you have just seen.
XOR built from three gates. Green means 1 and grey means 0.
XNOR
Finally XNOR, which is 1 when the inputs match: the XOR above with a NOT on its output. Every gate has a negated twin made this way, and the small circle on an output is the symbol for it.
XNOR: the XOR above, then a NOT.
The whole family at a glance
Whatever the transistors underneath, a gate is nothing more than its table. Here are the four behaviours worth carrying in your head:
| A | B | A AND B | A OR B | A XOR B |
|---|---|---|---|---|
| 0 | 0 | 0 | 0 | 0 |
| 0 | 1 | 0 | 1 | 1 |
| 1 | 0 | 0 | 1 | 1 |
| 1 | 1 | 1 | 1 | 0 |
And here is the whole family in the symbols that circuit diagrams use, which is how you will see them from now on. Toggle A and B and watch every gate decide.
The whole family as symbols. The dots on the left are the inputs, the small circle on an output is the "invert this" bubble, and a green wire or lamp means 1, a grey one 0. NOT ignores B, it just flips A.
That is the entire toolkit. Everything a GPU does, drawing a game, running a language model, is some vast arrangement of these tables, and any of them can be built from two to twelve transistors. In practice chip designers build almost everything from one gate, NAND, the natural single-stage gate we met above, which can be wired up to imitate any of the others. You will meet the word again in "NAND flash", the storage in an SSD, which is named after the gate because its memory cells are wired in series the same way.
Adding one column
Look at the XOR column of the table and then look at the right-hand column of the binary addition from earlier. Adding two bits gives 0 + 0 = 0, 0 + 1 = 1, 1 + 0 = 1, and 1 + 1 = 0 carry 1. The sum bit is exactly A XOR B. And the carry bit, which is 1 only when both inputs are 1, is exactly A AND B. One XOR gate and one AND gate add two bits. This circuit is called a half adder, and it was the moment the whole thing clicked for me: the arithmetic falls out of the tables, nobody had to invent it. Here it is, with both inputs to play with.
A half adder. Both inputs go to both gates. The XOR makes the sum bit and the AND makes the carry bit, and the two bits together are the answer written in binary.
A half adder is not quite enough, because every column except the first also has to take in the carry from the column to its right. A full adder takes three bits in (A, B, and carry-in) and puts two bits out (sum and carry-out). It is two half adders and an OR gate, about 28 transistors in the usual CMOS form. The first half adder adds A and B. The second adds the carry-in to that result. Each of the two can produce a carry, but never both at once (if A and B are both 1, their sum bit is 0, so the second half adder has nothing to carry), which is why a plain OR is enough to merge the two carries into the carry-out.
A full adder. The dashed boxes are the two half adders from above. The first adds A and B, the second adds the carry in to that sum, and the OR gate merges their carries. Try 1 + 1 + 1.
Chain 32 full adders together, each one's carry-out feeding the next one's carry-in, and you have a 32-bit adder that can add any two numbers up to four billion. The one below is a four-bit version you can play with. Click the bits.
A four-bit adder built from four full adders. Click any input bit to flip it and watch the sum and the carries change. The carry ripples from right to left, exactly as it does on paper.
There is one practical wrinkle worth knowing about, because it will come back when we talk about clock speed in Part 5. In the chain above, the leftmost adder cannot finish until the carry has rippled through all the ones to its right. For 32 columns that is 32 gate delays in a row, and real adders use cleverer wiring that works out the carries in parallel. The principle is unchanged. It is still XOR, AND, and OR, just arranged so the answer arrives sooner.
Multiplication is repeated addition, laid out flat
Now the first half of our running calculation, w × x. Long multiplication in binary is easier than in decimal, because multiplying by a single binary digit is either "copy the number" (times 1) or "write zeros" (times 0). So multiplying an 8-bit number by another 8-bit number means producing eight rows, each of which is either the top number shifted left or a row of zeros, and then adding the rows up.
Building that in hardware is direct. "Copy the number or write zeros" is an AND gate per bit, so eight rows of eight bits is 64 AND gates. Adding the rows is a grid of full adders. An 8-bit multiplier is therefore a small rectangular array of gates. A 32-bit multiplier is the same pattern with 32 rows of 32, which is 1,024 partial products instead of 64, sixteen times more gates for four times more bits. This is the fact to take away: the size of a multiplier grows with the square of the number of bits. Halve the bits and the multiplier shrinks to roughly a quarter, and gets faster and uses less energy into the bargain.
Side note: real multipliers are cleverer than the grid. The rows-of-partial-products layout above is called an array multiplier. It is the right mental model, but a production chip rarely builds it as drawn, and two refinements from the 1950s and 60s are in almost every CPU and GPU multiplier made today. The first is Booth encoding, from an algorithm Andrew Booth published in 1950. It reads the multiplier's bits in overlapping groups and turns a run of 1s into one subtraction and one addition, so that the usual radix-4 form halves the number of rows and handles negative numbers without any extra circuitry. The second is the Wallace tree (1964) or its thriftier cousin the Dadda tree (1965), which add all the rows at once in layers of carry-save adders that never wait for a carry to ripple, leaving one fast adder at the end to produce the answer. Booth encoding, then a compression tree, then a final adder is the standard recipe, and Intel's original Pentium used a radix-8 variant of Booth in its multiplier. None of it changes the fact that matters here: the hardware still grows with the square of the width, which is why the narrow formats in the next section pay off so handsomely. For the 3- and 4-bit mantissas of FP8 and FP4 there is hardly anything left to encode, and a plain little array does fine. Software has its own tricks for numbers thousands of bits long, such as Karatsuba's method, but those live in cryptography libraries, not in the arithmetic unit of a chip.
GPUs do not keep the multiplier and the adder separate. The unit at the heart of every GPU lane is a fused multiply-add (FMA), which computes a × b + c in one step, with one rounding at the end. It is w × x + b exactly, built in silicon, and it is the atom out of which the whole rest of this series is made. When a spec sheet says a card does 82 trillion floating-point operations a second, it is counting FMAs, two operations each, across all the lanes, every clock tick.
Side note: where are the subtractor and the divider? This part builds adders and multipliers and says nothing about the other two operations, and the reason is that a chip barely has them. Subtraction is addition in a thin disguise. Computers store negative whole numbers in a form called two's complement, in which negating a number means flipping every bit and adding 1, and that turns A minus B into A plus (B with its bits flipped) plus 1. The flipping is one XOR gate per bit, and the "plus 1" goes in through the carry-in of the rightmost full adder, which as you saw has a spare input for exactly this kind of thing. So a subtractor is an adder with a row of XOR gates bolted to one side, and one circuit does both jobs depending on a single control wire. For floating point it is simpler still: flip the sign bit of B, then add. Division is the opposite case, because there is no cheap trick for it. Long division in hardware produces one or two bits of the answer per step, so dividing two 24-bit mantissas takes a dozen or more rounds through the circuit against a single pass through a multiplier, and the CPUs that do have a dedicated divider count its cost in tens of clock ticks. Nvidia GPUs do not have one at all. Each group of lanes shares a few special function units, four for every 32 FP32 lanes on recent chips, which produce quick approximations of a reciprocal, a square root, a logarithm and the exponential that neural networks lean on, and a division is done by taking the reciprocal of the divisor and multiplying, with the answer polished by a couple of multiply-add steps. That arrangement suits the work. Running a neural network is trillions of multiplications and additions, one for every w × x + b, while divisions turn up a handful of times per layer, in normalising a row of numbers or in the softmax that chooses the next word. Building the chip around the fused multiply-add is building it around the operation that happens overwhelmingly more often.
Numbers with a decimal point
Whole numbers are not enough for a neural network. Weights are values like 0.0317 and negative 2.4. The convention for those is floating point, and it is the binary version of scientific notation. A floating-point number has three parts: a sign bit, an exponent that says where the point goes, and a mantissa (also called the significand) that holds the actual digits.
An example makes it concrete. In decimal scientific notation, 6500 is written +6.5 × 10³, and the three parts are already there: the sign is plus, the mantissa is 6.5, the string of digits, and the exponent is 3, the power of ten that says where the point belongs. Write a negative number the same way and only the sign changes: negative 6500 is −6.5 × 10³, same mantissa, same exponent. Binary does exactly the same with powers of two. Take 6.5. In binary it is 110.1 (a four, a two, and a half), and sliding the point along until one digit is left in front gives 1.101 × 2². So the sign is plus, the mantissa is 1.101, and the exponent is 2. FP32 stores those in 32 bits: 1 for the sign (0 means plus), 8 for the exponent, and 23 for the mantissa. Two small conventions make the stored bits look odd at first sight. The exponent is stored with 127 added to it, the bias, so that the negative exponents of numbers smaller than 1 do not need a sign of their own: 2 is stored as 129, which is 10000001. And because a binary number written this way always begins with a 1, that 1 is not stored at all, which buys one bit of precision for nothing: the mantissa field holds only the digits after the point, 101 followed by twenty zeros.
Side note: why add 127 to the exponent? The exponent field is 8 bits, which can hold the whole numbers 0 to 255 and nothing else. But exponents need to run negative as often as positive: 0.1 is 1.6 × 2−4, and the small weights in a neural network live around 2−5 to 2−10. One option would have been to spend one of the 8 bits as the exponent's own sign, the way the number as a whole has one. The designers of IEEE 754 chose something simpler: add a fixed offset to every exponent before storing it, so that −126 is stored as 1, 0 as 127, and +127 as 254. Every exponent that matters becomes a positive number that fits in the field, and reading it back is one subtraction. That offset is the bias, and it is always half the field's range, rounded down: 127 for an 8-bit exponent, 15 for FP16's 5-bit one, 7 for FP8's 4 bits, and 1 for FP4's 2 bits. The two ends of the stored range are held back for special cases. All zeros marks zero itself, and the tiny "subnormal" numbers just above it, which give up the implied 1 to squeeze a little closer to zero. All ones marks infinity and "not a number", the value you get from dividing zero by zero. There is a quiet bonus in the choice, too. With a bias rather than a sign, a bigger number always has a bigger bit pattern, so two positive floating-point numbers can be compared by treating their 32 bits as a plain whole number, and the circuit for "is a greater than b" is the same one the chip already has for integers.
| Number | As a power of two | True exponent | Stored: exponent + 127 | Exponent bits |
|---|---|---|---|---|
| 6.5 | 1.101 × 22 | 2 | 129 | 10000001 |
| 1 | 1.0 × 20 | 0 | 127 | 01111111 |
| 0.1 | 1.6 × 2−4 | −4 | 123 | 01111011 |
| 0.0317 | 1.0144 × 2−5 | −5 | 122 | 01111010 |
Any number goes through the same three steps, and the interesting part is what happens when the mantissa runs out of room. Try 0.1, which has no exact binary form at all (in binary it repeats for ever, the way one third does in decimal), and then watch it get rougher as the format narrows.
Any number, in any of the formats from the table. Type a number or pick one, choose a format, and see the three fields it is stored in and how close the stored value gets to what you asked for.
The name of a format tells you how many bits it spends on each field. Where a name contains an E and an M, they count the exponent and mantissa bits directly:
| Format | Stands for | Total bits | Exponent bits | Mantissa bits | Multiplier size, relative to FP32 |
|---|---|---|---|---|---|
| FP32 | floating point, 32 bits (IEEE "single precision") | 32 | 8 | 23 | 1 |
| TF32 | TensorFloat-32 (Nvidia) | 19 | 8 | 10 | about 1/5 |
| FP16 | floating point, 16 bits (IEEE "half precision") | 16 | 5 | 10 | about 1/5 |
| BF16 | brain floating point, 16 bits (Google Brain) | 16 | 8 | 7 | about 1/10 |
| FP8 (E4M3) | floating point, 8 bits: 4 exponent, 3 mantissa | 8 | 4 | 3 | about 1/36 |
| FP4 (E2M1) | floating point, 4 bits: 2 exponent, 1 mantissa | 4 | 2 | 1 | about 1/144 |
The last column is the point, and it deserves a proper explanation. To multiply two floating-point numbers you multiply their mantissas and add their exponents, exactly as 6.5 × 10³ times 2 × 10² is 13 × 10⁵. The two halves of that job cost wildly different amounts of hardware. Adding the exponents is a single small adder, 8 bits plus 8 bits for FP32, which is eight of the full adders from earlier in this part. Multiplying the mantissas is the grid from the previous section: one AND gate for every pair of bits, one bit from each number, plus the adders to sum the rows. An FP32 mantissa is 24 bits wide once the implied 1 is counted, so its grid has 24 × 24 = 576 cells. Eight full adders next to 576 cells is a rounding error, so the silicon a floating-point multiplier occupies is, to a good approximation, the mantissa grid, and the grid's area is the mantissa width squared. That is what the last column measures. FP16 has an 11-bit mantissa (10 stored plus the implied 1), so its grid is 11 × 11 = 121 cells, about a fifth of 576. FP8 has 4, so 16 cells, a thirty-sixth. The figure draws the five grids to scale.
The FP8 unit is dozens of times smaller, which means you can fit dozens of them in the space of one, running on the same power budget. That table is, in miniature, the story of every Nvidia generation from Volta onward: FP32 gave way to FP16, then BF16, then FP8 on Hopper, then FP4 on Blackwell, and each step bought several times the arithmetic for the same silicon and the same watts. The price is precision, and Part 10 is about why neural networks turn out to tolerate that price remarkably well.
"Gave way" needs a caveat, because a card does not switch wholesale from one format to the next. Each generation adds formats and keeps the old ones. An H100 will multiply in FP64, FP32, TF32, BF16, FP16, FP8 or 8-bit integers, and a Blackwell card adds FP4 and FP6 to the list. The choice is made by the programmer, per operation, simply by declaring what type the numbers are, and the different formats share the same tensor cores, which are built to work either as a few wide multipliers or as many narrow ones. That is why the rates on a spec sheet double at each step: an H100 manages about 495 trillion floating-point operations a second in TF32, 989 trillion in FP16 or BF16, and nearly 2,000 trillion in FP8, all on the same silicon. Choosing a narrower format does not change what calculation is done. It changes how many of them fit through the chip each second, and how precise each one is. The generations story above is really about which format became the default for training and running the big models, as each in turn proved good enough.
Side note: which of these are standards? The classic ones are. IEEE 754, first published in 1985 and revised in 2008 and 2019, is the standard that made floating point behave the same on every computer. It defines FP32 and FP64 (as binary32 and binary64), and since the 2008 revision FP16 as well (binary16). It fixes not just the bit layout but the rounding rules, the special values such as infinity and "not a number", and what happens on overflow, which is why the same calculation gives the same answer on an Nvidia GPU, an AMD one and a laptop CPU. The rest of the table grew up outside the standard. BF16 was devised at Google for its TPU chips. It keeps FP32's eight exponent bits and drops sixteen mantissa bits, so converting from FP32 is a truncation. TF32 is Nvidia's own, introduced with Ampere in 2020, and is really a 19-bit way of running FP32 data through the tensor cores. The 8-bit formats E4M3 and E5M2 were proposed in a 2022 paper by Nvidia, Arm and Intel and then written up as the OCP 8-bit Floating Point specification (OFP8), approved in June 2023 by the Open Compute Project, an industry group rather than a standards body. FP4 (E2M1) comes from the same group's Microscaling (MX) specification of late 2023, backed by AMD, Arm, Intel, Meta, Microsoft, Nvidia and Qualcomm, in which numbers are stored in blocks of 32 that share one 8-bit scale factor, and Blackwell adds Nvidia's own NVFP4 variant with blocks of 16. An IEEE working group, P3109, has been drafting a proper standard for these small machine-learning formats since 2022, and it was still a draft as I wrote this.
Where this leaves us
You now have the machine's entire vocabulary. Bits are wires. Numbers are agreements about which wires are hot. Gates are handfuls of transistors that implement a four-row table. Adders are chains of gates that follow the primary-school method, multipliers are grids of adders, and a fused multiply-add is w × x + b cast in silicon. The whole of a GPU's arithmetic reduces to that one unit, repeated.
It is worth pausing on how little of the chip the arithmetic actually is. A single FP32 FMA is somewhere in the tens of thousands of transistors. An RTX 4090 has 16,384 of them, so all the lanes together account for something like a billion transistors, give or take. The chip has 76 billion. Where did the other 75 go? Most of them are not doing arithmetic at all. They are storing numbers, moving numbers, and deciding what to do next, and that is the subject of the next two parts. An adder that has nobody to tell it what to add is just a very expensive paperweight.
Next: A Machine That Follows Instructions: registers, the clock, the fetch-decode-execute loop, and a six-instruction program that computes w × x + b on a processor you can single-step.