A 70-billion-parameter model at 16-bit precision takes up about 140 gigabytes of memory before it generates a single word, since each weight is stored in two bytes. In theory, 4-bit quantization shrinks the weights to 35 gigabytes, a fourfold reduction. In practice, the exact footprint depends on the compression format, and space for the KV cache, where the model stores intermediate results for the context it has already processed, still has to be allocated on top of the weights.
Memory capacity solves only half the problem. A model can fit entirely in the accelerator's memory and still respond slowly, because the GPU re-reads the weights it needs for every token it generates. When reading those weights takes longer than the math itself, adding more compute does not make the answer come out faster. This bandwidth requirement is why accelerators use HBM, where memory dies are stacked vertically in the same package as the accelerator and connected through a very wide interface.
Assembling and testing those stacks takes considerably more work than producing conventional memory chips, and the competition for factory capacity starts before assembly, at the die production stage. Micron estimates that producing a given number of HBM3E bits takes about three times the wafer capacity of producing the same number of DDR5 bits on a comparable process node. When a manufacturer shifts more wafers to HBM, it has less room to grow output of other memory from that capacity. Servers for AI workloads also need ordinary system memory, so deploying more accelerators both raises demand for standard server RAM and makes it harder for suppliers to ramp up production of that memory.
Expanding cleanroom space and bringing new packaging and testing lines up to volume takes far longer than placing a server order, so the memory market can stay tight even as total output grows. Micron's estimate is about manufacturing capacity and does not translate directly into the price of a specific DDR5 module next month, and accelerator demand alone cannot explain every change in memory pricing.
In the eight-minute video below, we walk through the whole chain, from the arithmetic of model weights and the KV cache to factory capacity. At the end we show how measurements on a real request can tell you whether the bottleneck is memory capacity, single-user response speed, or throughput under dozens of concurrent requests.