What does numerical precision mean?
Numbers take space. Using fewer bits can reduce memory use and traffic, but it can also change the result.
The same 500 million values
Precision and range differ
Precision is the detail a format preserves. Range is how large or small its values can be.
FP32 uses 32 bits per value. FP16 and BF16 use 16, but divide the bits differently. BF16 has wider exponent range than FP16 and less fractional detail.
Smaller does not automatically mean suitable.
Count all the memory
Seven billion weights at two bytes each need about 14 GB, in decimal units. That is weights only.
The running application also needs buffers and other state. Training needs additional state such as gradients and optimizer data.
Mixed precision uses different formats within a calculation. Quantization uses fewer representable levels, usually with scaling information. Both need output-quality checks.
Try it yourself
Ignoring metadata, how much space do 500 million values need at four, two, and one byte each?
Check your answer
2 GB, 1 GB, and 0.5 GB. The smallest representation is useful only if the hardware, software, and acceptable numerical error support it.