Rethinking Shrinkage Bias in LLM FP4 Pretraining: Geometric Origin, Systemic Impact, and UFP4 Recipe
arXiv:2606.20381v1 Announce Type: new Abstract: FP4 training promises substantial reductions in memory and computation cost for LLM pretraining, yet current FP4 hardware paths and recipes, including NVIDIA Blackwell/Rubin-class systems and AMD MI350-series GPUs, remain centered on E2M1 data elements. In this study, we identify a fundamental limitation of that choice: non-uniform formats such as E2M1 inherently suffer from Shrinkage Bias, a systematic negative rounding error caused by the geometric asymmetry of their representable bins. We show that this bias accumulates multiplicatively across
The paper identifies fundamental limitations in current FP4 deep learning hardware designs, particularly for LLM pretraining, as companies like NVIDIA and AMD are rolling out next-generation AI accelerators.
This research reveals a systemic design flaw in a core component of the next generation of AI compute, potentially impacting the efficiency and accuracy of large language model development.
Understanding the 'Shrinkage Bias' could lead to revised hardware designs, software optimizations, or new FP4 quantization recipes that improve the fidelity of LLM pretraining and reduce computational errors.
- · AI compute architects
- · LLM developers
- · Semiconductor companies adapting designs
- · Academic researchers in quantization
- · Hardware designs that don't account for shrinkage bias
- · LLMs trained sub-optimally on current FP4
- · Early adopters of uncorrected FP4 hardware
Hardware manufacturers will need to reassess and potentially redesign FP4 implementations, or provide software mitigations.
New quantization schemes may emerge, leading to more efficient and accurate AI training on next-generation hardware.
This could slightly delay the full realization of memory and computational savings expected from FP4 training, or necessitate faster iteration on hardware architectures.
This signal links to a primary source. Continuum Brief monitors and indexes it as part of the live intelligence stream — we do not republish source content.
Read at arXiv cs.AI