Full-Stack FP4: Stable LLM Pretraining with Quantized Projections, Optimizers, and Attention
arXiv:2607.04422v1 Announce Type: cross Abstract: Recent NVFP4 pretraining methods mainly target transformer linear layers, leaving optimizer states, optimizer arithmetic and attention underexplored in 4-bit pipelines. This critical gap blocks stable full-stack 4-bit pretraining, as the three core modules exhibit unique numerical failure patterns: linear layers hit hard quantization noise limits with dimension-propagated error amplification; AdamW second moments are heavy-tailed non-negative values fragile to low-precision denominators; attention carries error-prone computation paths demanding
This paper addresses a critical, under-explored area in LLM pretraining: achieving stable full-stack FP4 quantization for all core transformer components, which becomes increasingly urgent as AI models scale.
Achieving stable full-stack 4-bit pretraining promises significant reductions in computational cost and memory footprint for large language models, accelerating their development and deployment.
Current 4-bit quantization methods primarily focused on linear layers; this research expands it to include optimizers and attention mechanisms, enabling truly efficient end-to-end low-precision pretraining.
- · AI model developers
- · Cloud computing providers
- · Hardware manufacturers (specializing in low-precision ops)
- · Companies reliant on current high-precision pretraining costs
Reduced computational costs and energy consumption for LLM pretraining.
Faster iteration cycles for new AI models and potentially more accessible model development.
Proliferation of more complex and larger AI models due to loosened compute constraints, impacting various industries.
This signal links to a primary source. Continuum Brief monitors and indexes it as part of the live intelligence stream — we do not republish source content.
Read at arXiv cs.AI