Balancing Learning Rates Across Layers: Exact Two-Step Dynamics and Optimal Scaling in Linear Neural Networks
arXiv:2606.00340v1 Announce Type: new Abstract: We study optimal learning-rate selection in two-layer and three-layer linear neural networks trained to learn linear target functions. In particular, we derive the exact closed-form expressions for the gradients and test loss after one and two steps of gradient descent, enabling a precise characterization of early training dynamics. We characterize how learning rates should scale under the gradient approximation in the first two steps, and prove that performing updates with this approximation yields a tractable surrogate loss with a tight, small
This paper leverages recent advancements in understanding neural network dynamics to provide a more precise characterization of early training behavior.
Improved theoretical understanding of learning rate optimization can lead to more efficient and stable training of large language models and other AI systems, impacting development costs and capabilities.
The ability to precisely characterize early training dynamics and optimal learning rate scaling provides a foundation for developing more robust and performant AI training algorithms.
- · AI researchers
- · Deep learning framework developers
- · AI compute providers
- · Hardware-limited AI companies (if algorithms become too complex)
More efficient training regimes could reduce the computational resources needed to achieve state-of-the-art AI performance.
This efficiency gain could lower barriers to entry for AI development, potentially democratizing access to advanced AI capabilities.
Reduced compute needs might mitigate some aspects of the energy bottleneck currently associated with large-scale AI training, affecting the compute supply chain.
This signal links to a primary source. Continuum Brief monitors and indexes it as part of the live intelligence stream — we do not republish source content.
Read at arXiv cs.LG