Optimal Initialization in Depth: Lyapunov Initialization and Limit Theorems for Deep Leaky ReLU Networks
arXiv:2602.10949v2 Announce Type: replace-cross Abstract: Effective initialization in deep networks requires an understanding of random neural networks. In this work, a rigorous probabilistic analysis of deep bias-free random Leaky ReLU networks is provided. We prove a Law of Large Numbers and a Central Limit Theorem for the logarithm of the norm of network activations, establishing that, as the number of layers increases, their growth is governed by a parameter called the Lyapunov exponent. This parameter characterizes a sharp phase transition between vanishing and exploding activations, and
This research is published as the field of deep learning continues to push the boundaries of network depth and complexity, making optimal initialization increasingly critical for stable and efficient training.
Understanding the probabilistic behavior and growth dynamics of deep neural networks is fundamental for designing more stable and higher-performing AI models, directly impacting the maturity and reliability of AI systems.
This theoretical work provides a rigorous framework for understanding activation growth in deep Leaky ReLU networks which, if applied, could lead to more robust initialization techniques and improved training stability in deep learning.
- · AI researchers
- · Deep learning framework developers
- · Companies investing in complex AI models
- · Trial-and-error network initialization methods
Improved theoretical understanding of deep neural network training dynamics.
Development of new, more effective initialization strategies for deep learning models.
Accelerated development and deployment of larger, more stable deep neural networks across various AI applications.
This signal links to a primary source. Continuum Brief monitors and indexes it as part of the live intelligence stream — we do not republish source content.
Read at arXiv cs.LG