A Function-Space Dichotomy for Compositional Learning: Exponential Sub-Optimality of the Neural Tangent Kernel
arXiv:2607.06382v1 Announce Type: cross Abstract: A persistent empirical observation is that trained neural networks outperform their neural tangent kernel (NTK) limit on tasks with compositional structure, yet a quantitative account of $\textbf{when}$ and $\textbf{by how much}$ has been lacking. Working on the unit circle, we give such an account through a dichotomy between two complexity measures of the target: its $\textbf{Fourier complexity}$, which controls NTK kernel regression, and its $\textbf{architectural complexity}$, which controls learning over depth-$L$, width-$w$ ReLU networks w
This research provides a quantitative explanation for the performance gap between neural networks and their theoretical NTK limits, addressing a long-standing empirical observation as AI capabilities rapidly advance.
It quantifies the architectural advantage of modern neural networks over simpler kernel methods, guiding future AI development towards more efficient and powerful compositional learning architectures.
Our understanding of neural network training dynamics is refined, emphasizing the critical role of network architecture beyond simple kernel approximations for complex tasks.
- · AI researchers focused on novel architectures
- · Developers of advanced AI models
- · Enterprises leveraging compositional AI for complex problems
- · Simpler kernel-based machine learning approaches
Increased investment in research exploring architectural complexities for AI performance gains.
Development of new AI models that explicitly exploit this architectural advantage for improved efficiency and capability.
Accelerated deployment of AI in domains requiring high compositional reasoning, potentially leading to new applications.
This signal links to a primary source. Continuum Brief monitors and indexes it as part of the live intelligence stream — we do not republish source content.
Read at arXiv cs.LG