Cross-Trajectory Chimera Interventions Reveal Dissociable Roles of Weight Magnitude and Direction in Grokking
arXiv:2607.06628v1 Announce Type: new Abstract: Which properties of a partially trained network are causally portable to a different, independently trained network? Single-trajectory interventions show necessity within one run, not portability across runs. We introduce cross-trajectory chimera interventions: given two runs from different seeds, we split each weight vector into a norm and a unit direction, recombine one run's norm with the other's direction, and continue training. On two modular-arithmetic tasks that grok, the components dissociate. Direction carries a transferable, donor-speci
This research provides a more granular understanding of how neural networks learn and generalize, specifically addressing the 'grokking' phenomenon.
Understanding the dissociable roles of weight magnitude and direction could lead to more efficient and robust AI training, and potentially enable transfer learning between vastly different models.
Our understanding of neural network mechanics deepens, moving beyond monolithic views of weights to distinguishing transferable components crucial for generalization.
- · AI researchers
- · Deep learning framework developers
- · Companies investing in foundation models
- · Black-box AI development methodologies
Improved methods for network initialization and architectural design could emerge from this understanding.
Reduced computational costs for training and fine-tuning large models as more targeted transfer techniques become possible.
Acceleration in the development of truly generalized AI agents capable of learning from diverse experiences and transferring knowledge effectively.
This signal links to a primary source. Continuum Brief monitors and indexes it as part of the live intelligence stream — we do not republish source content.
Read at arXiv cs.LG