arXiv:2606.18246v1 Announce Type: new Abstract: Scaling model size, specifically depth and width, has driven significant progress in transformer-based language models. However, most architectures maintain a constant width across all layers, allocating a fixed parameter and computation budget evenly despite different layers potentially playing distinct computational roles. In this work, we empirically investigate nonuniform capacity allocation across network depth by proposing a $\times$-shaped > <former consistently outperforms parameter-matched uniform baselines on language modeling loss. By
Source: arXiv cs.CL — read the full report at the original publisher.