arXiv cs.CL
· Papers
Physics of Language Models: Part 4.1, Architecture Design and the Magic of Canon Layers
arXiv:2512.17351v2 Announce Type: replace Abstract: Understanding architectural differences in language models is challenging, especially at academic-scale pretraining (e.g., 1.3B parameters, 100B tokens), where results are often dominated by noise and randomness. To overcome this, we introduce controlled synthetic pre