arXiv cs.CL
· Papers
Cracks in the Foundation: Seemingly Minor Architectural Choices Impact Long Context Extension
arXiv:2608.10296v1 Announce Type: new Abstract: One might imagine that architectural variations within the dense transformer paradigm have a limited effect on accuracy. However, we demonstrate that this is not the case in the long context setting. Specifically, we show that a set of four minor architectural decisions -