arXiv cs.LG
· Papers
Multipole Semantic Attention: A Fast Approximation of Softmax Attention for Pretraining
arXiv:2509.10406v4 Announce Type: replace Abstract: Pretraining transformers on long sequences (entire code repositories, collections of related documents) is bottlenecked by quadratic attention costs. We present Multipole Semantic Attention (MuSe), which accelerates 64k-context pretraining by 36% while matching baseli