arXiv cs.LG
· Papers
Untied Ulysses: Memory-Efficient Context Parallelism via Headwise Chunking
arXiv:2602.21196v2 Announce Type: replace Abstract: Efficiently processing long sequences with Transformer models usually requires splitting the computations across accelerators via context parallelism. The dominant approaches in this family of methods, such as Ring Attention or DeepSpeed Ulysses, enable scaling over t