arXiv stat.ML
· Papers
No Subspace to Track: Non-Identifiability and Optimizer State in Low-Rank Training
arXiv:2607.05872v1 Announce Type: cross Abstract: Memory-efficient optimizers such as GaLore train large language models by projecting gradients onto a rank-r subspace recomputed every T steps, assuming this subspace is a slowly drifting object that can be tracked. We show that beyond a small reproducible core, there i