r/MachineLearning August 12, 2026 · Communities

The Loss Does Not See the Basis, But Adam Does [R]

In a factored model W = UV^T, the loss is invariant to rotations (U,V) → (UQ, VQ). Gradient Descent (GD) respects this property. Adam's per-coordinate second moment does not, because it depends on the specific basis in which the factors are written. The claim is that this single property dictates whether optimizers ret