LessWrong AI
· Communities
MSE loss does not generate superposition
If you're training any type of toy model of superposition, Mean Squared Error (MSE) loss is unusually bad. [1] Related work We aren't the first to notice that MSE loss doesn't work. In Toy Models of Superposition the effective loss function is mjx-container[jax="CHTML"] { line-height: 0; } mjx-container [space="1"] { m