X · @teortaxesTex
· X / Twitter
RT Artur Chakhvadze: RoPE is equivalent to KDA with imaginary log-decay factors ALiBi is KDA with real log-decay factors State-space models don’t nee…
RT Artur ChakhvadzeRoPE is equivalent to KDA with imaginary log-decay factorsALiBi is KDA with real log-decay factorsState-space models don’t need positional embeddings because positional embeddings are baked into the recurrencetokenbender: kimi guys noped the fuck out of rope at this scale.no positional encoding anywh