X · @rasbt
· X / Twitter
Interesting surprise drop from Thinky! The Inkling model looks pretty solid on benchmarks, and it has some little surprises in its architecture: – Sma…
Interesting surprise drop from Thinky! The Inkling model looks pretty solid on benchmarks, and it has some little surprises in its architecture:- Small conv layers in several places- An RMSNorm for the embeddings (before the block RMSNorm)- Rel. position bias instead of RoPEelie: first open weight thinking machine mode