New article: a visual tour of recent LLM architecture advances, from Gemma 4 to DeepSeek V4.I focus on long-context efficiency tweaks like KV sharing, per-layer embeddings,…
A little talk on what we can learn from implementing LLM architectures from scratch in Python and PyTorch. And how I approach new open-weight models, compare…
Interesting paper. What I like about this is that it is a relatively low-commitment attention modification.I.e., one can use it during most of training, switch back…
Back from a little family break! Lots has happened, and I’m planning to do a deeper dive into the most interesting architectural components (soon).Btw, are there…