And another open-weight release. Nemotron 3 Ultra has an ultra impressive capability:efficiency ratio! Design-wise, it carries forward the Mamba-2-attention hybrid stack and LatentMoE introduced in the…
It's been a while! 4 nice additions to the open-weight local-LLM-on-consumer-hardware ecosystem:
The MiniMax M2 series was one of the most widely used open-weight LLM series earlier this year. Now, we got a technical report with some interesting…
Added a DeepSeek Sparse Attention (DSA) from-scratch implementation to my LLMs-from-scratch repo thanks to an awesome new reader contrib. With motivation, overview, and GPT-style model reference…
It's been *almost* a bit quiet around LLM architecture releases in the past two weeks 😅Interesting tidbit is the parallel block design. Via the Cmd-A the…
New article: a visual tour of recent LLM architecture advances, from Gemma 4 to DeepSeek V4.I focus on long-context efficiency tweaks like KV sharing, per-layer embeddings,…
A little talk on what we can learn from implementing LLM architectures from scratch in Python and PyTorch. And how I approach new open-weight models, compare…
Interesting paper. What I like about this is that it is a relatively low-commitment attention modification.I.e., one can use it during most of training, switch back…
Back from a little family break! Lots has happened, and I’m planning to do a deeper dive into the most interesting architectural components (soon).Btw, are there…