X · @teortaxesTex
· X / Twitter
What I'm most curious about is how this number changes Realistically, something like 15-20 GB?
What I'm most curious about is how this number changes Realistically, something like 15-20 GB?Hensen Juang: ~55–70B (back-of-envelope) active right if it keeps k2.5 shape (61 layers, ~2K expert hidden dim), 2.8T ÷ 896 experts implies ~50M params per expert per layer, so 16 active experts ≈ ~50B of expert compute per to