X · @teortaxesTex
· X / Twitter
RT Teortaxes▶️ (DeepSeek 推特🐋铁粉 2023 – ∞): Re @meowbooksj the design space of sparsity is barely explored Like, why don't we use MLP routers…
RT Teortaxes▶️ (DeepSeek 推特🐋铁粉 2023 – ∞)Re @meowbooksj the design space of sparsity is barely exploredLike, why don't we use MLP routers in MoEs? Because le slow? How much does this cost in avoidable FFN redundancy?