Auto-fit vs tuned MoE offload: 564 → 1330 pp tok/s, unchanged decode (Qwen3.6-35B-A3B Q6 / RTX 3090)
TL;DR: On a Qwen3.6-35B-A3B Q6 setup sized for 64K context on a 24GB RTX 3090, spilling eight MoE expert layers to CPU freed enough VRAM to…
Latest
Final optimization: from ~10 tok/s to ~15 tok/s on DeepSeek-V4-Flash-0731 at 128K ctx – 1 RTX 3090
J'ai consacré beaucoup de temps à l'optimisation de DeepSeek-V4-Flash-0731 GGUF sur une seule RTX 3090. Mon exigence absolue pour chaque configuration était la suivante : Le…
Best llama cpp flags to run Deepseek-flash 0731
Hi all. These are my system specs: dual xeon e5 2696 v2 , 160gb DDR3 ram ECC(1600mhz), 3 gpus: 3060 12gb, p100 16gb, 3050 6gb. And…
Don’t forget to train and use your brain. Literally.
This is a screenshot from a recent Theo.gg video (Apple rant; tldr: he is realizing the lockdownness after...years.) and the fact he asked ChatGPT for an…
i just spent weeks rewriting my webUI from scratch, getting rid of all AI slop within the codebase and switching it over to a proper lightweight framework (alpine.js). i am now comfortable suggesting it as an alternative to openwebUI, librechat and the like! it is made for local models
[Fully open source under GPL3, made from the ground up for use with local models, no subscriptions, no corporate backing] When i first started this, it…
Qwen3.8-2.4T-A95B (aka Qwen3.8-Max) open release time: next wednesday
https://preview.redd.it/9zwlpcushphh1.png?width=972&format=png&auto=webp&s=18cb49c738caba9799d22d4916337c772baa830f https://modelscope.cn/models/Qwen/Qwen3.8-2.4T-A95B submitted by /u/HugeConsideration211 [link] [comments]
I get that AI labs need to make money, but zero-warning price spikes are a nightmare for production builds
Seen a ton of posts today about the DeepSeek API price hike. Half the feed is doom-posting, the other half is explaining basic GPU economics. Honestly,…
GLM/Qwen Appreciation Post
https://preview.redd.it/o6ik6qboeohh1.png?width=1134&format=png&auto=webp&s=4016f26c50c1d93bd3d0c7e880e9b55a2d75310f I have been running Qwen3.6 27b for a little while (mostly coding tasks) and recently trying out V4 flash 0731 in it's place. It was…
How many people in this sub try to train their own AI from scratch on their systems just for fun and to test out techniques from research papers?
As for me, I own a system with an RTX 5090, Ryzen 9 9950X3D2, and 64 GB of DDR5. Every time I see research come out…
Introducing BetterBench – more accurate PP and TPS measurement
I built this because the existing benchmarks were using random data and with MTP content types can vary a lot on what performance you see. 5%…