Anyone else amped up over Qwen 3.8?
I’ve been using 3.6 27B Q4, and that quant is fast on an M5. The code has been average, but consistently “good enough.” And, after a…
Latest
Qwen 35B-A3B MoE vs 27B dense in local coding tests: ~4× faster, much smaller quality gap than I expected
I compared Qwen 35B-A3B MoE against Qwen 27B dense on a series of local coding-maintenance tasks. On my R9700/llama.cpp setup, the MoE model generated about 3.9×…
4x2080Ti 22GB tensor parallel possible?
Hi there, I have a setup with 4x 2080Ti 22GB, but am not able to see much benefits in running models larger than 16-20GB in speed.…
PSA for anyone with multiple V620’s or other gfx1030 cards having problems making llama.cpp tensor split work — set "-ub 384" and -b to a multiple of that depending on number of GPUs
Basically what the title says. For me, it would always crash and burn trying to use tensor split. Apparently, there's some bug where GPU memory gets…
I got tired of my 300GB model loads taking 5min on RPC. PR 26291 speeds it 300% to 1min30sec (4060ti+ddr4) + (4060ti+ddr5)
- On b10173 - "state":"loading" 4min54sec. - With this PR and GGML_RPC_LOAD_THREADS 12 - "state":"loading" 1min38sec The PR is close to ready, will need a docs…
BeeLLama issues
Tried to use Beellama , and using the kvarn6 flag, i notice that its in llama-server --help but its not working. I must be doing something…
Is anyone else finding DeepSeek-V4-Flash unreliable for non-coding tasks?
(I am not a native speaker, written by myself, so please bear with me) I really want to like DeepSeek-V4-Flash-0731. But it has serious flaws that…
U.S. Department of Energy Launches the Genesis Open Models Initiative and, with Arcee, Unveils Genesis-Science-1 — Its First Open-Weight Model for Scientific Research
submitted by /u/johnnyApplePRNG [link] [comments]
Serving Deepseek v4 Flash 0731 on 2x DGX Spark — 5-7 GB OS headroom, what would you do to lower VRAM usage and increase OS available RAM?
Hey all, I'm serving DSv4Flash 0731 on a cluster of 2x DGX Sparks but am running into constant issues with having almost no RAM (unified memory)…
Qwen 3.6 27B flags/settings in llama.cpp
I run the following on a 5090 and have been okay with its performance, it does most things somewhere 80-100 t/s, though that can slow down…