Is turboquant any good?
I know im late to the party. I was thinking since some time has passed, has turboquant matured enough to be used? Do any of you…
Latest
I built a Triton backend for Falcon3-10B-1.58bit: 97.5 tok/s decode on an RTX 5070
Hi r/LocalLLaMA — I’m sharing an experimental GPU-only inference backend and looking for independent reproductions, not just stars. Model: tiiuae/Falcon3-10B-Instruct-1.58bit GPU: NVIDIA RTX 5070 Batch: 1…
BeeLlama.cpp v0.4.1: KVarN, KV precision tail, q2_0-q3_1 KV cache, improved support. KLD benchmarks: tail 1024 makes kvarn5 and q6_0 match q8_0, for much less VRAM
TL;DR llama.cpp fork with more KV cache quantization features, with all claims supported by benchmarks: KVarN, KV cache precision tail, additional types of standard KV cache…
90 agentic bakeoff runs: ThinkingCap vs Fable Fusion vs stock Qwen3.6-27B
Last week someone here said ThinkingCap and Fable Fusion "really do beat the OG" for agentic work, so I ran it: 6 self-grading tasks, 5 reps,…
World’s First(?) Underwhelming AMD Ryzen AI Halo Cluster
LTT Labs recently received the Linux version of the AMD Ryzen AI Halo for testing, but it turns out that AMD had intended to send the…
Do you want new Gemma?
https://x.com/i/status/2081398564345802934 submitted by /u/jacek2023 [link] [comments]
DGX Spark or M5 Max MacBook Pro 128GB RAM?
Title, for local inference and training, please. submitted by /u/Desperate_Tea304 [link] [comments]
Local-first LLM pipeline tracer — @trace on any function, dashboard at localhost. Feedback welcome.
Hey r/LocalLLaMA — maintainer here, obviously biased. OpenSmith is an open-source Python tracing tool for LLM pipelines. The idea: drop u/trace on any function, run opensmith…
Mixture of Sexperts
The server rack was humming at maximum capacity, fan speeds pegged at 100%, and thermal throttling was the absolute last thing on the Router’s mind. "Give…
Sources: OpenAI and Anthropic quietly lobby Washington regulators to restrict open-source AI models, even as Sam Altman publicly says he supports open source AI
submitted by /u/pscoutou [link] [comments]