Has anyone else found vLLM outputs noticeably worse than llama.cpp for the same model?
I'm wondering if anyone else has come across this. I've tested the same model on llama.cpp and vLLM with similar settings and quantizations. The performance and…
Latest
Sipp – an open-source library for in-browser inference built on llama.cpp
GitHub: https://github.com/noumena-labs/Sipp submitted by /u/lordhiggsboson [link] [comments]
Big News for AMD / Strix Halo+ Owners
Admittedly this is news for me, but I'm hoping it could be of some use to others here as well! So, THE NPU IS USABLE!! I've…
Build a LLM from Scratch using MLX
You probably have a burning desire to grasp the inner workings of LLMs. By now, terms like Attention, Transformers, and Tokenizers are likely ringing in your…
My micro-benchmark: how good are LLMs at simulating wetting behaviour?
Example surfaces that LLMs are asked to simulate, showing simulated liquid (green) shaped by solid constraints (orange). Overall score, pass count, and recorded token/cost totals for…
OpenAI and Broadcom unveil LLM-optimized inference chip
https://openai.com/index/openai-broadcom-jalapeno-inference-chip/ Quoted from the start of the blog post: Early testing shows that the first-generation accelerator will deliver performance per watt substantially better than current state-of-the-art…
The Swiss Federal Supreme Court is evaluating Heretic
“Oh no, are they banning abliterated models now?!?” If that was your first thought when you read the title I can’t blame you. But that’s actually…
I did some model hacks, and got GLM5.2 from about 2.5 tok/s to >50 tok/s on my GH200 system.
G'day. This is part 3 on my Local LLM adventures. I have a crazy system hacked server-to-desktop system: Component Spec GPUs 2x Hopper H100, 96 GB…
The Bank of Korea just released a report about AI productivity
I am sorry for sharing an article from a Korean website that you might not be familiar with. But South Korea is the only country currently…
Nex-N2-Mini-Ultra-Uncensored-Heretic Is Out Now, an Agentic Model With Agentic Thinking Now Uncensored With 5/100 Refusals and 0.0020 KLD, Available in Safetensors and GGUF Formats!
Safetensors: https://huggingface.co/llmfan46/Nex-N2-mini-ultra-uncensored-heretic GGUFs: https://huggingface.co/llmfan46/Nex-N2-mini-ultra-uncensored-heretic-GGUF Find all my models here: HuggingFace-LLMFan46 If you like my work and find my models useful, then I would really appreciate if…