Apparently you can skip entire transformer blocks at load time with minimal performance impact
Following recent (very cool) papers, I implemented this as a --skip-layers flag to a llama.cpp fork, so it just never instantiates the blocks you tell it…
Latest
DeepSeek V4 official version will be launch on mid-July
https://preview.redd.it/n7rwh262b7ah1.jpg?width=1024&format=pjpg&auto=webp&s=33d775b456843cd2dbd458de89384a6a7d6d87d1 Source: Email sent from deepseek (email only available for chinese user) used gpt image 2 translate image into english submitted by /u/External_Mood4719 [link] [comments]
Slow performance Unsloth Gemma 12B Q8
I recently replaced GPT-OSS 20B Q4 with Gemma 4 12B Q8 but i went from roughly 70 t/s to 10 t/s. Am I doing something wrong?…
Best way to test models at different quants before buying GPUs
I am determined to buy a bunch of GPUs. However, I would like to test the performance of models such as GLM-5.2 at different quantisation levels…
DeepSeek V4 by am17an · Pull Request #24162 · ggml-org/llama.cpp
now you can run DeepSeek V4 locally submitted by /u/jacek2023 [link] [comments]
GLM 5.2 Q1_S vs Qwen 27B Q8
TL;DR; GLM-5.2 Q1_S beats Qwen 3.6 27B Q8, both run at KV Q8 Disclaimer: This is a hobby/amateur comparison with n=1, so go easy on it.…
LibreChat or OpenWebUI ?
Hello, I have a friend that while technical, it doesn't know too much about AI, I've helped them with the infrastructure setting and that works like…
MiCA is now part of Hugging Face PEFT
Glad to share that MiCA, short for Minor Component Adaptation, has now been merged into the HuggingFace PEFT library. It is not yet included in the…
AMD MI210 64GB vs DCU K100 64GB
On the Chinese eBay there is a many DCU K100 64 GB GPU available for a very attractive price, between 6000 RMB and 15 000 (air…
Update: First Manual Results from Testing Procedural Skill Transfer in Small Models
Yesterday I posted an idea for testing whether a large model can transfer some of its procedural skill to a smaller model without fine-tuning. The short…