r/LocalLLaMA
· Communities
ExLlamaV3 v1.0.0 – Major Performance Upgrades
After over a year in development, ExLlamaV3 has had its first production release. Turboderp has been pulling 10 hour days with Fable to bring us this massive batch of improvements. Check out detailed performance metrics and a little write-up from him here. Some of the biggest changes: Removed flash-attention-2 and xfor