r/LocalLLaMA
· Communities
Going full linear or nearly there (almost no kv cache, always bf16)
I just checked the implications of the HOLA architecture and it seems a dream: - very tiny KV cache (1 Gb is likely 5/10M context or so) - better perplexity than full attention by a factor of 16% . To understand how much this is, we test quantization and 0.1/0.3 is enough to say a model isn't working perfectly anymore.