X · @teortaxesTex
· X / Twitter
RT Alexey Fateev: I reproduced an @NVIDIAAI paper in real vLLM on 2xRTX 3090s. A small model does the prefill, a completely different bigger model ans…
RT Alexey FateevI reproduced an @NVIDIAAI paper in real vLLM on 2xRTX 3090s. A small model does the prefill, a completely different bigger model answers from its KV cache.TTFT at 32K context drops 2.22x, 94% of the quality stays. And it needs no fork, the whole thing is one KV connector on stock vLLM.How it works and w