r/LocalLLaMA
· Communities
I ported vLLM’s serving stack to C++20: 66 MiB binary, no Python at inference, output checked token-for-token against vLLM
I'm the author, so discount the enthusiasm accordingly. This is an unaffiliated community port, not endorsed by the vLLM project, which it uses to verify its correctness. What started it: I love vLLM, but a vLLM install here is 9.1 GiB of virtualenv, and I wanted to embed inference inside other software, on machines wh