r/LocalLLaMA
· Communities
I got tired of my 300GB model loads taking 5min on RPC. PR 26291 speeds it 300% to 1min30sec (4060ti+ddr4) + (4060ti+ddr5)
- On b10173 - "state":"loading" 4min54sec. - With this PR and GGML_RPC_LOAD_THREADS 12 - "state":"loading" 1min38sec The PR is close to ready, will need a docs change if they want to keep the new GGML_RPC_LOAD_THREADS variable.. and hopefully they take it with the client-side being solved and someone else cant take on