r/LocalLLaMA
· Communities
Launch script for GLM5.2
I am using the script below. Please advise on what i am doing wrong. GLM5.2 is crawing at 5-9 t/s decode and 60 t/s prefill. The machine is a EPYC 9654 with 768 GB of DDR5 4800 MHz RAM (~460Gbps bandwidth theoretical). This is CPU only inference. I do have a RTX pro 6000 Max Q on the machine, but that's only 96 GB vRAM