r/LocalLLaMA
· Communities
PSA for anyone with multiple V620’s or other gfx1030 cards having problems making llama.cpp tensor split work — set "-ub 384" and -b to a multiple of that depending on number of GPUs
Basically what the title says. For me, it would always crash and burn trying to use tensor split. Apparently, there's some bug where GPU memory gets corrupted with the default microbatch (512) or higher. I will be opening an issue report on the llama.cpp GitHub if there isn't already one related to this. I'm using ROCm