r/LocalLLaMA
· Communities
[Paper] Automated Tensor Scheduling for Hybrid CPU-GPU LLM Inference on Consumer Devices
Running large language models on consumer devices such as laptops and desktops is challenging because model weights often exceed GPU memory capacity, making offloading inference necessary to extend effective model capacity with CPU memory. Existing offloading systems, however, typically rely on coarse layer-level or ex