HF Daily Papers
· Papers
GORGO: Online Tuning for Cross-Region Network-Aware LLM Serving
Increasingly, LLM inference services proxy client requests to engine replicas distributed globally. Load-balancing policies must jointly account for factors including KV-cache locality, replica load, and variable network latency when optimizing for metrics like latency and TTFT. However, existing sy