arXiv cs.AI
· Papers
Beyond Accuracy and Cost: Latency-Aware LLM Query Routing for Dynamic Workloads
arXiv:2607.18253v1 Announce Type: new Abstract: Modern language query routers improve inference efficiency by assigning each query to a model that balances response quality and monetary cost. However, current query routers are largely latency-agnostic and do not consider the generation latency experienced by queries at