arXiv:2607.03948v1 Announce Type: new Abstract: We study the online routing problem in large language model serving, where requests arrive sequentially and must be dispatched to parallel decode workers under tight batch-size and KV-cache constraints. Unlike widely used routing heuristics that are not tied to explicit service-level objectives (SLOs) and offer limited control over latency-throughput trade-offs, we introduce a multi-objective optimization framework that formulates routing as an online linear programming with interpretable decision rewards. We apply an efficient bid-price control
The rapid scaling of large language models and their deployment in real-world serving environments is exposing critical challenges in efficient and optimized resource allocation.
Optimizing LLM serving has direct implications for the cost, latency, and throughput of AI services, impacting both providers and consumers of advanced AI capabilities.
This research introduces a more principled, multi-objective optimization approach to LLM routing, moving beyond ad-hoc heuristics towards explicit service-level objective management.
- · AI infrastructure providers
- · Cloud computing platforms
- · Companies deploying large language models
- · AI researchers and developers
- · Companies relying on inefficient LLM serving infrastructure
- · Legacy AI inference optimization techniques
Improved efficiency and lower costs for large language model inference.
Accelerated development and deployment of more complex and accessible AI applications.
Potentially democratizing access to powerful LLMs by making them more economically viable at scale.
This signal links to a primary source. Continuum Brief monitors and indexes it as part of the live intelligence stream — we do not republish source content.
Read at arXiv cs.AI