Efficient Training-Free Online Routing for High-Volume Multi-LLM Serving

Sandeep Silwal, Fangzhou Wu

arXiv · PDF · Open in the Atlas

Code that ran

We lifted 1 functions out of this paper's own repositories and ran 1 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.

Repositories linked to this paper

• fzwark/PORT

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

Increasing demand for Large Language Models (LLMs) services imposes substantial deployment and computation costs on providers. LLM routing offers a cost-efficient solution by directing queries to the optimal LLM based on model and query features. However, existing works primarily focus on offline scenarios and struggle to adapt to online settings with high query volume and constrained token budgets. In this work, we introduce the first training-free algorithm for online routing scenarios. Our algorithm leverages approximate nearest neighbor search to efficiently estimate query features and performs a one-time optimization over a small set of initial queries to learn a routing strategy that guides future routing. We provide theoretical guarantees demonstrating that our algorithm achieves a competitive ratio of 1 -o(1) under natural assumptions, which is further validated by extensive experiments across 3 benchmark datasets and 8 baselines, showing an average improvement of 3.55× in overall performance, 1.85× in cost efficiency, and nearly 4.25× in throughput. Our code is available at https://github.com/fzwark/PORT.

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2509.02718")
get_code_for_paper("2509.02718")
have("2509.02718")

Connect an agent — have() is free.