In benchmark tests, a new serving engine called TokenRouter produced output tokens at 2.01 to 64.15 times the rate of the higher-throughput of two comparison systems, depending on the routing policy and workload. It is designed for tasks that switch language models while generating a single answer. The authors say their paper, submitted to arXiv on Oct. 8, was accepted at NeurIPS 2026.
Many current model routers make one choice for an entire prompt or conversation turn: send the request to a smaller model for lower cost or a larger one for more capability. Token-level routing makes a finer decision, allowing different models to generate different parts of one response. Earlier research has proposed such routing policies, but serving software built around one model at a time can leave GPUs waiting when requests move between models or arrive too irregularly to fill batches.
TokenRouter treats each model as an asynchronous subserver with its own scheduler and cache. Developers express when to change models, what context to pass, and how to resume through three functions: route, send and receive. When one request moves to another model, the first server preserves its state and continues processing its other requests rather than making the whole batch wait for the slower model.
A second feature, delayed batching, waits briefly for requests targeting the same model to collect. Later arrivals can join a fuller batch, reducing the average time between a request reaching a model server and beginning to run, though an early request waits longer by design. The authors derive a threshold for this trade-off with a throughput model; their analysis assumes routing probabilities stay stationary across decoding steps and requests make independent routing decisions.
The evaluation ran five routing algorithms on three workloads: short and long AIME 2024 reasoning tasks, plus multi-turn SWE-Smith software-agent trajectories. All tests used an eight-GPU A100-80G server. Comparisons included each algorithm's official implementation when available and an SGLang-based serving baseline. Across the 15 algorithm-workload combinations, TokenRouter reported producing output tokens 2.01 to 64.15 times as fast as the higher-throughput baseline. In a separate test of the existing R2R token-level routing method with eight simultaneous requests, the paper reports 372.48 output tokens per second for the full TokenRouter system, versus 134.9 for the R2R implementation.
These are author-reported results from the tested setups, not a universal speedup or evidence that routing improved answer quality. The baselines differ in implementation, and the official R-Stitch implementation was unavailable. TokenRouter's contribution is a runtime for executing routing methods, not a new language model or routing policy. Its value will depend on whether the serving gains hold for a deployment's own model mix, routing behavior and latency targets.
Image: Figure 2 from Fu et al., TokenRouter, CC BY 4.0.
Sources
TokenRouter paper on arXiv; authors' project page; official code repository.