Global Tech News For people figuring out what to build with AI.
AI

Queueing Model Speeds AI Server Capacity Planning in Tests

By ·

Rows of server racks in the NOIRLab headquarters computer room in Tucson

Deciding how many servers an AI service needs can require searching through millions of hardware configurations. Researchers have proposed a faster way to plan a system that splits prompt processing from answer generation. In a replay of a public week-long workload, their method stayed within 3.8% of an exhaustive search's on-time request throughput under extreme demand, at the same modeled hardware budget.

The method, called SARA, targets services with a latency requirement: a request counts toward “goodput” only if it meets the specified response-time thresholds. In a split system, one pool of processors reads the prompt and produces the first token. It then transfers a stored record of that work, known as the KV cache, to a second pool that generates the remaining tokens. More prompt-processing capacity alone does not help if the transfer link or generation pool becomes the bottleneck.

SARA uses queueing models for each of those three stages to estimate how waiting times change with request lengths and available resources. It then searches for a mix of prompt processors, transfer bandwidth and generation processors that maximizes goodput within a cost limit. The model is designed to account for unusually long requests, which can make average latency a poor guide to whether most users meet the target.

For the seven-day trace replay, the researchers used LLaMA 3.1 8B, NVIDIA A100 GPUs and a modeled $35-an-hour resource budget, recalculating allocations hourly. The 3.8% figure is the largest gap reported in that trace replay under extreme demand, not an average across all models. A separate timing comparison found that SARA calculated allocations in 0.44 to 1.60 seconds across six model-and-workload settings; exhaustive enumeration took 619 to 2,440 seconds in those settings.

The team used measurements on A100 hardware to set and check parts of its latency model, but the reported resource-allocation gains come from modeled and replayed workloads, not a live service deployment. The paper also shows a boundary: with a very large model and a tight budget, keeping prompt processing and generation together could outperform the split designs.

Sources