A paper posted on September 25 describes EAServe, which batches arriving multimodal requests and uses spare compute on the encoding GPU for some prompt processing. In tests on one eight-GPU server, it handled more requests per second than two serving baselines while meeting the researchers’ latency targets. The work is listed in the PACT 2026 program for October 19–22.
Unlike a text-only model, a multimodal large language model first encodes an image, video, or audio clip into data the language model can use. The server then processes the prompt, a stage called prefill, before generating the answer token by token, or decode. The paper says existing systems can run encoding one request at a time, leaving the encoder’s GPU underused while later stages wait for more input.
EAServe groups arriving requests into small, load-adaptive batches instead of relying on a fixed batch or timeout. It can also route some prompt processing to spare compute on the encoding GPU. Software changes how the GPU’s processing units are divided between encoding and local prompt work as batches grow or shrink. A separate configuration search picks the GPU allocation, batch limit, and offload share for each tested workload.
At the paper’s moderate latency tier, EAServe’s goodput—the number of requests per second meeting both 99th-percentile time-to-first-token and time-per-output-token limits—reached 1.93 requests per second for image input, compared with 1.11 for vLLM and 0.95 for NVIDIA Dynamo. On a separate speech workload, it reached 4.17 requests per second, versus 2.77 and 0.98. These are separate model and modality tests, not a comparison between image and audio on one workload.
The evaluation used LLaVA for images, Qwen2.5-VL for video, and Ultravox for audio on a server with eight NVIDIA RTX 6000 Ada GPUs. Each run processed 500 requests at controlled arrival rates; the reported figures are medians across three runs. Image and video inputs were synthetic, while audio samples came from LibriSpeech.
The results show a serving improvement under the paper’s tested configurations, not a production guarantee. The evaluation used one server and did not test multi-node deployments or cross-rack transfers. Whether the gains hold across other hardware, models, and real customer traffic remains open.
Sources
- EAServe paper, arXiv:2609.31551
- PACT 2026 technical program
- Figure: Figure 5 from the paper, licensed under CC BY 4.0.