An NVIDIA server kept 96.1% to 98.2% of its AI text-generation throughput with confidential-computing protection switched on, according to the company's September 22 technical report. The test used eight Blackwell B200 GPUs and varied how many requests the server handled at once.
Confidential computing is meant to keep a model and its prompts protected while a server processes them. In NVIDIA's setup, that protection spans a confidential virtual machine, the GPUs and their links. The extra security steps can also change how an inference framework moves data and times work, so a normal GPU run is not a reliable measure of the protected run's speed.
NVIDIA compared confidential computing on and off on one DGX B200 with eight B200 GPUs, holding the model, framework and workload constant. It used DeepSeek-R1 in a low-precision NVFP4 format with 32K input tokens and 1K output tokens per request, and tested one, two, four, eight and 16 concurrent requests. The long input and output were chosen to make overhead visible rather than hide it behind heavy parallel traffic.
The mean time needed to generate each output token rose 1.2% to 4.3% with protection enabled. Both the speed and latency results come from NVIDIA's controlled test of this model and workload.
Protected memory changes the inference runtime
One cause of overhead is data transfer between the confidential virtual machine and the GPU. In this B200 configuration, transfers pass through an encrypted intermediate buffer, so the framework cannot rely on its usual way of copying data while other work continues. That method normally uses pinned host memory, which the operating system keeps in a fixed location during the transfer. NVIDIA says TensorRT LLM selects memory accordingly and moves repeated token readback to a worker thread so the main scheduler need not wait on every protected copy.
The framework also uses a GPU timer for kernel tuning, selecting efficient versions of GPU operations, under confidential computing because ordinary CUDA event timing was unstable in the tested configuration. NVIDIA points to merged TensorRT LLM code changes for the memory and timing adaptations. The report does not isolate how much each change contributes to the final result.
Sources
- NVIDIA technical report: confidential AI inference performance and test configuration
- TensorRT LLM code change: confidential-computing memory and asynchronous readback
- TensorRT LLM code change: global-timer based autotuning
- Image: NVIDIA DGX B200 eight-GPU board by Pokiiri, CC BY-SA 4.0; illustrative 2025 photograph, not the measured server
