Global Tech News For people figuring out what to build with AI.
AI

Researchers Fine-Tune a 32B Model on One H100 GPU

By ·

Several NVIDIA H100 PCIe graphics cards shown in use.

Researchers used a new optimizer to fully fine-tune Qwen3-32B, a 32-billion-parameter language model, on one Nvidia H100 GPU with 80GB of memory. In an arXiv preprint posted October 1, the authors report peak use of 69.6GB to 75.9GB across three standard language-understanding benchmarks.

The paper calls this full-parameter fine-tuning because all model weights remain trainable, although TACO does not change every matrix entry on every step. For large models, the training memory budget must also hold gradients, temporary activations and the optimizer’s saved state: a history used to guide each weight update. In AdamW, those per-weight records can take tens of gigabytes.

The new method, called TACO, reduces that history for matrix-shaped weights. At each step it first calculates gradients for the matrix, then keeps a compressed record of recent gradient signals for each column. A gradient estimates how changing a weight affects the training loss. TACO selects the retained signal with the largest absolute value in each column and changes that one weight in the opposite direction from the signal’s sign. Its history can retain up to 16 candidate values per column in FP8 (an 8-bit format), but each step still changes just one matrix weight per column. Scalar and vector parameters use AdamW.

In the paper’s model-scaling table, the three-seed average peak memory for Qwen3-32B was 69.6GB on SST-2, 71.6GB on RTE and 75.9GB on BoolQ. These runs used a batch size of four, sequences up to 1,024 tokens and the default 2,000-step schedule.

A separate OPT-13B comparison shows the memory difference against AdamW8bit. On SST-2, TACO used 0.159GB for optimizer state and 27.54GB at peak; AdamW8bit used 27.708GB and 80.60GB, respectively, in 2,000-step runs. For this comparison, the authors report about 85.5 decimal GB of total H100 memory and say their peak-memory figures use the same decimal units, placing the AdamW8bit result below that stated total. TACO’s validation accuracy was 94.22%, versus 95.25% for AdamW8bit, so the memory reduction was not a score win on that task.

The results come from the authors’ own arXiv preprint, with no independent reproduction reported there. The 32B result covers three benchmark tasks and a short fine-tuning schedule; it does not establish that arbitrary datasets, longer jobs or pretraining will fit on one H100.

Sources