A recurrent vision transformer matched a standard model’s ImageNet accuracy while storing 73% fewer parameters, according to a new arXiv paper. The researchers’ reViT-B/16 scored 83.0% on ImageNet-1k with 23.6 million parameters, compared with 82.8% and 86.6 million for DeiT III-B/16. Both used about 34 billion floating-point operations per image, so the clearest gain is a smaller model rather than less computation.
Most vision transformers process an image through a stack of blocks, each with its own learned weights. reViT instead runs one Transformer block repeatedly. It shares the attention and normalization layers, then changes the feed-forward network, or FFN, at each depth. Attention lets image patches exchange information; the FFN then transforms each patch’s features.
To vary that FFN, reViT stores four learned FFN experts. At each step, a small router uses how far the model is through its repeated sequence to form a weighted blend of their parameters. The resulting dense FFN runs on all image patches at that step. The router does not send different patches to different experts, and adding experts increases stored capacity without adding another FFN calculation at a fixed depth.
In the supervised ImageNet-1k comparison, the four-expert reViT-B/16 reached 83.0% top-1 accuracy, against 82.8% for DeiT III-B/16. Top-1 counts a prediction as correct when the model’s highest-scoring class is the right one. The models used 34.3 and 33.9 GFLOPs per image, respectively. Their parameter counts were 23.6 million and 86.6 million. The paper reports reViT’s result as the mean of three training runs under a shared 300-epoch recipe.
The authors also measured a smaller reViT-S/16 on one NVIDIA H100. Its dynamic model graph was 3.6 times smaller and used 48% less total runtime memory at batch size one, but took 0.78 milliseconds per image versus 0.69 milliseconds for DeiT III-S/16. At batch size 64, the latency gap widened to 7.26 versus 5.22 milliseconds, while total memory was similar. The tests used a direct PyTorch implementation with generic weight-combination operations; the authors say caching or a custom kernel could change those results. For a fixed depth, the blended weights can also be computed ahead of time, trading the compact dynamic representation for a larger conventional model graph.
A separately trained, depth-flexible version used one checkpoint at several tested depths. Its ImageNet score rose from 82.6% at eight recurrent steps to 83.7% at 16, then stayed close at 24. More steps also mean more computation. The results come from an arXiv preprint and controlled experiments; they do not establish performance across production vision systems.
Illustration: dvgodoy / Wikimedia Commons, CC BY 4.0. The diagram shows a standard Transformer, not the reViT model.
Sources
The reViT paper and experimental tables; DeiT III comparison paper; Wikimedia Commons image and license details.
