A system called GAE gives an AI image generator a compact internal representation of a scene's three-dimensional structure. This representation, called a latent, comes from the features of a 3D perception model. The same compact latent can be decoded into images, depth, camera rays and point maps, giving generation and perception a shared representation instead of adding geometry after image synthesis.
Video generators can render plausible frames while objects drift between viewpoints or the recovered camera path departs from the requested motion. Pixel-based latent spaces preserve appearance, and semantic representation autoencoders preserve high-level content, but neither makes scene geometry directly readable. The GAE researchers treat that mismatch as a representation problem.
The system starts with the frozen encoder and geometry head from Depth Anything 3, or DA3. Its encoder produces four levels of features that capture complementary local, semantic and cross-view information. GAE normalizes and combines the hierarchy, compresses it from thousands of channels into a 64- or 128-channel spatial grid, then reconstructs all four levels for the original frozen geometry head. A separate decoder produces RGB images from the same bottleneck.
Freezing the geometry head matters because it cannot learn to compensate for information lost in compression. The latent must retain geometry that the existing DA3 head can still read as depth, camera rays and point maps. Token-level alignment makes the space easier for a flow model to traverse, while a second objective preserves relationships among patches that token alignment alone weakened.
A matched generator isolates the effect of the latent
The researchers compared GAE with single-image and video pixel autoencoders, a semantic representation autoencoder and raw DA3 feature layers. Each controlled variant used the same flow-model family, camera conditioning, training budget and sampling procedure. Evaluation used 64 held-out scenes from each of RealEstate10K and DL3DV, nine views per scene, one reference view, 50 Euler sampling steps and classifier-free guidance set to 2.
On camera-conditioned generation, the 64-channel GAE reduced Fréchet Video Distance, a distributional measure where lower is better, from the best non-GAE controlled result of 258.6 to 225.7 on RealEstate10K. On DL3DV it lowered the best competing score from 373.2 to 287.0. Those changes correspond to the paper's reported reductions of 12.7% and 23.1%.
A separate model, VGGT, recovered camera paths from the generated views to check their three-dimensional coherence. The researchers measured departures from the intended paths after aligning position, orientation and scale. GAE-64 reduced that error by 52.8% on RealEstate10K and 23.3% on DL3DV relative to the best non-GAE controlled comparison. Separate geometry checks generally supported the same direction, although Gen3R retained the best MEt3R consistency score on DL3DV.
Released model differs from the benchmark version
The compact codec also improved reconstruction of geometry from real frames. Compared with the raw DA3 L0 feature baseline, both GAE variants reduced depth, point-map and camera errors across the two datasets while retaining similar or better RGB reconstruction. That combination is the central technical change: the bottleneck is small enough for one standard flow model, yet still reconstructs the feature hierarchy expected by a frozen geometry decoder.
The released repository includes inference, training and evaluation code plus GAE-64 and GAE-128 codec weights. It also publishes a continued-training flow checkpoint and a one-billion-parameter GAE-D64 model. The authors warn that this released flow model used higher resolution, more frames and more training data than the submission-time model behind the paper's generation tables. Running the public evaluation scripts therefore will not reproduce Tables 5 through 7 exactly; the original research flow checkpoint is not included.
Other limits remain. Geometry targets used while training the codec come from the frozen DA3 head rather than direct ground truth. The main numerical comparisons are the authors' controlled experiments on two datasets, not independent replications or open-world video tests. The large-scale higher-resolution and 81-view examples use a separately trained model and are presented qualitatively, so they should not be mixed with the matched benchmark.
Sources
- GAE paper record and abstract, submitted September 21, 2026
- Full paper with method, controlled benchmarks and limitations
- Official project page with generated examples
- Official code, weights and checkpoint-provenance notes
- Lead image: Figure 1 from the paper, published under its CC BY 4.0 license.