A Tencent and Peking University team has built WorldCrafter, a video world model that keeps a compact memory of places it has already generated. In tests across 725 camera paths, the model reproduced revisited views more consistently than eight comparison systems and a faster variant generated 16 frames per second on four GPUs.
Interactive video models generate the next view as a user moves a virtual camera. Their short context window creates a basic problem: after the camera turns away, an earlier room, object or character can return with a different appearance. Keeping every old frame in attention would make the computation grow with the journey, while storing explicit 3D geometry can become brittle when the scene contains motion.
A memory shaped by the next camera view
WorldCrafter stores selected earlier frames in an implicit 3D-aware representation. “Implicit” means the system preserves learned scene features instead of constructing a visible mesh or depth map. A memory encoder starts from a model trained to reconstruct scenes from multiple views, then learns jointly with the video generator so that its compact representation retains both layout and appearance.
The system does not pass that entire representation to the generator. Before each new video segment, it looks at the requested camera path and reads out a fixed number of memory tokens, compact units of scene information, that are relevant to those coming viewpoints. It also keeps a short window of recent frames for local motion. This division lets the older memory recover a place after a long absence while the recent context carries movement from one segment to the next.
When the accumulated history becomes large, WorldCrafter retains the latest frame and selects additional frames whose combined fields of view cover the coming camera path. The encoder input and the generator's memory-token budget therefore stay fixed instead of growing with every generated segment.
Revisits improve without sacrificing the whole video
The researchers tested 145 static and dynamic scenes with five camera trajectories per scene. On matched first-visit and revisit frames, WorldCrafter recorded an LPIPS distance of 0.255, compared with 0.487 for Lyra 2.0, the strongest baseline on that measure; lower values indicate closer image similarity. WorldCrafter-fast, trained to produce video with fewer refinement steps through a process called distillation, reached 0.186. The same evaluation used reconstructed camera paths to check whether the generated motion followed the requested trajectory.
An ablation under the same memory-token budget helps isolate the contribution. Replacing the proposed memory with retrieved context frames raised LPIPS from 0.255 to 0.497. Freezing the memory encoder, removing pose-guided readout or using pairwise similarity for history retrieval also weakened revisit consistency. Those comparisons support the value of adapting the memory to the generator and allocating it around the next viewpoint, rather than merely adding more historical images.
The team also reports an overall VBench quality score of 81.910, the highest among the evaluated systems, while WorldCrafter-fast scored 80.285. This aggregate combines checks such as subject and background consistency, motion smoothness and visual quality; it measures the generated videos more broadly than the matching of revisited views. These are results from the authors' own benchmark protocol, not an independent deployment. The paper does not test whether the memory can support accurate physical simulation or reliable downstream decisions, and long interactive sessions can still accumulate errors from generated frames.
WorldCrafter narrows a specific gap between short video clips and explorable generated environments: a user can leave a view and return without forcing the model to reread its full history. The released project includes code and a model card, but the evidence so far is a controlled generation study across the team's selected scenes and paths.
Sources
- Yu et al., “WorldCrafter: Consistent Video World Model with Implicit 3D-aware Memory,” arXiv:2609.24984 (submitted September 21, 2026)
- WorldCrafter project page, code, model card and visual demonstrations
- Lead image: Figure 1 of the WorldCrafter paper, reused under the paper's CC BY 4.0 license.