Researchers at Carnegie Mellon University and Meta have developed TrackEverything, a system that follows the movement of visible surfaces through a video in three dimensions. It reuses a map of the scene instead of repeatedly storing the same surfaces, addressing a memory bottleneck that makes detailed tracking difficult in long videos.
For a computer, following motion means estimating where points on an object go from one frame to the next. A few selected points can show where an object moves. Dense tracking follows points across the visible scene, capturing more detail but creating much more information to process. Recording the same stationary wall again in every frame, for example, adds work without adding a new surface.
TrackEverything tackles that duplication by placing observations in a shared 3D map. It processes video in 16-frame windows, uses depth information to locate surfaces and merges repeated observations at each window boundary. It also separates stationary points from moving ones, calculating detailed paths only for the points that move.
A component called 3D WAFT reduces another source of memory use. Rather than storing a large set of possible matches between image features, it samples features at a point's starting position and predicted destination. Together, these changes aim to make memory use depend more on newly revealed parts of the scene than on video length alone.
In their Sept. 24 preprint, the authors report tracking all visible points in videos longer than 1,000 frames within 40 GB of GPU memory. A separate scaling plot runs through 900 frames, where the system stays below 30 GB on one Nvidia L40S GPU. That plot measures memory and processing-time trends, not accuracy: some comparison methods follow only 384 selected points, while TrackEverything follows the visible scene. The test uses a validation split of PointOdyssey, a dataset also used in training.
The paper tests accuracy separately on short clips from TAPVid-3D. Over the first 48 frames, TrackEverything scores 31.2 on APD-P, which measures how often tracked positions fall within a range of error tolerances; higher is better. The dense tracker VDPM scores 10.9. This is not a like-for-like comparison of the tracking methods alone: TrackEverything receives 3D position estimates from the VGGT-Ω geometry model that VDPM cannot use. The benchmark also evaluates labeled points, not every point the system tracks.
Reusing scene information offers a way to follow detailed motion for longer without storing every observation separately. But the approach depends on the quality of its depth estimates. Mistakes can merge distinct surfaces into one, and that merger cannot be undone. The authors also show failures with fast motion and repetitive or low-texture backgrounds. Their project page still lists the code as coming soon, leaving outside users waiting to test those trade-offs for themselves.
Sources
Image: Intel RealSense tracking demonstration photographed by Syced, via Wikimedia Commons (CC0). Illustrative file photograph, not the TrackEverything system.