Researchers from NYU, Advanced Machine Intelligence, INRIA Paris and Brown University report H-JEPA, a model for planning long visual tasks through shorter steps. In Visual AntMaze, a simulated task in which a robot navigates a maze, its three-level version reached 73.3% mean success, compared with 18.0% for the one-level LeWM baseline. Results were averaged over three runs with matched seeds for the model and planner.
H-JEPA is a world model: it uses camera observations and actions to predict a compact representation of what may happen next, rather than recreating every video frame. Each level predicts over a different time span in its own representation. The top level plans toward the distant goal and passes a predicted state down as a subgoal; lower levels refine that target into shorter steps until the lowest level outputs actions. The planner then updates its plan from a new observation.
The hierarchy can help because a distant goal does not always require tracking every fast-changing detail. In Visual AntMaze, the higher-level representation retained the agent’s position and maze layout while making leg-pose details less recoverable. A separate AntMaze sweep varied the planner’s computational budget; the three-level model recorded higher success with less planning computation than flat LeWM. Results varied by task: the three-level version performed poorly on Push-T, where short episodes may have limited the training data available for longer sequences.
The researchers also tested H-JEPA on DROID, a dataset of real teleoperated-robot videos. To keep changing backgrounds from overwhelming information about robot motion, they added an inverse-dynamics loss—a training task that asks the model to infer the action connecting two consecutive state representations. On the paper’s Fréchet path-fidelity measure, which scores how closely each planned robot-arm tool-tip path follows an expert path, H-JEPA scored higher than flat LeWM trained with the same inverse-dynamics loss. The evaluation was offline: it compared planned and expert paths on 16 held-out pick-and-place clips; no physical robot executed the plans. Closed-loop robot testing remains future work. These results are from an arXiv preprint.
Sources
H-JEPA paper · authors’ project page · code repository. Figure: H-JEPA authors; paper license CC BY 4.0.