A new study tested OnTrack, a monitor designed to warn about failures in AI coding agents while they work. In a replay of archived coding runs, it scored modestly better than a text-similarity baseline after eight steps, but the paper does not show it operating inside a live coding service.
OnTrack turns an agent’s tool calls and the files or other artifacts they use into a dependency map. It compares the partial run with records of successful runs using “optimal transport,” a way of softly matching steps and their dependencies even when two runs take different routes. With reference runs and tool rules, the design can also flag deviations from a plan or block an irreversible action whose required inputs are missing. Without that context, it can check for signs such as loops, stalls and repeated calls.
The authors replayed 2,294 recorded SWE-agent coding trajectories; 2,288 could be parsed, including 430 resolved and 1,858 unresolved runs. After the first eight steps, OnTrack’s AUROC was 0.631, compared with 0.574 for cosine text similarity. AUROC measures how well a score ranks one group above another; 0.5 is chance, so the result shows a modest separation, not 63.1% accuracy. Its advantage over the text baseline remained statistically clear through step 10 and was gone by step 15.
A small length-matched analysis further limits the claim. Among 50 pairs of resolved and unresolved runs with similar step counts, OnTrack’s AUROC was 0.526, close to chance and below cosine similarity at 0.540. The paper says the monitor detects process anomalies that correlate with failure; it does not predict whether an agent will succeed.
The paper also estimates what might happen if a monitor stopped runs after repeated severe warnings. At an illustrative threshold, the replay analysis saved 17.9% of compute on unresolved runs but falsely stopped 20.7% of resolved runs in a balanced sample. The authors say the threshold was chosen for illustration, not as a deployment recommendation, and would need separate calibration on real traffic. They also note that simpler stop rules can achieve similar or greater savings at some false-stop rates.
For now, OnTrack is a method tested against archived traces and controlled fault injections. The paper identifies live extraction of tool dependencies and larger, better-matched reference sets as open work; where a formal checker exists, it says that checker should take priority over a proximity score.
Sources
- OnTrack: Real-Time Monitoring and Intervention in LLM Agent Trajectories via Streaming Structure-Aware Optimal Transport (full paper, arXiv, Oct. 8, 2026).
- SWE-agent trajectory dataset card.
- Illustrative image: “Programming.jpg” by Lorenzo Cafaro via Wikimedia Commons, released under CC0 1.0.
