Software agents that follow instructions on a screen must locate the right button or icon before clicking. Jingchao Wang and Dingjiang Huang propose Seg2Click, a model that marks a candidate region across a screenshot before selecting one point. Their October 12 arXiv preprint reports that its 9-billion-parameter version placed clicks inside labeled targets in 92.06% of ScreenSpot examples and 55.22% of ScreenSpot-Pro examples.
A mask marks candidate pixels before the model picks a click
A graphical-user-interface (GUI) agent is software that uses a visual interface to carry out instructions. A model trained to output one coordinate learns a single target point even when many places inside a button would work. Patch-based methods score image blocks, which can include a small control and its neighbors in the same block.
Seg2Click instead learns a map of candidate click locations from target boxes already labeled in screenshots. At inference, it keeps the largest connected area with a predicted score above 0.5, then clicks the center of that area. The authors say this fixed rule uses one prediction on the full screenshot, without an extra verifier or zoom-in pass.
The authors report higher target-box accuracy than listed baselines on three benchmarks
The paper tests one predicted click per screenshot. ScreenSpot and ScreenSpot-v2 each contain 1,272 examples across mobile, desktop and web interfaces; ScreenSpot-Pro contains 1,581 examples from professional applications. A result counts as correct when the click falls inside the annotated target box. On those tests, the authors report 92.06% on ScreenSpot, 94.10% on ScreenSpot-v2 and 55.22% on ScreenSpot-Pro for Seg2Click with a Qwen3.5-9B backbone. They report that these scores exceed the strongest published baselines in their tables by 2.36, 1.60 and 7.52 percentage points, respectively.
The preprint also shows cases where a predicted area spills into a neighboring control and its center falls outside the labeled box. The authors note that a box only approximates where a real interface will accept a click: a static screenshot cannot show whether an action works. Their tests measure target localization, not whether an interactive agent completes a task or recovers after a miss.
Sources
arXiv preprint and version history; full paper, experiments and limitations; photo by Esra Korkmaz on Unsplash; Unsplash License.