In a controlled study of 100 selected benchmark tasks, human reviewers completed agent interactions in an average of 1.2 dialogue rounds using a generated form, compared with 3.4 when they saw the same task information as text. The forms came from GenUI-Harness, a system that searches for database records and then builds controls for the choices a person still needs to make. The study tested a benchmark workflow, not live customers.
GenUI-Harness splits the work between two AI components. A tool-using agent first retrieves relevant records and identifies unresolved choices. A second model generates a temporary interface showing the available options and asking the person to settle only what remains. The agent then receives the response as structured data and carries out the requested action. The paper illustrates the process with a flight-booking task.
The authors also evaluated the system on UI-Tau Bench, where agents must reach a specified database state across ten domains, including travel, hotels, e-commerce and retail. Its Lite split contains 300 tasks and the Full split 1,000. On Lite, the complete GenUI-Harness system averaged 4.48 percentage points more Pass@3 successes than smolagents across nine model backbones; the reported 95% confidence interval was 2.37 to 6.59 points. Pass@3 counts a task as successful if the exact target database state is reached in at least one of three runs. The comparison measures two complete systems, not the interface alone.
The reviewer comparison selected tasks for which interfaces from all four source models worked in all three setup runs. For each task, three human reviewers were randomly assigned to use the generated form and three to use text showing the same query and retrieved options; each reviewer saw only one version. One “round” counted a user message or form submission and the agent’s reply, so multiple clicks or field edits in one submission counted once. Each condition covered 300 task-reviewer interactions.
The broader benchmark used an AI simulator to operate the interface in a browser. The simulator also received the generated code and the task’s reference goal, information that a person looking at the screen would not see. The authors note this may make the simulator more capable than an ordinary user. Their human study covers only tasks with consistently usable interfaces and does not evaluate a deployed booking or shopping service. The paper is an arXiv preprint dated October 8, 2026.
Sources
- When Interfaces Speak: paper and abstract
- Full paper, figures and evaluation details
- Authors’ code repository
Image credit: Figure 1 from Xiaolong Li et al., When Interfaces Speak, arXiv:2610.11123v1, CC BY-SA 4.0.