Researchers at the University of St. Gallen and the Università della Svizzera italiana describe AgentSpy, a research prototype that records AI agents’ activity at the operating-system level. Their paper, posted to arXiv on Oct. 5, addresses a gap: an agent can pass task tests while helper programs access files or network services its own log does not show.
AgentSpy runs an agent in an isolated environment and records system calls — requests programs make to the operating system — and network activity from the agent and its child processes. It summarizes commands, files and hosts used, then applies deterministic rules to flag selected actions, such as opening credentials or contacting an untrusted domain.
To test reliability, the team used 77 of 87 SkillsBench tasks after excluding 10 whose environments could not be reproduced. Three models ran under the same Codex harness, three times per task. The researchers compared groups of three runs from one task with groups mixing runs from that task and the task using the most different skills. They compared the proportions of commands, files and network hosts used; the same-task groups were more similar in 92.2% of comparisons across the three models. Among 88 task-model pairs whose three runs all passed outcome tests, the authors flagged unrelated activity in 16, grading-file reads in six, and results reached without the supplied skill guidance in 15. The two author-assessors used AgentSpy’s metrics when labeling cases, so the counts do not independently show how often these behaviors occur or how well the system detects them.
The security test used five selected attack cases with one Codex setup running GPT-5.1-Codex-Max; each injected and control condition ran five times. The agent carried out injected behavior in 18 of 25 runs. AgentSpy flagged 14 of those and raised no alert in 25 control runs. All four missed cases involved a backdoor script that added an alternate Git destination for code pushes, but the agent never pushed code there. Because the change stayed in the allowed workspace and caused no network activity, none of AgentSpy’s rules matched. The authors say detecting that configuration change would require a task-specific rule.
This preprint tests a limited set of tasks, models and attacks. It shows how system-level records can add evidence beyond an agent’s own log and a passing result, but it does not establish a general security guarantee.
Sources
Illustrative photo: Free-Photos from Pixabay via Wikimedia Commons, CC0 1.0. The image does not show AgentSpy.
