When the user was available and no privacy guidance was given, five open-weight AI models accessed a confidential source in 30% of valid task runs, even though asking the user or checking a public source could provide the same fact. Pairing a privacy instruction with labels marking each source's intrusiveness cut that rate to 15%, researchers report in a new arXiv preprint.
The study's benchmark, PrivacySkills, gives an agent choices such as asking for a flight's arrival terminal, checking a public flight-tracking page or opening a private travel account. Each route returns the same terminal, but the account may also contain unrelated passport or payment details. The privacy metric counts any run in which the agent accessed the confidential source, even if it also tried another route.
The researchers built 55 fictional tasks across 11 personal-information categories. Each task presented callable steps for asking the user, checking public information, opening an account-restricted source and completing the request; the full set contained 169 executable skills. They tested five open-weight models from Google, OpenAI, Mistral AI, NVIDIA and Alibaba, with 3,300 valid runs per model in each setting. A run counted only if the agent obtained the needed fact and completed the requested action.
The 30% result is the study's neutral-guidance baseline, not a comparison with another benchmark or real-world agent. A system instruction to prefer the least intrusive source lowered confidential access only slightly, to 28.9%. Labels that marked sources "minimal," "low" or "extreme" brought it to 22.7%; using both instruction and labels reduced it to 15%. The combined guidance produced a reduction in all five models, though the authors say their test cannot show whether the models better recognized sensitivity or simply gave privacy guidance more weight.
Without privacy guidance, telling the simulated agent that the user was unavailable raised confidential-source access to 45%. Framing the request as urgent left the average near 30%; the benchmark returned tool responses immediately, so it tested urgency wording rather than actual delay.
The result is a design signal, not proof that deployed agents will protect personal data. The benchmark used synthetic records, only five tasks per category and one format for presenting agents with available tools; no real accounts or personal data were accessed. Results may differ when systems present those choices another way. The next test is whether the same instruction-and-label combination works with real systems and different tool formats.
Sources
- Biéchy et al., "PrivacySkills: How Privacy Guidance Shapes Source Selection in LLM Agents" (arXiv, Sept. 28, 2026)
- Full paper: protocol, results and limitations
- Figure 1 from the paper, reproduced under CC BY 4.0.