Microsoft has released run-assert-eval, an open-source developer tool that helps teams find ways an AI agent could fail, test selected risks in conversations, and draft rules to block those failures. A developer reviews the policy and its manifest and checks their connection to the agent before running the evaluation again.
Clarity helps identify possible failure modes; ASSERT turns selected risks into behavior-specific test suites and scores the agent’s conversations; and the Agent Control Specification (ACS) describes where a runtime policy can intercept its actions. The new skill connects those steps, which Microsoft says developers previously had to wire together by hand.
In Microsoft’s worked billing-support example, an agent should handle only one customer’s account. Clarity surfaces risks such as changing billing details before checking the caller’s identity and revealing another customer’s data. A person chooses which risks to test; teams can also skip discovery and provide risks they already know.
When a test exposes cross-account access, the skill drafts an ACS policy and manifest. The example blocks a tool call before it runs if the requested account does not match the caller’s, then withholds a disallowed result after the tool returns. A developer must review the policy and manifest and check their connection to the agent before testing the governed version.
Microsoft says the rerun keeps the behavior definitions, test cases and automated judge constant. In its example, cross-customer violations fell across the separately reported prompt and scenario test splits, without more failures on permitted requests. These are results from Microsoft’s own worked evaluation, not an independent study or a report from a deployed customer system.
The skill is available in the ASSERT repository, which includes worked examples; Microsoft says ASSERT and ACS are MIT-licensed. The guided Clarity step requires a coding assistant configured to connect to its risk-discovery server, but teams can skip discovery and supply known risks directly. This is developer tooling, not a hosted monitoring service.
Sources
- Microsoft’s announcement and worked evaluation
- ASSERT repository and run-assert-eval skill
- Agent Control Specification policy engine
Image: “Security lock symbol-blue.svg” by NikolasKHF, Wikimedia Commons, CC0 1.0.
