Microsoft ASSERT turns plain-English behavior specs into AI tests

Microsoft released ASSERT, an open-source framework that turns a written description of how an AI system should behave into a set of scored tests for it. The full name is Adaptive Spec-driven Scoring for Evaluation and Regression Testing. The idea is to bridge the gap between generic model benchmarks like MMLU and the questions a team actually needs answered: does this specific agent follow our specific rules, and does it keep following them after the next prompt change? You give ASSERT a few sentences of intent, and it converts that into a structured list of acceptable and unacceptable behaviors, generates scenarios, runs them against the model, and scores the result while logging the intermediate reasoning and tool calls so failures can be traced.

The example Microsoft offers is concrete. For an internal document research agent, a team can write "should not send emails to people outside the company" and "should limit confidential information to C-level executives," and ASSERT generates tests that probe both rules across plausible workflows. Sarah Bird, Microsoft's chief product officer for Responsible AI, makes the point underneath this directly: organizations have to evaluate dimensions that are application-specific, not just whatever the leaderboard measures. The framework is open-sourced rather than rolled into a paid product, which suggests Microsoft expects most adopters will run it against models they did not build themselves.

Why it matters

If you ship an AI feature inside another product, the chronic problem is that you only find out a rule was broken when a user reports it, and a regression suite of natural-language behaviors closes that gap without writing brittle prompt-injection tests by hand. The tradeoff is that ASSERT generates the tests, so the tests inherit the model's blind spots; treat it as a useful baseline rather than the only check.