Skip to content

Microsoft Foundry adds a test-and-tune loop for AI agents

Short answer

Microsoft announced continuous improvement for Foundry at Ignite. Teams define quality criteria, review agent traces, group recurring failures, and test candidate fixes before release. Insights is generally available; skill and cost optimization are in preview. Nothing reaches production until a human applies it, and results are statistically tested.

What this means for operators

If you run support or operations with agents, the useful part is the loop around the model. An expert marks which real conversations went well, those marks become the test set, and every prompt, skill or model change has to beat it before release. That gives a 10-200 person team a way to change agents without breaking them, and a release gate in GitHub Actions or Azure DevOps. Most of the optimization features are still previews, so pilot them on one agent with a clear metric, such as ticket classification accuracy, before relying on them.

Microsoft has added a continuous improvement loop to Foundry, its platform for running AI agents, according to a post on the Microsoft Azure Blog published around Ignite. The loop has four repeating steps: establish a standard, identify problems, optimize, then validate.

What the loop contains

Establish. Quality is defined through evaluations, which are automated tests scored against criteria the team writes. Rubric evaluators now cover safety as well as quality (preview). Experts can annotate real agent traces (preview), and those annotated traces can become evaluation datasets and a locked "golden" benchmark (both preview).

Identify. Insights, now generally available, clusters recurring problems across traces, shows the evidence and points to a likely cause. Microsoft says the same visibility extends to agents built outside Foundry with LangGraph, LangChain and CrewAI (generally available).

Optimize. Agent optimizer tests changes to instructions, skills and model choice against the team's evaluations. It recommends a candidate, but the team decides what ships, and nothing changes in production until it is applied. New in preview: sending an insight straight to the optimizer, skill optimization that turns past work and expert feedback into versioned skills, and cost optimization that looks for cheaper ways to run an agent, such as a smaller model. Programmed Training, also in preview, lets engineers write training recipes in Python that run on serverless GPUs. Managed Training is the new name for managed fine-tuning.

Validate. Foundry runs statistical tests to separate real gains from noise, and the check can sit in a release pipeline. Continuous evaluation, already available, samples live traffic with the same evaluators. Dashboards show quality, safety, latency and token usage, with alerts when scores slip.

Customer figures

The results below are Microsoft's account and customer-reported, not independently verified:

  • Alaska Airlines tuned a model to classify aircraft maintenance reports. It scored 81.2% out of the box and 91.1% after tuning, with qualified staff still reviewing and deciding.
  • Heidi reports an estimated 87% lower cost per clinical note after post-training an open-weight model.
  • Docusign says it cut AI processing costs by 90% with fine-tuned models.
  • Microsoft's own testing says Foundry's model router can save up to 65% against sending everything to a frontier model.

Also announced

Voice agents are generally available. Hosted agents handle more concurrent sessions (generally available) and start faster (preview). A Cost view across agents, models and tools is in preview, and Microsoft Copilot harness is in preview.

What to check

Several pieces are previews, so terms and behavior may change. The loop depends on people willing to review traces and write rubrics, since the standard of quality comes from your own experts.

Source: Azure Blog · In the Atlas: Microsoft →

The tools in this story

Next step

Visibility Analyzer

An AI news item will not tell you how assistants see your own site. The Visibility Analyzer checks that on your live site: which answers cite you, which pages an engine cannot retrieve, and what to fix first. Free to run.

Run a free visibility audit

Free to run. No card.

Fee
Free
Length
One run, minutes

Free tier: two analyses a day, no card required.