Microsoft has added a continuous improvement loop to Foundry, its platform for running AI agents, according to a post on the Microsoft Azure Blog published around Ignite. The loop has four repeating steps: establish a standard, identify problems, optimize, then validate.
What the loop contains
Establish. Quality is defined through evaluations, which are automated tests scored against criteria the team writes. Rubric evaluators now cover safety as well as quality (preview). Experts can annotate real agent traces (preview), and those annotated traces can become evaluation datasets and a locked "golden" benchmark (both preview).
Identify. Insights, now generally available, clusters recurring problems across traces, shows the evidence and points to a likely cause. Microsoft says the same visibility extends to agents built outside Foundry with LangGraph, LangChain and CrewAI (generally available).
Optimize. Agent optimizer tests changes to instructions, skills and model choice against the team's evaluations. It recommends a candidate, but the team decides what ships, and nothing changes in production until it is applied. New in preview: sending an insight straight to the optimizer, skill optimization that turns past work and expert feedback into versioned skills, and cost optimization that looks for cheaper ways to run an agent, such as a smaller model. Programmed Training, also in preview, lets engineers write training recipes in Python that run on serverless GPUs. Managed Training is the new name for managed fine-tuning.
Validate. Foundry runs statistical tests to separate real gains from noise, and the check can sit in a release pipeline. Continuous evaluation, already available, samples live traffic with the same evaluators. Dashboards show quality, safety, latency and token usage, with alerts when scores slip.
Customer figures
The results below are Microsoft's account and customer-reported, not independently verified:
- Alaska Airlines tuned a model to classify aircraft maintenance reports. It scored 81.2% out of the box and 91.1% after tuning, with qualified staff still reviewing and deciding.
- Heidi reports an estimated 87% lower cost per clinical note after post-training an open-weight model.
- Docusign says it cut AI processing costs by 90% with fine-tuned models.
- Microsoft's own testing says Foundry's model router can save up to 65% against sending everything to a frontier model.
Also announced
Voice agents are generally available. Hosted agents handle more concurrent sessions (generally available) and start faster (preview). A Cost view across agents, models and tools is in preview, and Microsoft Copilot harness is in preview.
What to check
Several pieces are previews, so terms and behavior may change. The loop depends on people willing to review traces and write rubrics, since the standard of quality comes from your own experts.