Building Prompt-Routed Workflow Selection with Azure AI Foundry Model Router and Microsoft Agent Framework for GraphRAG

A router-first architecture for predictable orchestration, safer fallbacks, and evidence-driven validation.

Introduction

In Part 4, we explored orchestration patterns. In Part 5, we added quality and safety evaluation. Part 6 answers a production question:

How do we make routing deterministic, observable, and resilient when prompts, models, and environment conditions keep changing?

The answer in this part is not to make the router bigger. It is to make the router more explicit. The production workflow uses Azure AI Foundry model router instead of binding routing to one fixed small language model, so the router can keep its policy stable while Azure selects the best underlying model for the request.

That distinction matters. It is the difference between a brittle demo and a router that can survive real traffic, model changes, and audit questions.

What this part adds

One production entry point: RouterWorkflow.
Explicit classifier contract (workflow, confidence, reason).
Deterministic fallback when classifier calls fail or labels are invalid.
First-class out_of_context handling.
Stage-aware CI governance (PR vs main).

Why Router-First Matters

Single-agent or pattern-only demos can work for simple requests, but production systems need explicit control over routing decisions.

Production pressure	Common failure mode	Router-first benefit
Deployment/model updates	Behavior shifts silently	Confidence thresholds and fallback policy make drift visible
Prompt changes	Wrong workflow chosen	The classifier contract keeps output shape stable
Unknown or unsafe requests	Overuse of retrieval workflows	out_of_context avoids unnecessary fan-out
Incident debugging	Hard to reconstruct decision path	Routed metadata is recorded in workflow steps
CI spend constraints	Expensive checks run too often	Stage-aware checks gate expensive runs on main

The important point is that the router is not just a convenience layer. It is the policy boundary that decides when a request should be delegated, when it should fall back, and when it should stop early.

What Changed in Part 6

Part 6 introduces a production routing contract backed by Azure AI Foundry model router deployments.

Area	Part 4/5 posture	Part 6 posture
Production entrypoint	Multiple workflow entry paths	RouterWorkflow is the only production entrypoint
Classifier runtime path	Mixed or legacy paths possible	Agent Framework OpenAI client path only
Underlying model strategy	Fixed router model	Azure model router selects the underlying model per request
Unknown labels	Inconsistent handling risk	Deterministic degrade to sequential
Failure policy	Ad-hoc handling	Retry transient failures, then fallback with metadata
Context mismatch queries	Could consume retrieval paths	Explicit out_of_context route
CI evaluation strategy	Broader or default runs	PR local gate, main full evaluation path

Important additions in this part

Layer	Component	Purpose
Routing config	src/agents/config.py	Foundry-first router deployment contract
Classifier	src/agents/router_classifier.py	Chat-completions classifier with response normalization
Orchestration	src/workflows/router.py	Retry/fallback/confidence routing policy
Runtime surface	run_devui.py	Router-first interactive execution
Connector surface	run_router_chatbot.py	/api/messages endpoint for channel integration
Governance	.github/workflows/router-*.yml	Stage-aware evaluation checks

Why Model Router, Not a Fixed SLM

Azure AI Foundry model router is a trained language model that routes prompts in real time to the most suitable underlying model. It is deployed like any other Foundry model, but it behaves differently from a fixed router SLM because the underlying model choice can change per request.

That gives the workflow three practical advantages:

A single deployment behaves like a managed routing layer instead of a hand-wired model alias.
Balanced, Cost, and Quality routing modes let you bias the system toward cheaper or more accurate decisions.
Automatic failover reduces the chance that a single underlying model outage turns into an application outage.

There is a tradeoff. The effective context window is limited by the smallest underlying model, so the routing layer cannot exceed the limits of the models it may select. That is a good reason to keep prompts tight and to treat model subset selection as part of the design, not as an afterthought.

In practice, this is the right fit for a router workflow because the workflow is deciding among patterns, not generating long-form content.

Key operational detail

Model router does not require every supported model to be deployed separately. The main exception is Claude, which must be deployed separately before model router can invoke it. That detail matters when you explain the deployment story to readers who are new to Foundry.

For the official reference, see Microsoft Learn: Model router for Microsoft Foundry.

Router-First Architecture

The important thing is the control boundary. The router takes a request, classifies it through model router, and then decides whether to delegate, fall back, or short-circuit the request safely.

That keeps routing policy separate from content generation.

Production Routing Contract

RouterWorkflow enforces deterministic behavior:

Rule	Behavior
Confidence threshold	Route as classified when confidence >= 80
Transient classifier error	Retry before fallback
Retry exhaustion	Fallback to sequential
Unknown workflow label	Fallback to sequential
out_of_context label	Return safe direct response path

This keeps routing predictable under both model and infrastructure instability.

Metadata Contract for Observability

RouterWorkflow preserves both classification intent and executed path in workflow metadata.

Field	Meaning
classified_workflow	Raw classifier label
routed_workflow	Final executed workflow after policy or fallback
classifier_status	Classifier outcome state
classifier_attempts	Retry count
fallback_reason	Deterministic explanation when fallback occurs

The router uses a quantitative confidence score. Scores below 80 degrade to sequential for safer coverage while still preserving the classified route for audit.

{"route": "sequential","confidence_score": 84,"reason": "Mixed request with retrieval and summarization needs"}

The exact shape is not the point. The point is that the router emits a structured decision, not a free-form guess.

Code Walkthrough: Router Policy in Python

The implementation in src/workflows/router.py keeps routing policy explicit and testable. Three constants define the operational defaults:

_CLASSIFIER_MAX_ATTEMPTS = 3
_CLASSIFIER_RETRY_DELAY_SECONDS = 0.6
_MIN_ROUTING_CONFIDENCE_SCORE = 80

That means classifier reliability and fallback are policy-controlled in code, not hidden behind prompt wording. When transient failures occur, the router retries, then degrades deterministically to sequential while preserving fallback metadata.

for attempt in range(1, _CLASSIFIER_MAX_ATTEMPTS + 1):
  try:
    classification = await self._classifier.classify(query)
    return RouterOutcome(...)
  except (...):
    if attempt < _CLASSIFIER_MAX_ATTEMPTS and self._is_retryable_classifier_error(exc):
      await asyncio.sleep(_CLASSIFIER_RETRY_DELAY_SECONDS)
      continue
    break

Confidence gating is also explicit in code, not hidden in prompts:

def _resolve_workflow_decision(self, router_outcome: RouterOutcome) -> WorkflowType:
        decision = router_outcome.classification.workflow
        score = router_outcome.classification.confidence_score

        if score is None:
            router_outcome.fallback_reason = router_outcome.fallback_reason or "missing_confidence_score"
            return _LOW_CONFIDENCE_FALLBACK
        if score < _MIN_ROUTING_CONFIDENCE_SCORE:
            router_outcome.fallback_reason = router_outcome.fallback_reason or "low_confidence_score"
            return _LOW_CONFIDENCE_FALLBACK

        if decision in self._workflow_factories:
            return decision

        router_outcome.fallback_reason = router_outcome.fallback_reason or "unknown_workflow"
        return WorkflowType.SEQUENTIAL

Code Walkthrough: Classifier Contract

In src/agents/router_classifier.py, the classifier prompt requires compact JSON with strict fields and no extra text:

{
  "workflow": "sequential" | "concurrent" | "handoff" | "out_of_context",
  "confidence_score": 0..100,
  "reason": "short explanation"
}

The parser enforces defaults safely:

unknown or invalid workflow values degrade to sequential
malformed JSON is tolerated and does not crash routing
confidence is normalized to an integer score

That combination keeps the classifier useful even when output format varies.

Tracing and Span Correlation

Two spans are central for troubleshooting a routing decision end-to-end:

router.classifier.classify in router_classifier.py(classifier call path)
router.workflow.select in router.py (final routing policy selection)

The workflow-selection span captures the decision context needed for RCA:

router.classified_workflow
router.routed_workflow
router.confidence_score
router.classifier_status
router.classifier_attempts
router.fallback_reason

This is why the OTel capture is important. It is not cosmetic telemetry. It is the policy evidence trail.

Quick Start

This sequence keeps startup intent explicit: first tool surface, then router runtime, then connector validation.

1. Start the MCP server (tool surface)

uv run python run_mcp_server.py

Why: this exposes GraphRAG tools over MCP so routed workflows can call retrieval operations.

2. Start DevUI with router-first execution (workflow surface)

uv run python run_devui.py

Why: this is the fastest way to inspect routing decisions, workflow events, and execution flow in one place.

3. Optional: start connector endpoint (chat surface)

uv run python run_router_chatbot.py

Why: this exposes /api/messages for Teams or Agents Playground integration backed by RouterWorkflow.

4. Optional: validate from Microsoft 365 Agents Playground

Install CLI:

winget install agentsplayground

npm install -g @microsoft/m365agentsplayground

Run against local connector endpoint:

agentsplayground -e http://localhost:3978/api/messages -c msteams

Why: this validates the connector path independently from DevUI, which is useful for channel integration checks.

Workflow Surface Alignment

uv run python run_devui.py is the interactive runtime surface. Router remains the production default workflow, while sequential and concurrent patterns are retained for debugging and targeted validation.

For Teams or Agents Playground compatibility, run_router_chatbot.py exposes a connector-oriented endpoint at /api/messages backed by RouterWorkflow.

These two surfaces are independent.

Use DevUI when you want to inspect the workflow graph, the events panel, and the live trace for a request. Use Agents Playground when you want to validate the connector surface over the chat endpoint. You do not need both for the same story, and the article should say that clearly.

Testing Workflows and Observability

We can see how the system make decisions.

Figure. DevUI router execution view with workflow graph and timeline events.

Figure. OTel spans view for the same DevUI turn, showing classifier, workflow selection, and tool activity.

Figure. Microsoft 365 Agents Playground conversation routed through the Part 6 backend.

The DevUI view is the most useful single capture because it shows the workflow graph, the events panel, and the execution timeline together. A second capture of the OTel spans view from the same run makes the evidence stronger because it shows how the classifier call, workflow selection, and tool activity line up across the same turn.

If you include a log excerpt, keep it short and purposeful. The goal is not to dump logs. The goal is to prove that the UI and the backend agree about what happened.

Router classifier invoking chat completions on deployment '<deployment>'
Router classifier received response in 5.22s
Router selected 'sequential' workflow

Retry and fallback behavior is also directly visible when transient errors happen:

Router classifier attempt 1/3 failed (TimeoutError). Retrying in 0.6s.
Router classifier failed after 3 attempt(s). Falling back to sequential workflow.

That combination gives readers a full chain of evidence: the visible flow, the traced execution, and the lower-level log record.

CI Governance and Evaluation Spend

Router evaluation follows a staged policy:

PR path: local-only router checks for merge gating.
Main path: full router batch plus Foundry publish and red-team checks when relevant files changed.

Workflow split:

.github/workflows/ci.yml: core lint/type/test pipeline for push and PR validation.
.github/workflows/router-evaluation.yml: reusable evaluator workflow for local, Foundry, and optional red-team toggles.
.github/workflows/router-main-evaluation.yml: main-only workflow that runs full router evaluation with Foundry publish and red-team when relevant files changed.
.github/workflows/router-pr-merge-gate.yml: required PR gate workflow triggered on PR lifecycle updates and review submissions; expensive router evaluation remains conditional.

This preserves deterministic merge enforcement while reducing unnecessary evaluation spend.

Testing Strategy

tests/agents/test_router_classifier.py validates:

Request payload structure, including the JSON-only response contract and temperature control.
Credential handling for API key versus Azure CLI token provider path.
Workflow parsing, confidence normalization, and reason extraction.
Metadata overrides when the Foundry response includes router annotations.
Error handling when the router deployment rejects or fails the request.

tests/workflows/test_router.py validates router orchestration behavior:

Unknown classifier labels degrade to sequential while preserving classified versus routed workflow metadata.
Transient classifier failures retry and recover without manual intervention.
Non-retryable classifier failures degrade to sequential with explicit fallback metadata.

The tests do not just verify that the system responds. They verify that the router still tells the truth about how it responded.

Common Operational Pitfalls

Pitfall	Typical symptom	Practical mitigation
Router deployment mismatch	Classifier errors or low-confidence bursts	Validate deployment names, endpoint, and parser contract
Unknown workflow labels	Inconsistent behavior across requests	Enforce deterministic sequential fallback
Over-routing unsupported queries	Unnecessary retrieval calls	Keep out_of_context as a first-class route
Missing decision metadata	Difficult RCA after regressions	Persist classified and routed workflow fields
Unscoped expensive CI checks	Avoidable credit burn	Keep PR local gate and main full-eval split

Practical Release Guidance

Treat router promotion as a policy rollout, not only a model rollout.

Use this release lens:

Routing correctness: classified versus routed workflow consistency.
Fallback stability: retries and deterministic degradation behavior.
Operational traceability: metadata and logs sufficient for RCA.
Cost discipline: staged CI checks aligned with branch lifecycle.

Key Takeaways

Part 6 makes RouterWorkflow the production control plane for orchestration.
Reliability comes from explicit policy, not implicit model behavior.
out_of_context routing is essential for safe and efficient traffic handling.
Step-level metadata is non-negotiable for debugging and auditability.
Stage-aware CI governance balances quality evidence with evaluation spend.

What Comes Next

Part 7 can build on this foundation by adding conversational session readiness:

session-aware routing context
memory-safe turn handling
session-level diagnostics tied to router decisions

Part 8 can then add human-in-the-loop approval checkpoints on top of the same router policy surface.

Reference Material

Microsoft Learn: Model router for Microsoft Foundry
Project repository: cristofima/maf-graphrag-series
Part 6 implementation notes: part6-implementation-notes.md