From Custom ML Pipelines to Foundry: Modernizing Enterprise AI

Enterprises that started their AI journey in the “custom ML pipeline” era are now hitting a wall: brittle orchestration, fragmented tools, and slow iteration whenever models or business rules change. Azure AI Foundry provides a workflow-first, agent-ready, and governance-aware platform that enables teams to modernize legacy processes into reusable, observable, and easier-to-own AI systems without sacrificing their existing .NET and Azure investments. This shift is less about replacing tools and more about building a native AI infrastructure where models, data, and workflows can be assembled as building blocks.

The Legacy State: Custom ML Pipelines Everywhere

In many organizations, the first wave of AI looked like this: notebooks built into Azure Machine Learning models, exported, and wrapped in custom web APIs and orchestrators (logic apps, functions, or custom schedulers) that tied everything together. Typical characteristics:

Multiple services: Azure ML for training, Azure Functions for scoring, custom REST APIs, and separate ETL for feature engineering.
Hard-coded pipelines: Data paths, model versions, and thresholds baked into code and config files.
Weak governance: Limited audit trails, scattered logging, and unclear ownership of each step.

As AI use scales across teams and regions, this patchwork creates real pain: every new use case wants its “own pipeline,” leading to duplication, inconsistent patterns, and operational risk.

What Azure AI Foundry Changes

Azure AI Foundry introduces a unified environment for building, managing, and deploying AI solutions, including workflow orchestration, agent services, and integration with Azure OpenAI and Azure AI Search. Instead of stitching dozens of services manually, you build AI workflows on a visual canvas or via declarative definitions that:

Orchestrate prompts, tools, and models (foundation models, custom models, search) as nodes in a single workflow.
Run in a governed environment with policy controls, access management, and centralized observability.
Support hybrid patterns: workflow-first for orchestration, code-first for custom logic, and hybrid for large, complex systems.

For enterprises, this means AI solutions move from “projects” to “products”—versioned, secure, reusable, and easier to evolve.

Before and After: A Concrete Architecture

Before: Custom ML Pipeline

Imagine a customer churn prediction system built three years ago:

Data ingestion through Azure Data Factory into a data lake.
Model training and batch scoring via Azure Machine Learning pipelines.
A custom API layer in .NET exposing predictions to CRM and support tools.
Separate scripts or Logic Apps to post-process results (send emails, open tickets, etc.).

Each component is owned and deployed separately, with inconsistent logging and manual hand-offs between teams. A change to the feature set or business rules can require editing AML pipelines, API code, and scheduled jobs.

After: Azure AI Foundry Workflow

Modernized in Azure AI Foundry, the same use case can look like:

A workflow that:
- Ingests or receives events (e.g., “customer updated,” “contract nearing end”).
- Calls a model (Azure ML model endpoint or Azure OpenAI with RAG) via a model node.
- Uses tools for CRM lookups, discount policy checks, and messaging.
- Outputs an action: “flag for outreach,” “auto-generate email,” or “open support case.”
A single workflow definition manages orchestration; underlying models are swapped by configuration.

Everything runs under a unified security and governance model, with monitoring that tracks each step of the AI decision process.

Migration Strategy: Step-by-Step Refactoring

Modernizing an enterprise AI landscape doesn’t mean rewriting everything from scratch. A pragmatic step-by-step approach works best.

1. Inventory Existing Pipelines and Pain Points

Identify your core ML pipelines (e.g., churn, fraud detection, demand forecasting, document classification).
For each, document: entry points, dependencies (data, models, APIs), SLAs, and failure modes.
Prioritize those with high change frequency or operational complexity.

2. Wrap Existing Models as First-Class Services

Before moving orchestration, ensure models are cleanly exposed:

Deploy existing ML models as endpoints using Azure Machine Learning managed endpoints or Azure Kubernetes Service.
Define standard contracts (input/output schemas) so they can be consumed as “model nodes” in Foundry workflows later.

Example: A .NET client calling an existing churn model endpoint that you’ll soon wire into Foundry:

using System.Net.Http.Json;

public class ChurnModelClient
{
    private readonly HttpClient _http;

    public ChurnModelClient(HttpClient http)
    {
        _http = http;
    }

    public async Task<double> PredictChurnAsync(object features)
    {
        var response = await _http.PostAsJsonAsync("/score", features);
        response.EnsureSuccessStatusCode();
        var result = await response.Content.ReadFromJsonAsync<ChurnResult>();
        return result?.Probability ?? 0.0;
    }

    private record ChurnResult(double Probability);
}

3. Rebuild Orchestration as a Foundry Workflow

Once model endpoints are stable, replicate the existing orchestration logic in Azure AI Foundry:

Define a workflow that:
- Receives an event or request.
- Calls the churn model endpoint (model/tool node).
- Applies business rules (which can be moved into prompts, .NET tools, or both).
- Emits a decision (e.g., “Offer discount A,” “Route to retention team”).

Foundry’s workflow designer plus code-based configuration lets you represent what used to be multiple services as one orchestrated AI flow.

4. Introduce LLMs and RAG Incrementally

With orchestration in place, you can:

Use Azure OpenAI models to interpret complex contexts (e.g., combine numerical churn risk with free-text customer feedback).
Add RAG via Azure AI Search to ground recommendations in your policies, playbooks, or knowledge bases.

Instead of rewriting the model, you augment it: the Foundry workflow combines ML predictions with LLM reasoning and retrieval in a governed, auditable way.

Example: Modernizing a Support Triage Pipeline

Legacy Support Triage

A typical support triage pipeline in a SaaS company:

Emails and tickets are ingested into a queue.
A text classification model (built in Azure ML) predicts category/severity.
A custom rules engine routes tickets to teams and sets SLAs.
A separate service generates canned responses.

As volumes grow, this setup strains: models drift, rule sets become unmanageable, and engineers struggle to understand why tickets were routed a certain way.

Foundry-Based Triage

With Azure AI Foundry, the same scenario becomes an AI workflow:

Input Node: Ticket arrives (subject, body, metadata).
Retriever Node: Fetch relevant KB articles and historical resolutions via Azure AI Search.
Model Node (LLM): Use an Azure OpenAI model to:
- Summarize the issue.
- Propose category/severity.
- Draft a first response, referencing retrieved docs.
Custom Tool Node (C#): Enforce routing rules and SLA logic.
Output Node: Create/update ticket in your support tool, optionally send suggested reply.

Example C# tool that enforces routing/SLA rules in the workflow:

using Microsoft.SemanticKernel;

public class TicketRoutingTool
{
    [KernelFunction("route_ticket")]
    public TicketRoutingResult RouteTicket(
        string category,
        string severity,
        string customerTier)
    {
        var team = category switch
        {
            "Billing" => "Finance",
            "Security" => "SecurityOps",
            "Performance" => "Platform",
            _ => "GeneralSupport"
        };

        var slaHours = (severity, customerTier) switch
        {
            ("Critical", "Enterprise") => 1,
            ("Critical", _) => 2,
            ("High", _) => 4,
            ("Medium", _) => 8,
            _ => 24
        };

        return new TicketRoutingResult(team, slaHours);
    }

    public record TicketRoutingResult(string Team, int SlaHours);
}

This function is called from within a Foundry workflow whenever the LLM proposes a category/severity; the final ticket routing remains deterministic and transparent.

Impact:

Faster adaptation: Changing routing logic means updating one tool, not multiple services.
Better explainability: Workflow logs show which node made which decision.
Higher automation: LLMs handle summaries and drafts, humans review only edge cases.

Example: Finance – From Batch Risk Scoring to Real-Time Risk Agents

Legacy Risk Pipeline

In banking or lending, a batch risk pipeline might:

Run nightly: scoring portfolios using custom ML models.
Post scores to a database and BI dashboards.
Require manual review for suspicious patterns.

This leads to delayed reactions to fast-emerging risk.

Foundry Risk Agent

Modernization with Foundry could deliver a near-real-time risk agent:

Streaming events (transactions, portfolio updates) feed a Foundry workflow.
A model node calls existing risk models (deployed as endpoints).
A retriever node pulls regulatory rules, internal policy docs, and customer history.
An LLM node generates an explanation and recommended action.
A C# tool node applies thresholds and submissions to regulatory reporting systems.

You get:

Always-on agents that monitor risk continuously instead of in batches.
Rich, natural-language justifications for decisions (auditable and explainable).
A central place to update which rules, documents, or models are in play.

Governance, Observability, and Compliance in the New World

One of the biggest gains from moving to Foundry is better governance:

Unified access control: Azure RBAC and role-based access to workflows, models, and data sources.
Audit trails and lineage: Logs for which model version and workflow path produced which decision or content.
Policy enforcement: Integration with Azure Policy and content safety filters to control where data flows and how outputs behave.

From an observability standpoint, workflows can be instrumented end-to-end:

Each node emits telemetry (latency, error rates, token usage).
Traces can be correlated with Application Insights or your existing monitoring stack for full visibility.

For regulated industries (finance, healthcare, public sector), this is a critical factor in moving from “AI experiments” to “AI in production at scale.”

Putting It Into Practice

A practical modernization plan could look like this:

Pick one high-impact pipeline (support triage, churn, risk).
Stabilize model endpoints and schemas.
Rebuild orchestration as a Foundry workflow, reusing existing models.
Add LLMs and RAG to improve reasoning and explainability.
Wire in governance (RBAC, logging, policies) and monitor adoption.

This “from custom ML pipelines to Foundry” journey transforms AI from fragile scripts and siloed services into a strategic platform. Enterprises gain faster iteration, safer deployment, and a foundation for the next wave of AI: workflow-native agents that collaborate with your teams across apps, data, and infrastructure.