Getting one AI agent working for five people on your engineering team is exciting. Keeping that same agent fast, affordable, and online when 15,000 employees start hammering it at 9:00 AM on Monday morning is an entirely different challenge.

What happens if one runaway script in the marketing department sends 5,000 requests in a minute and eats up your entire Azure token quota, locking out your customer support team? Or what happens if the East US region has a temporary hiccup?

To handle enterprise scale, you pair Microsoft Foundry with two things: an AI Gateway in Azure API Management (APIM) for traffic control, and Agent 365 Publishing to deliver your agent right inside Microsoft Teams and Microsoft 365 Copilot.

Why Put an AI Gateway in Front of Microsoft Foundry?

Inside the Microsoft Foundry portal (under Manage > AI Gateway), you can link Azure API Management directly to your Foundry endpoints across all three routing standards (/openai, /models, and openai/v1/).

The AI Gateway sits like a smart traffic cop between your apps and your Foundry models:

[ Apps / Teams / Departments ]
              |
              v
[ Azure API Management AI Gateway ]
   |-- 1. Token Rate Limit (Caps each department at 50,000 tokens/min)
   |-- 2. Semantic Cache (Returns instant cached answers for repeated questions)
   +-- 3. Load Balancer & Failover
              |
      +-------+---------------+
      | (Primary: 99% traffic)| (Failover on HTTP 429/503)
      v                       v
[ Foundry East US ]     [ Foundry Sweden Central ]

Three Gateway Policies Every Architect Uses

  1. Token Limit Policy (llm-token-limit): Standard API gateways only count requests per minute. That doesn't work for AI because one request might be 10 tokens and the next might be 80,000 tokens. The AI Gateway counts actual tokens per minute (TPM) per user or department and blocks noisy neighbors before they exhaust your quota.
  2. Semantic Caching (llm-semantic-cache-lookup): If 500 employees ask "When is the company holiday party?" on the same morning, you shouldn't pay the LLM to generate the exact same answer 500 times. Semantic caching uses vector similarity to spot repeated questions and returns the cached answer in 15 milliseconds at zero token cost.
  3. Multi-Region Circuit Breaker: You deploy your Foundry model in two regions (for example, Primary PTU capacity in East US and overflow Standard capacity in West US). If East US ever returns HTTP 429 (Too Many Requests), the gateway automatically retries the request in West US without the user ever noticing.

Sample APIM Policy XML for Token Limiting & Token Metrics

<policies>
    <inbound>
        <base />
        <!-- Authenticate to Microsoft Foundry using the Gateway's Managed Identity -->
        <authentication-managed-identity resource="https://ai.azure.com" />

        <!-- Limit each client subscription to 50,000 tokens per minute -->
        <llm-token-limit
            counter-key="@(context.Subscription.Id)"
            tokens-per-minute="50000"
            estimate-prompt-tokens="true"
            remaining-tokens-header-name="x-ratelimit-remaining-tokens" />

        <!-- Emit per-department token usage metrics to Application Insights -->
        <llm-emit-token-metric namespace="foundry-gateway">
            <dimension name="ClientSubscription" value="@(context.Subscription.Id)" />
            <dimension name="ClientIP" value="@(context.Request.IpAddress)" />
        </llm-emit-token-metric>
    </inbound>
</policies>

Publishing Your Agent to Microsoft Teams and M365 Copilot

Once your agent is tested, secured, and backed by a gateway, how do employees actually use it? Most companies do not want to build and maintain a brand-new React frontend for every internal agent they create. Employees already live inside Microsoft Teams and Microsoft 365 Copilot.

In the current Microsoft Foundry portal, Agent Publishing to M365 and Teams is Generally Available (GA):

  1. Open your tested agent inside Build > Agents in the Foundry portal.
  2. Select Publish and choose Microsoft 365 Copilot & Microsoft Teams.
  3. Foundry automatically generates the app manifest, wires your agent's endpoint and Entra Agent ID into the Microsoft 365 admin center, and makes your agent available in the Teams app store for the user groups you approve.
Identity Passthrough Tip: When an employee chats with your published Foundry agent inside Microsoft Teams, use UserEntraToken passthrough on your Work IQ or SharePoint Toolbox connections. That way, the agent can only search files that the specific logged-in Teams user is already allowed to read!