Introduction
AI applications often require more infrastructure than a traditional web application.
A production AI workload may need:
An application host
Managed identity
Container resources
Storage
Databases
Secret management
Monitoring
AI model access
Networking
Role assignments
Creating these resources manually may work for the first deployment, but it becomes difficult to reproduce the same environment for development, testing, staging, and production.
This is where the Azure Developer CLI can be useful.
The important question, however, is not simply whether an application can be deployed once.
The more useful question is:
Can the same infrastructure and application deployment be executed repeatedly and produce the expected result without accumulating configuration drift?
For AI applications, repeatability is particularly important because the application and its supporting resources can change frequently.
This article explains how to test repeatable infrastructure deployments, identify sources of drift, validate configuration, and build a deployment process that can be safely executed multiple times.
What Does Repeatable Deployment Mean?
A repeatable deployment should produce the same intended infrastructure state when executed multiple times with the same configuration.
Consider:
Source Code
|
v
Infrastructure Definition
|
v
Deployment
|
v
Azure Resources
Running the deployment again should not unexpectedly create duplicate resources or modify unrelated configuration.
A simple example:
First deployment:
Create resource group
Create application
Create storage
Create identity
Second deployment:
Verify/update existing resources
No unintended duplicates
This property is commonly associated with idempotent infrastructure operations.
Why AI Infrastructure Needs Repeatability
AI applications often depend on several services simultaneously.
For example:
+----------------+
| AI Application |
+-------+--------+
|
+-----------------+-----------------+
| | |
v v v
AI Service Storage Database
|
v
Model Access
A manual deployment may leave small differences between environments.
For example:
Development:
Managed identity enabled
Logging enabled
Model A
Staging:
Managed identity enabled
Logging partially configured
Model B
Production:
Secret-based authentication
Different logging settings
Model A
These differences can create problems that are difficult to reproduce.
Define Infrastructure as Code
Repeatability starts with describing infrastructure declaratively.
The deployment should define things such as:
The exact infrastructure technology can vary, but the important property is that infrastructure configuration lives alongside the application rather than only in a portal.
Separate Application and Infrastructure Configuration
Avoid putting environment-specific values directly into deployment definitions.
For example:
Application
|
+--> Code
+--> Configuration
Infrastructure
|
+--> Resource definitions
+--> Identity
+--> Networking
Environment-specific values should be supplied through controlled configuration.
For example:
Environment = staging
Region = selected region
AI Model = selected deployment
This makes the same deployment process reusable.
Establish a Baseline Environment
Before testing repeatability, create a clean environment.
Record:
Resource Group
Resources
Resource Types
Resource Configuration
Role Assignments
Application Settings
You need this baseline to compare later deployments.
A simple inventory might look like:
| Resource | Type | Expected |
|---|
| Application | App hosting | 1 |
| Storage | Storage | 1 |
| Database | Database | 1 |
| Identity | Managed identity | 1 |
| Monitoring | Monitoring | 1 |
Run the First Deployment
The first run establishes the initial infrastructure state.
Conceptually:
azd deploy
The exact command sequence depends on the project structure and deployment configuration.
After deployment, validate:
Application starts
AI service reachable
Identity configured
Storage accessible
Logs available
Do not treat a successful deployment command as proof that the environment is correct.
Test the Second Deployment
The second run is the first important repeatability test.
Run the same deployment again without changing the infrastructure definition.
The expected behavior is:
Existing resources
|
v
Reconcile desired state
|
v
Stable environment
Watch for:
Test the Third Deployment
Two successful deployments are not always enough.
Run the deployment several times.
For example:
Run 1 -> Create
Run 2 -> Reconcile
Run 3 -> Reconcile
Run 4 -> Reconcile
The infrastructure should remain stable.
This is especially useful for discovering operations that are accidentally non-idempotent.
Capture Deployment State
Create a deployment record for each run.
For example:
public sealed record DeploymentResult(
string Environment,
string DeploymentId,
int ResourceCount,
int ChangedResources,
int FailedResources,
TimeSpan Duration,
bool ApplicationHealthy);
This allows deployments to be compared systematically.
Measure More Than Deployment Success
A deployment can technically succeed while producing an incorrect environment.
Track:
| Metric | Purpose |
|---|
| Deployment success | Basic deployment status |
| Duration | Detect performance regressions |
| Resource count | Detect duplicates |
| Changed resources | Detect unexpected updates |
| Failed resources | Detect partial failures |
| Application health | Validate runtime behavior |
| Configuration drift | Validate desired state |
| Permission checks | Validate identity |
Test Clean Deployment
Repeatability should start with a clean environment.
The process should be tested as:
Empty Environment
|
v
Deployment
|
v
Expected Infrastructure
Validate that all required resources are created.
This catches missing dependencies that incremental deployments can hide.
Test Incremental Deployment
Next, deploy an application change without rebuilding the entire environment.
For example:
Version 1
|
v
Infrastructure + Application
Version 2
|
v
Application Update
The test should confirm that infrastructure remains intact.
This is particularly important for AI applications where application changes may happen much more frequently than infrastructure changes.
Test Infrastructure Changes
Modify one infrastructure property at a time.
For example:
Before:
Application SKU = A
After:
Application SKU = B
Deploy again and verify:
Only expected resource changes
This helps determine whether the infrastructure definition is truly declarative.
Detect Resource Duplication
A common deployment problem is accidental resource creation.
Suppose the expected state is:
Storage Accounts: 1
After repeated deployments:
Storage Accounts: 3
The deployment is not behaving as intended.
Create an inventory check:
static bool HasUnexpectedResourceCount(
int actual,
int expected)
{
return actual != expected;
}
For production systems, resource identity should be validated rather than relying only on counts.
Test Configuration Drift
Configuration drift occurs when the actual environment differs from the intended definition.
For example:
Desired:
Logging = Enabled
Actual:
Logging = Disabled
Another example:
Desired:
Managed Identity = Enabled
Actual:
Managed Identity = Disabled
The deployment process should detect and correct such differences where appropriate.
Introduce Controlled Drift
A useful test is to deliberately modify the deployed environment.
For example:
Deployment
|
v
Manual Configuration Change
|
v
Redeployment
|
v
Drift Reconciliation
Change one non-production setting manually.
Then run the deployment again.
The result should be predictable:
Expected state
|
v
Deployment
|
v
Configuration restored
or, if the setting is intentionally external to infrastructure management, the deployment should leave it alone.
The important point is that the behavior must be understood.
Test Identity Configuration
AI applications often use managed identities or other identity mechanisms to access resources.
Validate:
Application Identity
|
+--> AI Service
|
+--> Storage
|
+--> Database
For each dependency, verify:
Identity exists
Required role is assigned
Role is assigned to the correct identity
Application can authenticate
Unauthorized access is rejected
Do not validate identity configuration only by checking that the resource exists.
Test the actual authorization path.
Test Role Assignment Repeatability
Role assignments deserve special attention.
An incorrect deployment can create:
Identity A -> Role X
Identity A -> Role X
Identity A -> Role X
or assign a role to the wrong principal.
After repeated deployments, verify:
Expected principal
+
Expected resource
+
Expected role
Test AI Model Configuration
An AI application may depend on a specific model deployment.
For example:
Application
|
v
AI Endpoint
|
v
Model Deployment
Validate that the application points to the intended deployment.
A deployment that succeeds but silently points staging to the wrong model can create difficult-to-diagnose behavior.
Test Secret Handling
Secrets should not be embedded directly into source code or deployment templates.
Instead, use appropriate managed configuration and secret-management mechanisms.
Validate:
Secret exists
Secret is accessible to intended identity
Secret is not exposed in logs
Secret is not committed to source
Repeatability should never mean duplicating secrets unnecessarily.
Test Environment Isolation
A repeatable deployment should work across environments without accidentally sharing resources.
For example:
Development
|
+--> Dev resources
Staging
|
+--> Staging resources
Production
|
+--> Production resources
Test that environment-specific deployments do not accidentally reference another environment's resources.
Use Environment Parameters
Environment differences should be explicit.
For example:
Environment
Region
Resource Prefix
AI Endpoint
Model Deployment
A conceptual configuration might look like:
{
"environment": "staging",
"region": "selected-region",
"resourcePrefix": "myapp-stg"
}
The same deployment process can then be used for multiple environments.
Test Failure Recovery
A deployment can fail halfway through.
For example:
Resource A -> Created
Resource B -> Created
Resource C -> Failed
Resource D -> Not Created
The important question is whether the next deployment can recover.
Run:
Failed Deployment
|
v
Fix Configuration
|
v
Redeploy
|
v
Complete Environment
A repeatable deployment should not require manually deleting half the environment every time something fails.
Test Partial Infrastructure
Intentionally create some resources before deployment.
For example:
Resource Group -> Exists
Storage -> Exists
Identity -> Missing
Application -> Missing
Run the deployment.
This tests whether the deployment can reconcile an existing partial environment.
Test Application Health After Deployment
Infrastructure deployment is only half the test.
After deployment, run application-level checks.
For an AI application:
Health Check
|
+--> Application reachable
+--> Identity works
+--> AI endpoint reachable
+--> Storage accessible
+--> Database accessible
+--> Basic AI request succeeds
This provides a deployment-to-runtime validation path.
Build a Smoke Test
A simple application smoke test can verify the critical path.
public async Task<bool> RunSmokeTestAsync(
HttpClient client,
CancellationToken cancellationToken)
{
using var response = await client.GetAsync(
"/health",
cancellationToken);
return response.IsSuccessStatusCode;
}
For an AI application, a separate test can validate a controlled AI request.
The test should use a deterministic, low-cost scenario rather than a complex production workflow.
Test Deployment Duration
Deployment duration can also reveal problems.
For example:
| Run | Duration | Resources Changed |
|---|
| 1 | 8m 10s | 12 |
| 2 | 2m 15s | 2 |
| 3 | 2m 08s | 0 |
| 4 | 2m 11s | 0 |
A later deployment unexpectedly taking 10 minutes may indicate unnecessary resource replacement or configuration churn.
Identify Resource Replacement
Some configuration changes can cause resources to be replaced rather than updated.
That can be dangerous for stateful services.
Before accepting an infrastructure change, determine:
Update in place?
or
Resource replacement?
For databases and persistent storage, this distinction is particularly important.
Test Deployment From a Clean Machine
A deployment that works only on the original developer workstation is not truly reproducible.
Test from a clean environment containing only the documented prerequisites.
The test should validate:
Fresh Environment
|
v
Install Required Tooling
|
v
Clone Repository
|
v
Configure Environment
|
v
Deploy
This catches hidden dependencies such as:
Test CI/CD Execution
Once local repeatability is established, run the deployment from CI/CD.
The pipeline should use:
Source
|
v
Build
|
v
Infrastructure Validation
|
v
Deployment
|
v
Smoke Tests
The deployment should not depend on a developer's interactive session.
Store Deployment Metadata
For each deployment, record:
Commit
Environment
Infrastructure Version
Application Version
Deployment Time
Duration
Result
This makes it possible to determine what changed when a deployment behaves differently.
Example Deployment Gate
A simple deployment validation rule could look like:
static bool DeploymentPassed(
DeploymentResult result)
{
return result.FailedResources == 0
&& result.ApplicationHealthy;
}
A more advanced gate might additionally verify:
No unexpected resources
No critical permission failures
No configuration drift
Smoke tests passing
Test Repeatability Across Versions
Repeatability should also be tested after infrastructure changes.
For example:
Infrastructure v1
|
v
Deploy x3
|
v
Infrastructure v2
|
v
Deploy x3
Compare:
Resource changes
Deployment duration
Application health
Permission behavior
Runtime configuration
This can identify regressions in the deployment definition itself.
Common Mistakes
Testing Only the First Deployment
A successful first deployment does not prove repeatability.
Assuming a Successful Command Means a Healthy Application
Infrastructure can deploy successfully while the application remains unusable.
Ignoring Configuration Drift
Manual changes can silently create environment differences.
Testing Only From a Developer Machine
Local credentials and cached state can hide missing dependencies.
Ignoring Partial Failures
Recovery from failed deployments is an important part of repeatability.
Duplicating Resources
Resource naming and identity should be deterministic.
Hard-Coding Environment Values
Environment-specific configuration should remain configurable.
Ignoring Role Assignments
The resource can exist while the application still lacks permission to use it.
Storing Secrets in Templates
Repeatable deployment should not mean insecure secret distribution.
Skipping Application Smoke Tests
Infrastructure validation alone does not prove application functionality.
Best Practices
Define infrastructure declaratively.
Keep environment-specific configuration separate.
Establish a clean deployment baseline.
Run the same deployment multiple times.
Verify resource identity after every run.
Test deliberate configuration drift.
Validate managed identity and role assignments.
Test partial and failed deployments.
Verify application health after deployment.
Use deterministic smoke tests.
Test deployment from a clean environment.
Run the deployment through CI/CD.
Record deployment metadata.
Watch for unexpected resource replacement.
Test both clean and incremental deployments.
Keep secrets outside source-controlled deployment definitions.
Validate environment isolation.
Re-run repeatability tests after infrastructure changes.
Frequently Asked Questions
Is a repeatable deployment the same as an idempotent deployment?
They are closely related but not identical. Idempotency means repeated execution produces the same intended state without unwanted additional effects. Repeatability is broader and includes being able to reproduce the environment and deployment behavior consistently.
Should every deployment produce zero changes after the first run?
Not necessarily. If the desired infrastructure definition changed, subsequent deployments should make the corresponding changes. The important point is that unchanged configuration should not produce unexpected resource modifications.
How many times should I run a deployment to test repeatability?
There is no universal number. Multiple consecutive deployments are useful, and the test should also include clean environments, partial environments, controlled drift, and failure recovery.
Should AI model deployments be part of the infrastructure test?
If the application depends on them, yes. The evaluation should verify that the intended model endpoint and deployment configuration are available and correctly referenced.
Why test from a clean machine?
A clean environment exposes hidden dependencies on local credentials, configuration files, cached resources, or manually configured infrastructure.
What is the most important deployment test?
A strong minimum test is: deploy from a clean environment, verify the complete resource state, run application smoke tests, intentionally introduce controlled drift, redeploy, and verify that the environment returns to the intended state.
Conclusion
Repeatable infrastructure deployment is a fundamental part of operating AI applications reliably.
The goal is not simply to make an Azure deployment succeed once. The deployment process should be capable of creating the intended environment from scratch, safely reconciling an existing environment, recovering from partial failures, and producing predictable results across repeated executions.
The Azure Developer CLI can be part of that workflow, but the real engineering discipline comes from testing the deployment process itself.
A strong deployment test therefore covers clean deployment, repeated deployment, configuration drift, identity and permissions, partial failures, environment isolation, application health, and CI/CD execution.
When these tests become part of the development workflow, infrastructure stops being a manually maintained environment and becomes a reproducible, testable part of the application lifecycle.