Terraform is often used with the expectation that infrastructure can be described as code and then kept consistent with that definition. You write the configuration, run a plan, apply the changes, and expect the infrastructure to remain aligned with the Terraform files.
That works well when Terraform is the only system changing the infrastructure. Production environments are rarely that simple.
Cloud consoles, deployment pipelines, platform teams, autoscaling systems, Kubernetes controllers, managed services, and cloud provider defaults can all change resources outside the Terraform workflow. Terraform may then report that everything is correct in one situation while the infrastructure changes again shortly afterward.
This is the basic problem behind infrastructure drift.
Drift does not always mean that Terraform is broken. In many cases, it means something else changed infrastructure after Terraform created or updated it.
What Infrastructure Drift Actually Means
Infrastructure drift happens when the real infrastructure no longer matches the configuration and state that Terraform expects.
For example, suppose Terraform defines an application server with:
resource "aws_instance" "api" {
instance_type = "t3.medium"
tags = {
Environment = "production"
}
}
Terraform applies this configuration and creates the instance.
Later, someone changes the instance type manually:
Terraform configuration: t3.medium
Actual infrastructure: t3.large
Terraform configuration and production infrastructure now disagree.
That difference is drift.
Drift can also happen without anyone manually changing a resource. A cloud service may modify an attribute automatically, an autoscaling system may adjust capacity, or another automation tool may update a property.
The important point is that the source of drift is not always a human.
Terraform State Is Part of the Picture
To understand drift properly, it is important to understand Terraform state.
Terraform does not simply compare your .tf files directly against every resource in the cloud. It also maintains state describing the infrastructure Terraform manages.
A simplified model looks like this:
Terraform configuration
↓
Terraform state
↓
Real infrastructure
The configuration describes what you want.
The state records what Terraform believes it manages and tracks important resource attributes.
The cloud provider contains the actual infrastructure.
During planning, Terraform refreshes information about managed resources and compares the known infrastructure with the desired configuration.
If something has changed outside Terraform, the plan may identify a difference.
For example:
Configuration
instance_type = t3.medium
Actual resource
instance_type = t3.large
Terraform plan
change detected
The plan is useful because it gives you an opportunity to decide whether that difference should be corrected.
Why Production Can Drift After a Clean Plan
A common situation looks like this:
Terraform configuration is correct.
Terraform plan shows no changes.
Terraform apply completes successfully.
Production changes later.
Terraform is run again.
A difference appears.
This can be confusing because the earlier plan was completely clean.
There is no contradiction.
A Terraform plan describes the infrastructure at the point when the plan was generated. It does not prevent another system from changing the infrastructure afterward.
For example:
10:00 Terraform plan → no changes
10:05 Terraform apply → infrastructure updated
10:30 Admin changes resource in cloud console
11:00 Terraform plan → drift detected
The clean plan was correct at 10:00. The infrastructure simply changed afterward.
Manual Cloud Console Changes Are a Common Cause
One of the simplest causes of drift is manual modification through a cloud provider's console.
A developer or administrator may change:
Instance size
Security group rules
Database settings
Storage configuration
Network settings
Tags
Scaling configuration
Access policies
The change may solve an immediate production problem, but it creates a difference from the Terraform configuration.
For example, someone may increase a database instance size during an incident:
Terraform:
db.t3.medium
Production:
db.t3.large
The production change may be completely reasonable as an emergency action.
The problem occurs later when nobody records that change in the infrastructure code.
The next Terraform plan can then propose changing the database back.
This creates an operational conflict between the infrastructure code and the actual production environment.
Drift Can Be Caused by Other Automation
Not all changes come from engineers.
Consider a Kubernetes cluster managed partly through Terraform.
Terraform may create a resource, but another controller may modify some of its properties later.
The same pattern can occur with cloud services that automatically manage resources.
For example:
Terraform
↓
Creates resource
↓
Cloud service/controller modifies resource
↓
Actual resource differs from Terraform expectation
If the changed property is managed by another system, Terraform may repeatedly detect the difference.
This can create a plan that appears to "undo" a change every time Terraform runs.
That is a sign that ownership needs to be clarified.
Why Terraform Plan Can Keep Showing the Same Change
A particularly frustrating drift problem looks like this:
terraform plan
→ change detected
terraform apply
→ change applied
terraform plan
→ same change detected again
When this happens, do not keep applying the same plan.
First determine who is modifying the resource.
A useful investigation is:
Terraform configuration
↓
Terraform state
↓
Provider/API
↓
Actual resource
↓
Other automation
If another system changes the resource immediately after Terraform applies it, Terraform is not the source of the repeated difference.
The configuration and ownership model need to be examined.
Some Attributes Should Not Be Controlled by Terraform
In shared infrastructure, it is sometimes intentional for another system to manage specific attributes.
For example, suppose Terraform creates an autoscaling resource, but a platform service dynamically adjusts a particular value.
If Terraform continuously attempts to restore that value, the two systems are effectively fighting each other.
Terraform provides lifecycle settings that can sometimes be used to intentionally ignore changes to specific attributes.
For example:
resource "aws_instance" "api" {
instance_type = "t3.medium"
lifecycle {
ignore_changes = [
tags
]
}
}
This tells Terraform not to treat changes to the specified attribute as something it should correct.
However, ignore_changes should not be used simply to make a noisy plan disappear.
It should represent an intentional ownership decision.
If Terraform ignores an important production setting, the configuration may look clean while the real infrastructure is becoming increasingly difficult to understand.
Drift and Terraform State Are Different Problems
Another common mistake is assuming that every mismatch means Terraform state is corrupted.
Drift usually means the real resource changed outside Terraform.
State problems are different.
For example:
Terraform configuration
↓
Terraform state
↓
Real infrastructure
If the state correctly represents the resource but the real resource was modified manually, that is normal drift.
If Terraform state no longer correctly identifies or represents the managed resource, the problem may involve state management.
This distinction matters because the recovery approach is different.
Do not manually edit the state file just because Terraform shows a difference.
First determine whether the resource itself changed.
How to Investigate Production Drift
When Terraform detects an unexpected change, start with the plan.
Run:
terraform plan
Read the proposed changes carefully.
Do not immediately run:
terraform apply
Instead, identify exactly what changed.
For example:
~ resource "example" "api" {
instance_type = "t3.medium" -> "t3.large"
}
Then ask:
Was this change intentional?
Who made it?
When did it happen?
Was it made manually?
Was it caused by another automation system?
Should Terraform own this attribute?
Should the Terraform configuration be updated?
This turns drift investigation into an ownership question rather than simply a Terraform problem.
Check the Resource Outside Terraform
After identifying the changed attribute, verify the actual infrastructure directly through the provider's CLI or API.
For example, if an instance size changed, verify the current resource configuration using the provider's tooling.
The goal is to establish three facts:
Desired configuration
Actual infrastructure
Terraform state
These three values should be compared rather than looking at Terraform configuration alone.
A useful investigation table is:
Item | Value |
|---|
Terraform configuration | What the code declares |
Terraform state | What Terraform currently tracks |
Cloud resource | What actually exists |
Change source | Who or what modified it |
Intended owner | Which system should control it |
This makes recurring drift much easier to diagnose.
Use Version Control to Track Infrastructure Changes
Infrastructure changes should normally go through version control.
Instead of changing production manually:
Engineer
↓
Cloud Console
↓
Production changed
prefer:
Engineer
↓
Terraform change
↓
Pull request
↓
Review
↓
Terraform plan
↓
Apply
↓
Production
This provides a history of why the infrastructure changed.
It also makes emergency changes easier to reconcile.
If an engineer temporarily increases a resource during an incident, the team should decide afterward whether that change belongs in Terraform.
Otherwise, the next infrastructure deployment may unintentionally reverse the emergency fix.
Detect Drift Before It Becomes a Deployment Problem
Drift is much easier to manage when it is detected regularly.
A CI pipeline can run Terraform plan and report unexpected changes.
For example:
terraform init
terraform plan
The exact workflow depends on the team's infrastructure pipeline, but the important principle is to make drift visible before a production deployment depends on it.
A plan that detects changes should trigger investigation rather than automatically applying them.
This is especially important for production because automatically correcting drift can overwrite a legitimate emergency change.
Common Mistakes
Treating Every Plan Difference as an Error
Not every difference is automatically bad.
Some changes are intentional and should be incorporated into the Terraform configuration.
Others belong to a different automation system.
The important question is why the difference exists.
Using ignore_changes Everywhere
Adding ignore_changes to every noisy attribute can make plans look clean while hiding real infrastructure changes.
Use it only when ownership is intentionally delegated elsewhere.
Making Manual Changes Without Reconciliation
Emergency changes happen. The problem is leaving them undocumented.
After the incident, decide whether the Terraform configuration should be updated or the infrastructure should be returned to the declared state.
Automatically Applying Every Plan
A Terraform plan can contain a destructive or unexpected change caused by drift.
Production pipelines should have appropriate review and approval controls before applying significant infrastructure changes.
Fixing State Before Understanding the Resource
State manipulation should not be the first response to drift.
First inspect the actual resource and determine what changed.
Best Practices for Keeping Production in Sync
Keep Terraform as the source of truth for resources it owns.
Use version control and pull requests for infrastructure changes.
Limit direct production console changes.
Clearly define which system owns each infrastructure attribute.
Run regular plan checks to detect drift.
Review emergency production changes afterward and reconcile them with Terraform.
Use ignore_changes only for intentional ownership boundaries.
Avoid automatically applying unexpected drift without review.
Monitor recurring plan differences. If the same attribute changes repeatedly, investigate the automation that is modifying it.
Summary
Terraform drift happens when production changes outside the Terraform configuration or when multiple systems try to manage the same resource.
A clean plan does not guarantee that production will remain unchanged. It only represents the infrastructure state at that point in time.
When drift appears, compare the Terraform configuration, state, and actual resource. Then identify who owns the changed attribute before deciding whether to update Terraform, revert the production change, or intentionally ignore it.
Join the conversation! Your thoughts help the community grow.