Infrastructure alerts are useful because they tell us that something may be wrong. The real challenge begins after the alert arrives, when an engineer has to decide whether it is important, what systems are affected and what should be investigated first.
After experimenting with AI-assisted log analysis, I started thinking about a broader operational problem. Logs are only one part of an incident.
An engineer may also need to review monitoring alerts, CPU and memory metrics, recent deployments, Kubernetes events, service health and network connectivity.
In this article, I will explore how AI can be used as an incident triage assistant that brings multiple operational signals together and converts them into a structured incident summary.
The goal is not to let AI automatically fix infrastructure. The goal is to help engineers understand an incident faster and decide what to investigate next.
Incident Triage Flow
Infrastructure Alert
↓
Collect Metrics + Logs + Events + Recent Changes
↓
Normalise and Correlate Evidence
↓
AI Assisted Analysis
↓
Incident Summary + Severity + Possible Causes
↓
Recommended Diagnostic Checks
↓
Engineer Reviews and Decides
Why Incident Triage Is Difficult
A monitoring platform may send an alert such as:
Alert: API error rate exceeded 10% for five minutes.
This tells us what threshold was crossed, but it does not tell us why.
The engineer may then need to check:
The challenge is not only collecting information. It is deciding which signals may be related.
A Simple Incident Scenario
Imagine a production API begins returning HTTP 500 errors.
At approximately the same time:
API error rate increases
Database latency increases
One application pod restarts
A new application version was deployed 12 minutes earlier
CPU utilisation remains normal
Looking at each signal individually may not provide enough context.
Alert
↓
API Errors
+
Database Latency
+
Pod Restart
+
Recent Deployment
↓
Possible Related Incident
Defining the Incident Data Model
Before involving AI, I prefer representing operational information in a structured form.
A Python dictionary can represent the incident:
incident = {
"alert": {
"name": "High API Error Rate",
"severity": "critical",
"value": "14.2%"
},
"metrics": {
"cpu_percent": 42.5,
"memory_percent": 68.1,
"database_latency_ms": 4200
},
"kubernetes": {
"pod_restarts": 1,
"ready_pods": 2,
"desired_pods": 3
},
"deployment": {
"version": "1.4.2",
"minutes_ago": 12
},
"service_health": {
"api": "DEGRADED",
"database": "SLOW"
}
}
Structured data gives the analysis process a clear input.
It also becomes easier to add or remove operational signals later.
Start with Deterministic Rules
I do not want AI to be responsible for every decision.
Some incident signals are better handled through predictable rules.
For example:
def calculate_base_severity(incident):
alert_severity = incident["alert"]["severity"]
if alert_severity == "critical":
return "HIGH"
if alert_severity == "warning":
return "MEDIUM"
return "LOW"
This behaviour is predictable.
AI can then add context around that known severity rather than replacing it.
Detecting Important Correlations
We can also perform simple correlation before sending anything to AI.
For example, if an application was deployed shortly before an incident, that information may be relevant.
def find_correlations(incident):
correlations = []
deployment = incident["deployment"]
if deployment["minutes_ago"] <= 15:
correlations.append(
"A deployment occurred within "
"15 minutes of the alert."
)
metrics = incident["metrics"]
if metrics["database_latency_ms"] > 3000:
correlations.append(
"Database latency is significantly elevated."
)
kubernetes = incident["kubernetes"]
if kubernetes["pod_restarts"] > 0:
correlations.append(
"At least one application pod restarted."
)
return correlations
This gives us evidence that is already organised before AI analysis begins.
Building the Incident Context
I can now create a structured context object.
def build_incident_context(incident):
return {
"base_severity": calculate_base_severity(
incident
),
"correlations": find_correlations(
incident
),
"alert": incident["alert"],
"metrics": incident["metrics"],
"kubernetes": incident["kubernetes"],
"deployment": incident["deployment"],
"service_health": incident["service_health"]
}
The benefit of this approach is that the AI does not receive a random collection of raw information.
It receives a clear operational context.
Adding Relevant Logs
Logs are still useful, but I prefer adding only the most relevant entries.
For example:
relevant_logs = [
"WARN Database latency 3800ms",
"ERROR Database connection timeout",
"ERROR Request /api/orders returned 500",
"WARN Kubernetes readiness probe failed",
"INFO Pod cloud-api-7f8 restarted"
]
These can be added to the incident context after any required redaction.
Building the AI Triage Prompt
The prompt should clearly explain what the AI is allowed to do.
import json
def build_triage_prompt(context, logs):
incident_json = json.dumps(
context,
indent=2
)
log_text = "\n".join(logs)
return f"""
You are assisting a DevOps engineer
with incident triage.
Use only the evidence provided.
Return:
1. Incident summary
2. Operational impact
3. Important correlations
4. Possible causes
5. Recommended diagnostic checks
6. Confidence level
Rules:
- Do not claim a root cause unless
the evidence confirms it.
- Separate facts from assumptions.
- Do not execute or recommend
destructive actions.
- Do not restart, delete or modify
infrastructure automatically.
- The engineer remains responsible
for operational decisions.
Incident context:
{incident_json}
Relevant logs:
{log_text}
"""
This keeps the model focused on triage rather than remediation.
Example AI Triage Result
Based on our sample incident, the analysis might produce something similar to:
Incident Summary: The API is experiencing elevated HTTP 500 errors together with increased database latency and reduced Kubernetes pod availability.
Operational Impact: Customer requests to the API may fail or respond slowly.
Important Correlations: A new application version was deployed approximately 12 minutes before the alert. Database latency increased during the same period and one application pod restarted.
Possible Causes: A regression introduced by the recent deployment, database performance degradation, connection pool exhaustion or another dependency issue.
Recommended Checks: Compare version 1.4.2 with the previous release, review database connection metrics, inspect pod restart reasons and compare error rates before and after deployment.
Confidence: Medium. The timing of the deployment is relevant, but the current evidence does not confirm that the release caused the incident.
Why Correlation Is Not Causation
One of the most important parts of AI-assisted incident analysis is avoiding overconfidence.
A deployment happening shortly before an incident does not automatically mean the deployment caused it.
For example:
Weak conclusion:
The deployment caused the outage.
A better conclusion would be:
Evidence-based conclusion:
A deployment occurred shortly before the incident and should be investigated, but additional evidence is required before confirming it as the root cause.
Adding Kubernetes Events
Kubernetes events can provide useful operational context.
For example:
kubernetes_events = [
{
"reason": "Unhealthy",
"message": "Readiness probe failed"
},
{
"reason": "Killing",
"message": "Container failed liveness probe"
}
]
Adding these events can help distinguish between an application issue and a broader infrastructure issue.
Adding Recent Deployment Information
Deployment history is another useful signal.
Instead of only storing the latest version, I can include:
deployment_history = {
"current_version": "1.4.2",
"previous_version": "1.4.1",
"deployment_time": "10:08",
"incident_start": "10:20"
}
AI can then mention the temporal relationship without automatically treating it as proof.
Prioritising Incidents
Another useful application is helping prioritise multiple alerts.
Imagine three alerts arrive:
Alert A: CPU reached 82% for two minutes.
Alert B: Customer API error rate reached 18%.
Alert C: Disk usage reached 76%.
A basic triage system can combine thresholds with business impact.
def calculate_priority(
severity,
customer_impact
):
if (
severity == "HIGH"
and customer_impact
):
return "P1"
if severity == "HIGH":
return "P2"
return "P3"
AI can then explain the context rather than independently inventing a priority.
Returning Structured Triage Results
If the result will be shown inside an incident management platform, structured output is useful.
{
"priority": "P1",
"summary": "Production API is returning elevated HTTP 500 responses.",
"impact": "Customer requests may fail.",
"correlations": [
"Database latency increased",
"One application pod restarted",
"Version 1.4.2 was deployed 12 minutes before the alert"
],
"possible_causes": [
"Application regression",
"Database performance degradation",
"Connection pool exhaustion"
],
"recommended_checks": [
"Compare application versions",
"Review database metrics",
"Inspect pod restart reason",
"Check connection pool utilisation"
],
"confidence": "medium"
}
Human Approval Before Remediation
This is one boundary I consider especially important.
An AI system may suggest:
Compare the current deployment with the previous stable version.
That is useful.
But I would not allow the model itself to run:
kubectl rollout undo deployment/cloud-native-api
without an approved operational workflow.
AI Recommendation
↓
Engineer Review
↓
Permission Check
↓
Approved Automation
↓
Infrastructure Action
Why I Separate Analysis from Execution
Traditional automation is strongest when the expected behaviour is clearly defined.
For example:
IF deployment_approved
AND user_has_permission
AND target_environment == "test"
THEN execute deployment
AI is stronger at interpreting less structured information.
I therefore prefer:
AI = Interpretation
↓
Human = Decision
↓
Automation = Controlled Execution
Building an Incident Timeline
One feature I find particularly useful is automatically organising events by time.
For example:
10:08 Application version 1.4.2 deployed
10:16 Database latency begins increasing
10:18 Readiness probe failures begin
10:20 API error rate exceeds threshold
10:21 Application pod restarts
10:22 Critical incident alert generated
This makes the sequence easier for an engineer to understand.
Incident Triage Architecture
Monitoring Platform
↓
Infrastructure Alert
↓
Incident Collector
↓
Metrics + Logs + Events + Deployment History
↓
Deterministic Rules
↓
Correlation Layer
↓
AI Triage Assistant
↓
Summary + Impact + Possible Causes
↓
Recommended Diagnostic Checks
↓
Engineer
↓
Approved Operational Workflow
Where I Would Take This Next
The architecture in this article can be extended further.
Possible improvements include:
Historical incident comparison
Automatic incident timelines
Alert deduplication
Incident clustering
Change correlation
Kubernetes event analysis
Service dependency mapping
Runbook recommendations
Structured post-incident summaries
Feedback from engineers to improve future analysis
What I Learned from AI Assisted Incident Triage
This was an important step in how I started thinking about AI in operations.
My earlier automation focused mainly on answering:
Is the system healthy or unhealthy?
Incident triage introduces a more difficult question:
What is happening across these systems, what signals may be related, and what should an engineer investigate first?
That is where AI can add value because the problem requires interpretation rather than only threshold evaluation.
At the same time, I still prefer deterministic rules and controlled automation for actual infrastructure changes.
Conclusion
In this article, I explored how AI can help transform infrastructure alerts into more useful incident insights.
The incident triage workflow:
Receives monitoring alerts
Collects operational context
Uses deterministic severity rules
Correlates recent changes and system events
Adds relevant application logs
Builds a structured incident timeline
Uses AI to summarise evidence
Identifies possible causes
Suggests diagnostic checks
Keeps operational decisions with engineers
For me, the most useful role for AI in incident management is not automatic remediation.
It is reducing the amount of manual investigation required before an engineer can make an informed decision.
By combining deterministic monitoring, operational context and AI-assisted interpretation, incident triage can become faster without removing the controls that production environments require.
My next step: After exploring AI-assisted incident triage, I started looking at how development teams could interact with infrastructure automation more easily. In the next stage of my journey, I will explore self-service deployment workflows and how platform engineering can give developers a simpler and safer path to deploy applications.
Join the conversation! Your thoughts help the community grow.