Infrastructure alerts are useful because they tell us that something may be wrong. The real challenge begins after the alert arrives, when an engineer has to decide whether it is important, what systems are affected and what should be investigated first.

After experimenting with AI-assisted log analysis, I started thinking about a broader operational problem. Logs are only one part of an incident.

An engineer may also need to review monitoring alerts, CPU and memory metrics, recent deployments, Kubernetes events, service health and network connectivity.

In this article, I will explore how AI can be used as an incident triage assistant that brings multiple operational signals together and converts them into a structured incident summary.

The goal is not to let AI automatically fix infrastructure. The goal is to help engineers understand an incident faster and decide what to investigate next.

Incident Triage Flow

Infrastructure Alert
↓
Collect Metrics + Logs + Events + Recent Changes
↓
Normalise and Correlate Evidence
↓
AI Assisted Analysis
↓
Incident Summary + Severity + Possible Causes
↓
Recommended Diagnostic Checks
↓
Engineer Reviews and Decides

Why Incident Triage Is Difficult

A monitoring platform may send an alert such as:

Alert: API error rate exceeded 10% for five minutes.

This tells us what threshold was crossed, but it does not tell us why.

The engineer may then need to check:

The challenge is not only collecting information. It is deciding which signals may be related.

A Simple Incident Scenario

Imagine a production API begins returning HTTP 500 errors.

At approximately the same time:

Looking at each signal individually may not provide enough context.

Alert
↓
API Errors
+
Database Latency
+
Pod Restart
+
Recent Deployment
↓
Possible Related Incident

Defining the Incident Data Model

Before involving AI, I prefer representing operational information in a structured form.

A Python dictionary can represent the incident:

incident = {
    "alert": {
        "name": "High API Error Rate",
        "severity": "critical",
        "value": "14.2%"
    },

    "metrics": {
        "cpu_percent": 42.5,
        "memory_percent": 68.1,
        "database_latency_ms": 4200
    },

    "kubernetes": {
        "pod_restarts": 1,
        "ready_pods": 2,
        "desired_pods": 3
    },

    "deployment": {
        "version": "1.4.2",
        "minutes_ago": 12
    },

    "service_health": {
        "api": "DEGRADED",
        "database": "SLOW"
    }
}

Structured data gives the analysis process a clear input.

It also becomes easier to add or remove operational signals later.

Start with Deterministic Rules

I do not want AI to be responsible for every decision.

Some incident signals are better handled through predictable rules.

For example:

def calculate_base_severity(incident):
    alert_severity = incident["alert"]["severity"]

    if alert_severity == "critical":
        return "HIGH"

    if alert_severity == "warning":
        return "MEDIUM"

    return "LOW"

This behaviour is predictable.

AI can then add context around that known severity rather than replacing it.

Detecting Important Correlations

We can also perform simple correlation before sending anything to AI.

For example, if an application was deployed shortly before an incident, that information may be relevant.

def find_correlations(incident):
    correlations = []

    deployment = incident["deployment"]

    if deployment["minutes_ago"] <= 15:
        correlations.append(
            "A deployment occurred within "
            "15 minutes of the alert."
        )

    metrics = incident["metrics"]

    if metrics["database_latency_ms"] > 3000:
        correlations.append(
            "Database latency is significantly elevated."
        )

    kubernetes = incident["kubernetes"]

    if kubernetes["pod_restarts"] > 0:
        correlations.append(
            "At least one application pod restarted."
        )

    return correlations

This gives us evidence that is already organised before AI analysis begins.

Building the Incident Context

I can now create a structured context object.

def build_incident_context(incident):
    return {
        "base_severity": calculate_base_severity(
            incident
        ),

        "correlations": find_correlations(
            incident
        ),

        "alert": incident["alert"],

        "metrics": incident["metrics"],

        "kubernetes": incident["kubernetes"],

        "deployment": incident["deployment"],

        "service_health": incident["service_health"]
    }

The benefit of this approach is that the AI does not receive a random collection of raw information.

It receives a clear operational context.

Adding Relevant Logs

Logs are still useful, but I prefer adding only the most relevant entries.

For example:

relevant_logs = [
    "WARN Database latency 3800ms",
    "ERROR Database connection timeout",
    "ERROR Request /api/orders returned 500",
    "WARN Kubernetes readiness probe failed",
    "INFO Pod cloud-api-7f8 restarted"
]

These can be added to the incident context after any required redaction.

Building the AI Triage Prompt

The prompt should clearly explain what the AI is allowed to do.

import json


def build_triage_prompt(context, logs):
    incident_json = json.dumps(
        context,
        indent=2
    )

    log_text = "\n".join(logs)

    return f"""
You are assisting a DevOps engineer
with incident triage.

Use only the evidence provided.

Return:

1. Incident summary
2. Operational impact
3. Important correlations
4. Possible causes
5. Recommended diagnostic checks
6. Confidence level

Rules:

- Do not claim a root cause unless
  the evidence confirms it.
- Separate facts from assumptions.
- Do not execute or recommend
  destructive actions.
- Do not restart, delete or modify
  infrastructure automatically.
- The engineer remains responsible
  for operational decisions.

Incident context:

{incident_json}

Relevant logs:

{log_text}
"""

This keeps the model focused on triage rather than remediation.

Example AI Triage Result

Based on our sample incident, the analysis might produce something similar to:

Incident Summary: The API is experiencing elevated HTTP 500 errors together with increased database latency and reduced Kubernetes pod availability.

Operational Impact: Customer requests to the API may fail or respond slowly.

Important Correlations: A new application version was deployed approximately 12 minutes before the alert. Database latency increased during the same period and one application pod restarted.

Possible Causes: A regression introduced by the recent deployment, database performance degradation, connection pool exhaustion or another dependency issue.

Recommended Checks: Compare version 1.4.2 with the previous release, review database connection metrics, inspect pod restart reasons and compare error rates before and after deployment.

Confidence: Medium. The timing of the deployment is relevant, but the current evidence does not confirm that the release caused the incident.

Why Correlation Is Not Causation

One of the most important parts of AI-assisted incident analysis is avoiding overconfidence.

A deployment happening shortly before an incident does not automatically mean the deployment caused it.

For example:

Weak conclusion:

The deployment caused the outage.

A better conclusion would be:

Evidence-based conclusion:

A deployment occurred shortly before the incident and should be investigated, but additional evidence is required before confirming it as the root cause.

Adding Kubernetes Events

Kubernetes events can provide useful operational context.

For example:

kubernetes_events = [
    {
        "reason": "Unhealthy",
        "message": "Readiness probe failed"
    },
    {
        "reason": "Killing",
        "message": "Container failed liveness probe"
    }
]

Adding these events can help distinguish between an application issue and a broader infrastructure issue.

Adding Recent Deployment Information

Deployment history is another useful signal.

Instead of only storing the latest version, I can include:

deployment_history = {
    "current_version": "1.4.2",
    "previous_version": "1.4.1",
    "deployment_time": "10:08",
    "incident_start": "10:20"
}

AI can then mention the temporal relationship without automatically treating it as proof.

Prioritising Incidents

Another useful application is helping prioritise multiple alerts.

Imagine three alerts arrive:

Alert A: CPU reached 82% for two minutes.

Alert B: Customer API error rate reached 18%.

Alert C: Disk usage reached 76%.

A basic triage system can combine thresholds with business impact.

def calculate_priority(
    severity,
    customer_impact
):
    if (
        severity == "HIGH"
        and customer_impact
    ):
        return "P1"

    if severity == "HIGH":
        return "P2"

    return "P3"

AI can then explain the context rather than independently inventing a priority.

Returning Structured Triage Results

If the result will be shown inside an incident management platform, structured output is useful.

{
  "priority": "P1",
  "summary": "Production API is returning elevated HTTP 500 responses.",
  "impact": "Customer requests may fail.",
  "correlations": [
    "Database latency increased",
    "One application pod restarted",
    "Version 1.4.2 was deployed 12 minutes before the alert"
  ],
  "possible_causes": [
    "Application regression",
    "Database performance degradation",
    "Connection pool exhaustion"
  ],
  "recommended_checks": [
    "Compare application versions",
    "Review database metrics",
    "Inspect pod restart reason",
    "Check connection pool utilisation"
  ],
  "confidence": "medium"
}

Human Approval Before Remediation

This is one boundary I consider especially important.

An AI system may suggest:

Compare the current deployment with the previous stable version.

That is useful.

But I would not allow the model itself to run:

kubectl rollout undo deployment/cloud-native-api

without an approved operational workflow.

AI Recommendation
↓
Engineer Review
↓
Permission Check
↓
Approved Automation
↓
Infrastructure Action

Why I Separate Analysis from Execution

Traditional automation is strongest when the expected behaviour is clearly defined.

For example:

IF deployment_approved
AND user_has_permission
AND target_environment == "test"

THEN execute deployment

AI is stronger at interpreting less structured information.

I therefore prefer:

AI = Interpretation
↓
Human = Decision
↓
Automation = Controlled Execution

Building an Incident Timeline

One feature I find particularly useful is automatically organising events by time.

For example:

10:08 Application version 1.4.2 deployed

10:16 Database latency begins increasing

10:18 Readiness probe failures begin

10:20 API error rate exceeds threshold

10:21 Application pod restarts

10:22 Critical incident alert generated

This makes the sequence easier for an engineer to understand.

Incident Triage Architecture

Monitoring Platform
↓
Infrastructure Alert
↓
Incident Collector
↓
Metrics + Logs + Events + Deployment History
↓
Deterministic Rules
↓
Correlation Layer
↓
AI Triage Assistant
↓
Summary + Impact + Possible Causes
↓
Recommended Diagnostic Checks
↓
Engineer
↓
Approved Operational Workflow

Where I Would Take This Next

The architecture in this article can be extended further.

Possible improvements include:

What I Learned from AI Assisted Incident Triage

This was an important step in how I started thinking about AI in operations.

My earlier automation focused mainly on answering:

Is the system healthy or unhealthy?

Incident triage introduces a more difficult question:

What is happening across these systems, what signals may be related, and what should an engineer investigate first?

That is where AI can add value because the problem requires interpretation rather than only threshold evaluation.

At the same time, I still prefer deterministic rules and controlled automation for actual infrastructure changes.

Conclusion

In this article, I explored how AI can help transform infrastructure alerts into more useful incident insights.

The incident triage workflow:

For me, the most useful role for AI in incident management is not automatic remediation.

It is reducing the amount of manual investigation required before an engineer can make an informed decision.

By combining deterministic monitoring, operational context and AI-assisted interpretation, incident triage can become faster without removing the controls that production environments require.

My next step: After exploring AI-assisted incident triage, I started looking at how development teams could interact with infrastructure automation more easily. In the next stage of my journey, I will explore self-service deployment workflows and how platform engineering can give developers a simpler and safer path to deploy applications.