When a Kubernetes pod keeps restarting, the first reaction is often to open the application logs and search for an exception.
Logs are important, but they are not always the best place to start.
A pod can restart because the application crashed, but it can also restart because Kubernetes killed the container, a health probe failed, the container ran out of memory, the startup process took too long, the node became unhealthy, or the container was terminated for another infrastructure-related reason.
This is why Kubernetes events are useful during troubleshooting. Events provide information about actions taken by Kubernetes around the pod, such as failed probes, container termination, scheduling problems, image failures, and resource-related events.
A good troubleshooting process starts by answering a simple question:
What caused Kubernetes to restart the container?
Once that is known, application logs become much more useful because you know which part of the system to investigate.
What Is a Kubernetes Pod Restart?
A pod is the basic execution unit in Kubernetes. It represents one or more containers that are scheduled and managed together.
In a simple application, a pod might contain one application container:
Pod
|
+-- Application Container
Kubernetes continuously manages the desired state of the application. If a container inside the pod terminates and its configuration allows it to be restarted, Kubernetes can start the container again.
For example:
Application starts
|
v
Container running
|
v
Container exits
|
v
Kubernetes restarts container
|
v
Container running again
A restart does not necessarily mean that the entire pod object was deleted and recreated. The container inside the existing pod can be restarted.
This distinction matters when troubleshooting because the restart count and pod events can provide useful information about what happened.
Why Start With Kubernetes Events?
Kubernetes events are records generated when something significant happens to a Kubernetes resource.
For a pod, events can provide information about actions such as:
For example, a pod may show an event indicating that a liveness probe failed.
That immediately changes the investigation.
Instead of spending an hour searching application logs for a random exception, you can first investigate the health check that caused Kubernetes to restart the container.
You can inspect pod events with:
kubectl describe pod <pod-name> -n <namespace>
The output contains several sections. The Events section near the bottom is particularly useful during restart investigations.
Check the Pod Status First
Before reading logs, check the current state of the pod.
kubectl get pod <pod-name> -n <namespace>
You may see something like:
NAME READY STATUS RESTARTS AGE
orders-api 1/1 Running 7 15m
The important value here is the restart count.
A pod with seven restarts in 15 minutes deserves investigation even though its current status is Running.
A pod can therefore look healthy at the exact moment you check it while still having a serious restart problem.
You can also watch the pod:
kubectl get pod <pod-name> -n <namespace> -w
This is useful when the restart happens frequently and you want to observe the state transition.
Use Describe Before Going Deep Into Logs
The next step should usually be:
kubectl describe pod <pod-name> -n <namespace>
The output contains information about the pod, including:
State
Last State
Reason
Exit Code
Restart Count
Conditions
Events
These fields can tell you whether the container is currently running and what happened during its previous execution.
For example:
Last State: Terminated
Reason: OOMKilled
Exit Code: 137
Restart Count: 5
This is a much stronger troubleshooting signal than simply seeing that the application restarted.
It tells you that the container was terminated because it exceeded its memory limit.
Understand the Difference Between Current and Previous State
A common mistake is checking only the current container state.
Suppose the container currently shows:
State: Running
That does not tell you why the previous container instance stopped.
Kubernetes also keeps information about the last container state.
For example:
State: Running
Last State: Terminated
Reason: Error
Exit Code: 1
The last state can provide the clue needed to investigate the previous failure.
This is particularly useful when the application restarts quickly and the current logs no longer show the original failure.
Check the Previous Container Logs
When a container has restarted, the logs from the current container may not contain the error that caused the previous instance to terminate.
Use:
kubectl logs <pod-name> -n <namespace> --previous
The --previous option asks Kubernetes for logs from the previous terminated container instance.
For example:
kubectl logs orders-api -n production --previous
This is often the correct next step after kubectl describe pod identifies a container termination.
The important sequence is:
kubectl get pod
|
v
kubectl describe pod
|
v
Check Last State and Events
|
v
kubectl logs --previous
This gives you both sides of the investigation: what Kubernetes observed and what the application reported.
Common Cause: OOMKilled
OOMKilled means the container was terminated because it exceeded the memory limit assigned to it.
For example, a deployment might specify:
resources:
requests:
memory: "256Mi"
limits:
memory: "512Mi"
If the application uses more than the allowed memory limit, the container can be terminated.
The important distinction is between a memory request and a memory limit.
The request tells Kubernetes how much memory the workload needs for scheduling purposes.
The limit defines the maximum memory the container is allowed to use.
If the application regularly approaches the limit, a traffic increase or temporary memory spike can push it over the boundary.
The result can look like:
Normal traffic
|
v
Memory usage increases
|
v
Memory reaches container limit
|
v
Container terminated
|
v
Kubernetes restarts container
If the pod shows OOMKilled, investigate the application's memory usage before simply increasing the limit.
A higher limit may hide a memory leak without solving it.
Check for Memory Leaks
If memory usage continuously increases over time, the application may have a memory leak or an inefficient memory-management pattern.
For example:
Memory
|
| /
| /
| /
| /
|__________/
+--------------------> Time
If memory repeatedly rises until the container is killed, increasing the container limit may only make the restart happen later.
Look for:
Large collections kept in memory
Unbounded caches
Large object allocations
Excessive buffering
Unreleased resources
Large response bodies
Memory-heavy serialization
Incorrect object lifetime management
Application-level memory metrics are especially useful here.
Common Cause: Liveness Probe Failure
A liveness probe tells Kubernetes whether a running container is still functioning correctly.
A liveness probe is different from a readiness probe.
The purpose of a liveness probe is to answer a question similar to:
Is this container still alive and capable of recovering on its own?
For example:
livenessProbe:
httpGet:
path: /health/live
port: 8080
initialDelaySeconds: 20
periodSeconds: 10
If the liveness probe repeatedly fails according to its configured thresholds, Kubernetes can restart the container.
The flow is:
Container running
|
v
Liveness probe fails
|
v
Failure threshold reached
|
v
Kubernetes restarts container
This means an application can be completely free of an unhandled exception and still restart because its health endpoint is failing.
Why Liveness Probes Can Cause Restart Loops
A poorly designed liveness probe can create a restart loop.
Suppose the application takes 60 seconds to initialize, but the liveness probe starts checking it after only 5 seconds.
The probe may fail repeatedly:
Application starting
|
v
Probe starts too early
|
v
Probe fails
|
v
Container restarted
|
v
Application starts again
|
v
Probe fails again
The application never gets enough time to become healthy.
This is why startup behavior must be considered when configuring probes.
For applications with slow initialization, a startup probe can be useful.
Startup Probe vs Liveness Probe
A startup probe is designed for applications that need time to initialize.
Its purpose is to give the application enough time to start before Kubernetes begins relying on the liveness probe for ongoing health detection.
For example:
startupProbe:
httpGet:
path: /health/startup
port: 8080
failureThreshold: 30
periodSeconds: 5
The exact values depend on the application.
The important idea is:
Startup probe
|
v
Application initialization
|
v
Liveness probe
|
v
Long-term health monitoring
This separation prevents slow startup from being incorrectly treated as an application failure.
Common Cause: Application Crash
Sometimes the simplest explanation is correct: the application itself crashed.
For example, the previous state may show:
Reason: Error
Exit Code: 1
In this case, inspect the previous container logs:
kubectl logs <pod-name> --previous
Look for:
The important point is that Kubernetes may only be restarting the container because the application exited.
Kubernetes is not necessarily the source of the problem.
Common Cause: Configuration Problems
A container may start successfully and then exit because required configuration is missing.
For example, an application may require:
DATABASE_CONNECTION
API_KEY
MESSAGE_BROKER_URL
If one of these values is missing or invalid, the application may fail during startup.
The pod could then repeatedly restart:
Container starts
|
v
Application reads configuration
|
v
Required configuration missing
|
v
Application exits
|
v
Kubernetes restarts container
This can look like a Kubernetes problem even though the root cause is application configuration.
Check ConfigMaps, Secrets, environment variables, mounted files, and configuration precedence when investigating startup failures.
Common Cause: Dependency Failure
An application may also exit because it cannot connect to a required dependency.
For example:
Application
|
+---- PostgreSQL
|
+---- Redis
|
+---- RabbitMQ
|
+---- External API
If the application requires one of these services during startup and treats connection failure as fatal, the container may terminate.
Before changing Kubernetes settings, determine whether the dependency was actually available.
For example, check:
DNS resolution
Network policies
Service endpoints
Credentials
TLS configuration
Connection strings
Dependency health
Check Exit Codes
Exit codes provide another useful clue.
For example:
Exit Code: 0
usually indicates normal process termination.
An application that repeatedly exits with code 1 may have an application-level failure.
An exit code associated with memory termination can indicate a resource problem.
Do not treat an exit code as the complete diagnosis. Use it together with the termination reason, events, and logs.
The combination is much more useful than any individual signal.
Check Events for Probe Failures
If the restart is caused by a health probe, events often make this visible.
You may see information indicating that a liveness or startup probe failed.
The important detail is the actual failure.
For example:
HTTP probe failed with status code 500
means something very different from:
connection refused
An HTTP 500 may indicate that the application is running but its health endpoint is returning an error.
A connection refusal may indicate that the application is not listening on the expected port or is not ready to accept connections.
Investigate the specific probe failure instead of simply increasing probe timeouts.
Readiness Failures Are Different
A readiness probe tells Kubernetes whether a pod is ready to receive traffic.
If readiness fails, Kubernetes can remove the pod from service endpoints so that new requests are not sent to it.
This normally does not mean the container should be restarted.
For example:
Readiness fails
|
v
Pod removed from traffic
|
v
Container continues running
A liveness failure can lead to:
Liveness fails repeatedly
|
v
Container restarted
Confusing these two probes can cause incorrect troubleshooting and poor health-check design.
Check the Deployment Configuration
If the pod keeps restarting, inspect the deployment or workload configuration.
For example:
kubectl get deployment <deployment-name> -n <namespace> -o yaml
Review:
Container image
Environment variables
ConfigMaps
Secrets
Resource requests
Resource limits
Liveness probe
Readiness probe
Startup probe
Volume mounts
Command
Arguments
A configuration change may have introduced the restart problem even if the application code itself has not changed.
Check Whether the Node Has Problems
Not every restart originates inside the application container.
The Kubernetes node running the pod may experience resource pressure or other infrastructure problems.
For example, Kubernetes can report conditions related to:
MemoryPressure
DiskPressure
PIDPressure
If several unrelated pods on the same node begin behaving strangely, investigate the node rather than focusing on one application.
This is especially important when multiple workloads show failures around the same time.
Look for Patterns Across Multiple Pods
Suppose a deployment has five replicas:
orders-api-1 -> Restarting
orders-api-2 -> Running
orders-api-3 -> Running
orders-api-4 -> Restarting
orders-api-5 -> Restarting
If several replicas restart around the same time, look for a shared cause.
Possible examples include:
If only one pod behaves differently, investigate pod-specific conditions such as node placement or local resource pressure.
A Practical Troubleshooting Sequence
When a pod keeps restarting, use this sequence instead of immediately searching through large application logs.
1. Check the Pod
kubectl get pod <pod-name> -n <namespace>
Look at status, readiness, and restart count.
2. Describe the Pod
kubectl describe pod <pod-name> -n <namespace>
Check the container state, previous state, termination reason, exit code, and Events section.
3. Check Previous Logs
kubectl logs <pod-name> -n <namespace> --previous
Use this when the container has restarted and the current logs do not contain the original failure.
4. Check Resource Usage
Investigate memory and CPU usage and compare them with configured requests and limits.
5. Check Health Probes
Verify whether liveness, readiness, or startup probes are failing.
6. Check Dependencies
Verify database, cache, messaging, APIs, DNS, and network connectivity where applicable.
7. Check the Workload Configuration
Review the deployment, environment variables, ConfigMaps, Secrets, commands, arguments, and mounted volumes.
8. Check the Node
If multiple pods are affected, investigate node health and resource pressure.
9. Check Recent Changes
Look for recently deployed images, configuration changes, probe changes, resource-limit changes, or dependency changes.
10. Reproduce the Failure
If possible, reproduce the startup or health problem in a non-production environment. A repeatable failure is much easier to diagnose than a restart that disappears before you can inspect it.
Common Mistakes
Starting With Application Logs Only
Logs may show what the application experienced, but they may not explain why Kubernetes terminated the container.
Start with pod state and events.
Ignoring the Previous Container
Current logs can belong to a newly started container. Use --previous when investigating the earlier failure.
Increasing Memory Without Investigating
A larger memory limit can temporarily stop OOMKills but may hide an underlying memory leak or excessive allocation.
Increasing Probe Timeouts Blindly
A probe failure may be caused by a broken health endpoint rather than insufficient timeout values.
Treating Readiness and Liveness as the Same
Readiness controls whether traffic should reach the pod. Liveness determines whether Kubernetes should restart the container.
Assuming Every Restart Is an Application Crash
Kubernetes can restart containers for several reasons, including failed probes and resource termination.
Ignoring Events
Events often provide the quickest indication of what Kubernetes was doing when the failure occurred.
Best Practices
Define health probes based on the actual behavior of the application rather than copying the same configuration to every workload.
Use startup probes for applications that require significant initialization time.
Set memory requests and limits based on measured application behavior.
Monitor restart counts and alert on unusual increases rather than waiting for the application to become completely unavailable.
Keep application logs, Kubernetes events, resource metrics, and deployment history available together for troubleshooting.
Use graceful shutdown handling so the application can close connections and finish safe cleanup when Kubernetes terminates a container.
When a restart occurs, investigate the first failure in the sequence. Repeated restarts can produce secondary errors that make the original problem harder to identify.
Summary
A restarting Kubernetes pod does not always mean the application crashed.
Start with pod status, container state, previous state, and Kubernetes events. Then use previous logs, resource metrics, health probes, dependency checks, and deployment configuration to identify the actual cause.
The fastest path to the root cause is usually to understand why Kubernetes restarted the container before digging deeply into the application logs.
Join the conversation! Your thoughts help the community grow.