Kubernetes applications can fail to communicate even when the pods themselves appear healthy. A service may return connection timeouts, an API request may never reach its destination, or traffic may be rejected somewhere between the Kubernetes node and a cloud network resource.
These problems become harder to diagnose when the Kubernetes cluster runs inside a cloud Virtual Private Cloud (VPC).
Kubernetes logs can tell you what an application sees, but they may not explain what happened to a packet before it reached the workload. VPC Flow Logs provide another layer of visibility by recording information about network traffic observed by supported cloud network interfaces.
This makes flow logs useful when investigating connectivity problems involving Kubernetes nodes, load balancers, private endpoints, databases, NAT gateways, firewall rules, and other cloud resources.
This article explains how to use VPC Flow Logs as part of a practical Kubernetes API connectivity troubleshooting process.
Understanding the Network Path
Before looking at logs, understand the path traffic is expected to follow.
For example, an application running in Kubernetes might communicate with a private API:
Kubernetes Pod
|
v
Kubernetes Node
|
v
VPC Network
|
v
Private Endpoint / Service
|
v
API Server
There may be several additional components between these points:
A timeout at the application layer does not identify which component caused the problem.
That is where flow logs can help.
What VPC Flow Logs Can Tell You
VPC Flow Logs generally provide metadata about network traffic rather than application-level request details.
Depending on the cloud provider and configuration, useful fields can include:
Field | Purpose |
|---|
Source address | Where traffic originated |
Destination address | Where traffic was sent |
Source port | Originating TCP/UDP port |
Destination port | Target service port |
Protocol | TCP, UDP, or another protocol |
Action | Whether traffic was accepted or rejected |
Timestamp | When traffic was observed |
Interface | Network interface associated with traffic |
Bytes | Amount of traffic observed |
The exact fields and behavior depend on the cloud provider and flow-log implementation.
Flow logs should therefore be treated as network evidence, not as a replacement for Kubernetes or application logs.
Start With the Kubernetes Error
Do not begin troubleshooting by searching millions of flow-log records.
Start with the actual application failure.
For example:
dial tcp 10.20.30.40:443: i/o timeout
This immediately gives you useful information:
Now you have something specific to investigate in the network layer.
Other useful errors include:
connection refused
no route to host
network is unreachable
context deadline exceeded
TLS handshake timeout
Each error can point toward a different class of problem.
Test Connectivity From Inside the Cluster
First confirm the problem from the same network environment as the application.
Create a temporary debugging pod when appropriate:
kubectl run network-debug \
--image=curlimages/curl \
--restart=Never \
--rm -it -- sh
Then test the destination:
curl -v https://10.20.30.40:443
For a TCP connectivity test:
nc -vz 10.20.30.40 443
If DNS is involved, test the hostname:
nslookup api.internal.example
This helps determine whether the problem is related to DNS, TCP connectivity, TLS, or the application itself.
Identify the Kubernetes Node
If the affected pod is running on a particular node, identify it:
kubectl get pod orders-api-7d8f9c6d4f-x7abc \
-n production \
-o wide
The output includes the node where the pod is running.
For example:
NAME READY STATUS IP NODE
orders-api-7d8f9c6d4f 1/1 Running 10.20.4.25 worker-02
The pod IP and node information are both useful during investigation.
Depending on the cloud networking architecture, flow logs may show traffic associated with the node's network interface rather than the pod IP directly.
This distinction is important.
Find the Relevant Network Interface
The next step is to identify which cloud network interface handled the traffic.
This depends on the Kubernetes networking model.
In some environments, pods have addresses associated with cloud network interfaces. In others, traffic may be translated or routed through the node.
You therefore need to understand how the cluster's CNI handles pod traffic.
Useful Kubernetes commands include:
kubectl get nodes -o wide
and:
kubectl describe node worker-02
Cloud-side network interface information can then be correlated with the node or pod networking configuration.
Do not assume that the pod IP will always appear directly in VPC Flow Logs.
Search Flow Logs Using a Narrow Time Window
Once you know:
search the flow logs using a narrow time range.
For example, if the application reported the failure at 10:15:32, investigate traffic around that time rather than searching an entire day.
Conceptually, you are looking for:
Source: 10.20.4.25
Destination: 10.20.30.40
Protocol: TCP
Port: 443
Time: 10:15
Action: ACCEPT / REJECT
The exact query syntax depends on the cloud provider and logging destination.
A narrow search reduces noise and makes packet-path analysis easier.
Distinguish Accepted and Rejected Traffic
One of the most useful pieces of information in flow logs is whether traffic was accepted or rejected.
Suppose the application reports:
connection timeout
and the corresponding flow-log records show rejected traffic to port 443.
That strongly suggests the traffic is being blocked somewhere in the cloud network path.
Possible causes include:
If flow logs show accepted traffic, the problem may be further along the network path or at the application layer.
However, an accepted flow does not prove that the application successfully completed its request.
Check Security Rules After Finding a Rejection
Once a rejected flow has been identified, inspect the network controls associated with the source and destination.
For example, verify:
Source
|
+-- Security group / firewall
|
+-- Route
|
+-- Network ACL
|
+-- Destination security controls
|
Destination
Check whether the destination port is allowed.
For an HTTPS service, the relevant port is commonly 443. For a PostgreSQL database, it may be 5432. For other services, use the actual application configuration rather than assuming the port.
The important point is to correlate the flow-log evidence with the network rule that should permit the traffic.
Investigate Routing Problems
A rejected flow is not the only possible problem.
Traffic can also fail because the source does not have a valid route to the destination.
For example:
Pod
|
v
Node
|
v
Route Table
|
X
Destination
Check the route configuration for the relevant subnet.
A private destination may require a route through a specific gateway, endpoint, transit connection, or peering relationship.
If the application is accessing a private cloud service, confirm that the destination is reachable from the subnet where the Kubernetes node or pod is running.
Check NAT and Internet Access
Applications running in private subnets may require NAT for outbound internet access.
A common path looks like:
Pod
|
Node
|
Private Subnet
|
NAT Gateway
|
Internet
|
External API
If the application can reach internal services but cannot reach external APIs, investigate the outbound path separately.
Check:
Private subnet route
NAT configuration
Internet gateway path
Firewall rules
Destination restrictions
Flow logs can help establish whether traffic reaches the expected network interface.
Do Not Ignore DNS
Connectivity problems sometimes look like network failures but are actually DNS problems.
For example:
curl https://api.example.com
may fail because the hostname resolves to an unexpected address.
Test DNS separately:
nslookup api.example.com
Then test the resolved address directly when appropriate:
curl -v https://203.0.113.10
Be careful when doing this with TLS because the hostname can affect certificate validation and HTTP routing.
The purpose of the test is to separate DNS resolution from network connectivity.
Correlate Kubernetes and VPC Logs
The most useful troubleshooting approach combines multiple sources.
Layer | Evidence |
|---|
Application | Timeout or connection error |
Kubernetes | Pod, node, Service state |
DNS | Name resolution result |
VPC Flow Logs | Network traffic metadata |
Firewall | Allow/reject rule |
Routing | Path to destination |
Destination | Server-side logs |
For example:
10:15:30 Application reports timeout
|
10:15:31 Pod connectivity test fails
|
10:15:32 Flow log shows REJECT
|
10:15:33 Firewall rule identified
|
10:15:40 Rule corrected
|
10:16:00 Connectivity succeeds
This type of correlation is much more reliable than changing network rules randomly until the application starts working.
Common Mistakes
Searching Flow Logs Without a Time Range
Large flow-log datasets can contain enormous amounts of traffic.
Always start with the application's failure timestamp.
Assuming Pod IP Equals VPC Source IP
The relationship between pod addresses and cloud network interfaces depends on the cluster networking architecture.
Understand the CNI and routing model before interpreting flow-log records.
Treating ACCEPT as Application Success
An accepted flow only provides network-level evidence. The application request may still fail because of TLS, authentication, application errors, or destination-side filtering.
Changing Firewall Rules Without Evidence
Opening broad network access may temporarily hide the problem while creating a security risk.
Use flow logs and routing information to identify the actual blocked path.
Ignoring Return Traffic
Network communication is bidirectional.
A request may leave the Kubernetes environment successfully while response traffic is blocked or routed incorrectly.
Always consider both directions when troubleshooting.
Best Practices
Enable flow logging for the network resources where it provides meaningful troubleshooting and security visibility.
Use centralized log storage and retention appropriate for your operational requirements.
Standardize timestamps across application, Kubernetes, and cloud logs.
Document the expected network path for critical applications.
Keep network rules under change management.
Use least-privilege firewall rules rather than broad allow rules.
Create dashboards or queries for recurring connectivity failures.
When possible, automate common diagnostic checks so engineers do not have to manually inspect multiple systems during an incident.
Advantages and Disadvantages
Advantages | Disadvantages |
|---|
Provides network-level visibility | Does not show application payloads |
Helps identify rejected traffic | Requires understanding of cloud networking |
Useful for Kubernetes connectivity issues | Large datasets can be difficult to search |
Supports incident investigation | Pod-to-interface correlation can be complex |
Helps validate firewall changes | Accepted traffic does not guarantee application success |
Troubleshooting Checklist
When a Kubernetes application cannot reach an API or cloud service:
Capture the exact application error and timestamp.
Identify the affected pod and node.
Confirm DNS resolution.
Test connectivity from inside the cluster.
Identify the expected destination IP and port.
Determine the relevant cloud network interface.
Search VPC Flow Logs around the failure time.
Check whether traffic was accepted or rejected.
Inspect routing and firewall configuration.
Check return traffic and destination-side logs.
Correct the specific network issue.
Repeat the connectivity test.
This process prevents troubleshooting from becoming a trial-and-error exercise.
Summary
VPC Flow Logs provide valuable network-level evidence when Kubernetes applications cannot communicate with cloud services or other network destinations.
The most effective approach is to start with the application error, identify the affected pod and network path, and then correlate Kubernetes information with VPC Flow Logs, routing configuration, and firewall rules.
Flow logs should not be used alone. They become much more useful when combined with Kubernetes events, DNS tests, application logs, and cloud network configuration.
By following a structured troubleshooting process, developers and platform teams can identify whether a connectivity problem is caused by routing, firewall rules, network policy, NAT, DNS, or the destination service itself without making unnecessary changes to the environment.
Join the conversation! Your thoughts help the community grow.