Kubernetes applications can fail to communicate even when the pods themselves appear healthy. A service may return connection timeouts, an API request may never reach its destination, or traffic may be rejected somewhere between the Kubernetes node and a cloud network resource.

These problems become harder to diagnose when the Kubernetes cluster runs inside a cloud Virtual Private Cloud (VPC).

Kubernetes logs can tell you what an application sees, but they may not explain what happened to a packet before it reached the workload. VPC Flow Logs provide another layer of visibility by recording information about network traffic observed by supported cloud network interfaces.

This makes flow logs useful when investigating connectivity problems involving Kubernetes nodes, load balancers, private endpoints, databases, NAT gateways, firewall rules, and other cloud resources.

This article explains how to use VPC Flow Logs as part of a practical Kubernetes API connectivity troubleshooting process.

Understanding the Network Path

Before looking at logs, understand the path traffic is expected to follow.

For example, an application running in Kubernetes might communicate with a private API:

Code
Kubernetes Pod
     |
     v
Kubernetes Node
     |
     v
VPC Network
     |
     v
Private Endpoint / Service
     |
     v
API Server

There may be several additional components between these points:

  • Container Network Interface (CNI)

  • Route tables

  • Security groups or firewall rules

  • Network ACLs

  • NAT gateway

  • Load balancer

  • Private endpoint

  • Cloud firewall

  • Service mesh

A timeout at the application layer does not identify which component caused the problem.

That is where flow logs can help.

What VPC Flow Logs Can Tell You

VPC Flow Logs generally provide metadata about network traffic rather than application-level request details.

Depending on the cloud provider and configuration, useful fields can include:

Field

Purpose

Source address

Where traffic originated

Destination address

Where traffic was sent

Source port

Originating TCP/UDP port

Destination port

Target service port

Protocol

TCP, UDP, or another protocol

Action

Whether traffic was accepted or rejected

Timestamp

When traffic was observed

Interface

Network interface associated with traffic

Bytes

Amount of traffic observed

The exact fields and behavior depend on the cloud provider and flow-log implementation.

Flow logs should therefore be treated as network evidence, not as a replacement for Kubernetes or application logs.

Start With the Kubernetes Error

Do not begin troubleshooting by searching millions of flow-log records.

Start with the actual application failure.

For example:

Code
dial tcp 10.20.30.40:443: i/o timeout

This immediately gives you useful information:

  • Destination IP: 10.20.30.40

  • Destination port: 443

  • Protocol: TCP

  • Failure type: timeout

Now you have something specific to investigate in the network layer.

Other useful errors include:

Code
connection refused
no route to host
network is unreachable
context deadline exceeded
TLS handshake timeout

Each error can point toward a different class of problem.

Test Connectivity From Inside the Cluster

First confirm the problem from the same network environment as the application.

Create a temporary debugging pod when appropriate:

Bash
kubectl run network-debug \
  --image=curlimages/curl \
  --restart=Never \
  --rm -it -- sh

Then test the destination:

Bash
curl -v https://10.20.30.40:443

For a TCP connectivity test:

Bash
nc -vz 10.20.30.40 443

If DNS is involved, test the hostname:

Bash
nslookup api.internal.example

This helps determine whether the problem is related to DNS, TCP connectivity, TLS, or the application itself.

Identify the Kubernetes Node

If the affected pod is running on a particular node, identify it:

Bash
kubectl get pod orders-api-7d8f9c6d4f-x7abc \
  -n production \
  -o wide

The output includes the node where the pod is running.

For example:

Code
NAME                     READY   STATUS    IP           NODE
orders-api-7d8f9c6d4f    1/1     Running   10.20.4.25   worker-02

The pod IP and node information are both useful during investigation.

Depending on the cloud networking architecture, flow logs may show traffic associated with the node's network interface rather than the pod IP directly.

This distinction is important.

Find the Relevant Network Interface

The next step is to identify which cloud network interface handled the traffic.

This depends on the Kubernetes networking model.

In some environments, pods have addresses associated with cloud network interfaces. In others, traffic may be translated or routed through the node.

You therefore need to understand how the cluster's CNI handles pod traffic.

Useful Kubernetes commands include:

Bash
kubectl get nodes -o wide

and:

Bash
kubectl describe node worker-02

Cloud-side network interface information can then be correlated with the node or pod networking configuration.

Do not assume that the pod IP will always appear directly in VPC Flow Logs.

Search Flow Logs Using a Narrow Time Window

Once you know:

  • Source IP

  • Destination IP

  • Destination port

  • Approximate failure time

  • Relevant network interface

search the flow logs using a narrow time range.

For example, if the application reported the failure at 10:15:32, investigate traffic around that time rather than searching an entire day.

Conceptually, you are looking for:

VB.NET
Source:      10.20.4.25
Destination: 10.20.30.40
Protocol:    TCP
Port:        443
Time:        10:15
Action:      ACCEPT / REJECT

The exact query syntax depends on the cloud provider and logging destination.

A narrow search reduces noise and makes packet-path analysis easier.

Distinguish Accepted and Rejected Traffic

One of the most useful pieces of information in flow logs is whether traffic was accepted or rejected.

Suppose the application reports:

Code
connection timeout

and the corresponding flow-log records show rejected traffic to port 443.

That strongly suggests the traffic is being blocked somewhere in the cloud network path.

Possible causes include:

  • Security group rules

  • Network ACL rules

  • Cloud firewall rules

  • Incorrect routing

  • Destination-side restrictions

If flow logs show accepted traffic, the problem may be further along the network path or at the application layer.

However, an accepted flow does not prove that the application successfully completed its request.

Check Security Rules After Finding a Rejection

Once a rejected flow has been identified, inspect the network controls associated with the source and destination.

For example, verify:

Code
Source
  |
  +-- Security group / firewall
  |
  +-- Route
  |
  +-- Network ACL
  |
  +-- Destination security controls
  |
Destination

Check whether the destination port is allowed.

For an HTTPS service, the relevant port is commonly 443. For a PostgreSQL database, it may be 5432. For other services, use the actual application configuration rather than assuming the port.

The important point is to correlate the flow-log evidence with the network rule that should permit the traffic.

Investigate Routing Problems

A rejected flow is not the only possible problem.

Traffic can also fail because the source does not have a valid route to the destination.

For example:

Code
Pod
 |
 v
Node
 |
 v
Route Table
 |
 X
Destination

Check the route configuration for the relevant subnet.

A private destination may require a route through a specific gateway, endpoint, transit connection, or peering relationship.

If the application is accessing a private cloud service, confirm that the destination is reachable from the subnet where the Kubernetes node or pod is running.

Check NAT and Internet Access

Applications running in private subnets may require NAT for outbound internet access.

A common path looks like:

Code
Pod
 |
Node
 |
Private Subnet
 |
NAT Gateway
 |
Internet
 |
External API

If the application can reach internal services but cannot reach external APIs, investigate the outbound path separately.

Check:

  • Private subnet route

  • NAT configuration

  • Internet gateway path

  • Firewall rules

  • Destination restrictions

Flow logs can help establish whether traffic reaches the expected network interface.

Do Not Ignore DNS

Connectivity problems sometimes look like network failures but are actually DNS problems.

For example:

Bash
curl https://api.example.com

may fail because the hostname resolves to an unexpected address.

Test DNS separately:

Bash
nslookup api.example.com

Then test the resolved address directly when appropriate:

Bash
curl -v https://203.0.113.10

Be careful when doing this with TLS because the hostname can affect certificate validation and HTTP routing.

The purpose of the test is to separate DNS resolution from network connectivity.

Correlate Kubernetes and VPC Logs

The most useful troubleshooting approach combines multiple sources.

Layer

Evidence

Application

Timeout or connection error

Kubernetes

Pod, node, Service state

DNS

Name resolution result

VPC Flow Logs

Network traffic metadata

Firewall

Allow/reject rule

Routing

Path to destination

Destination

Server-side logs

For example:

Code
10:15:30 Application reports timeout
        |
10:15:31 Pod connectivity test fails
        |
10:15:32 Flow log shows REJECT
        |
10:15:33 Firewall rule identified
        |
10:15:40 Rule corrected
        |
10:16:00 Connectivity succeeds

This type of correlation is much more reliable than changing network rules randomly until the application starts working.

Common Mistakes

Searching Flow Logs Without a Time Range

Large flow-log datasets can contain enormous amounts of traffic.

Always start with the application's failure timestamp.

Assuming Pod IP Equals VPC Source IP

The relationship between pod addresses and cloud network interfaces depends on the cluster networking architecture.

Understand the CNI and routing model before interpreting flow-log records.

Treating ACCEPT as Application Success

An accepted flow only provides network-level evidence. The application request may still fail because of TLS, authentication, application errors, or destination-side filtering.

Changing Firewall Rules Without Evidence

Opening broad network access may temporarily hide the problem while creating a security risk.

Use flow logs and routing information to identify the actual blocked path.

Ignoring Return Traffic

Network communication is bidirectional.

A request may leave the Kubernetes environment successfully while response traffic is blocked or routed incorrectly.

Always consider both directions when troubleshooting.

Best Practices

Enable flow logging for the network resources where it provides meaningful troubleshooting and security visibility.

Use centralized log storage and retention appropriate for your operational requirements.

Standardize timestamps across application, Kubernetes, and cloud logs.

Document the expected network path for critical applications.

Keep network rules under change management.

Use least-privilege firewall rules rather than broad allow rules.

Create dashboards or queries for recurring connectivity failures.

When possible, automate common diagnostic checks so engineers do not have to manually inspect multiple systems during an incident.

Advantages and Disadvantages

Advantages

Disadvantages

Provides network-level visibility

Does not show application payloads

Helps identify rejected traffic

Requires understanding of cloud networking

Useful for Kubernetes connectivity issues

Large datasets can be difficult to search

Supports incident investigation

Pod-to-interface correlation can be complex

Helps validate firewall changes

Accepted traffic does not guarantee application success

Troubleshooting Checklist

When a Kubernetes application cannot reach an API or cloud service:

  1. Capture the exact application error and timestamp.

  2. Identify the affected pod and node.

  3. Confirm DNS resolution.

  4. Test connectivity from inside the cluster.

  5. Identify the expected destination IP and port.

  6. Determine the relevant cloud network interface.

  7. Search VPC Flow Logs around the failure time.

  8. Check whether traffic was accepted or rejected.

  9. Inspect routing and firewall configuration.

  10. Check return traffic and destination-side logs.

  11. Correct the specific network issue.

  12. Repeat the connectivity test.

This process prevents troubleshooting from becoming a trial-and-error exercise.

Summary

VPC Flow Logs provide valuable network-level evidence when Kubernetes applications cannot communicate with cloud services or other network destinations.

The most effective approach is to start with the application error, identify the affected pod and network path, and then correlate Kubernetes information with VPC Flow Logs, routing configuration, and firewall rules.

Flow logs should not be used alone. They become much more useful when combined with Kubernetes events, DNS tests, application logs, and cloud network configuration.

By following a structured troubleshooting process, developers and platform teams can identify whether a connectivity problem is caused by routing, firewall rules, network policy, NAT, DNS, or the destination service itself without making unnecessary changes to the environment.