TopolearnDebugging odd latency, connection resets or lost source IPs requires knowing what the packets really do.
The Kubernetes network model requires every pod to have its own IP and reach every other pod without NAT. The CNI plugin implements this: overlays (VXLAN, Geneve), routed networks (BGP with Calico), or cloud-native VPC IPs per pod (AWS VPC CNI, Azure CNI).
A ClusterIP isn't bound to any interface. kube-proxy on each node watches Services and EndpointSlices and programs the kernel so that packets to ClusterIP:port are DNATed to one ready backend pod:
- iptables mode: a chain per Service picking a backend at random; rule updates get slow with many thousands of Services.
- IPVS mode: kernel load balancer with hash tables and more algorithms.
- nftables mode (GA since 1.33): the modern replacement for iptables mode, with better update performance.
eBPF dataplanes such as Cilium can replace kube-proxy entirely, doing the translation at socket or TC level.
Consequences:
- Load balancing is per connection, not per request: long-lived HTTP/2 or gRPC connections stick to one pod. Use client-side balancing (headless Service) or a mesh/L7 proxy.
- Traffic from outside through NodePort/LoadBalancer may hop nodes and be SNATed, hiding the client IP.
externalTrafficPolicy: Localpreserves it and avoids the hop, but only nodes with local pods receive traffic. internalTrafficPolicyand topology-aware routing can keep traffic in-zone to save cross-zone costs.
Example
# On a node using kube-proxy in iptables mode:
iptables -t nat -L KUBE-SERVICES -n | grep shop/api
# KUBE-SVC-XYZ tcp -- 0.0.0.0/0 10.96.112.7 /* shop/api:http cluster IP */ tcp dpt:80
iptables -t nat -L KUBE-SVC-XYZ -n
# KUBE-SEP-AAA statistic mode random probability 0.33333
# KUBE-SEP-BBB statistic mode random probability 0.50000
# KUBE-SEP-CCC
kubectl -n kube-system get cm kube-proxy -o yaml | grep -m1 'mode:'
# preserve client source IPs for an external Service
kubectl patch svc web -p '{"spec":{"externalTrafficPolicy":"Local"}}'Remember
- CNI gives pods routable IPs; kube-proxy or eBPF implements ClusterIPs
- Balancing is per connection: watch out with gRPC/HTTP2
- externalTrafficPolicy: Local keeps the client IP and avoids an extra hop
Watch out for
- Expecting a ClusterIP to spread gRPC requests across pods when clients hold one long connection.