January 12, 2025 12 min to read

Kubernetes Network Traffic Flow

Understanding network communication in CSP and On-premise environments

• image reference link

Overview

Understanding Kubernetes network traffic flow is crucial for proper cluster architecture, security, and troubleshooting. This guide explores networking in both Cloud Service Provider (CSP) and on-premise environments, detailing how traffic moves through different components of a Kubernetes cluster.

Network Traffic Paths

Kubernetes networking involves several distinct traffic paths, each with its own characteristics and components.

Network Environments and Traffic Flow

CSP Environment Traffic Flow

In cloud service provider environments (AWS, GCP, Azure, etc.), Kubernetes leverages cloud infrastructure for external connectivity.

Key Components in CSP Environments

1. External Load Balancer:
   • Provided by the cloud provider (e.g., AWS ELB/ALB, GCP Load Balancer, Azure Load Balancer)
   • Located outside the Kubernetes cluster in the cloud provider’s infrastructure
   • Automatically provisioned when a Service of type LoadBalancer is created
   • Routes external traffic to nodes in the cluster
2. Ingress Controller/Service:
   • Ingress Controllers handle HTTP/HTTPS traffic routing based on hostnames and paths
   • Services of type LoadBalancer expose non-HTTP protocols
   • Receive traffic from the cloud load balancer and route it to the appropriate pods
3. Pods:
   • End destinations for network traffic
   • Run application containers
   • Each has a unique IP address within the cluster

On-premise Environment Traffic Flow

On-premise Kubernetes deployments require different networking approaches since they don’t have cloud provider integration.

Key Components in On-premise Environments

1. External Load Balancer/Router:
   • Self-managed hardware or software load balancer (e.g., F5, NGINX, HAProxy)
   • Located within the on-premise data center
   • Manually configured to route traffic to Kubernetes nodes
2. Ingress Controller/Service:
   • Similar role as in CSP environments
   • Often exposed via NodePort services or with MetalLB
   • Receive traffic from the external load balancer and route it to appropriate pods
3. Pods:
   • Functionally identical to pods in CSP environments
   • Network connectivity may be implemented differently depending on CNI plugin choice

Ingress Controllers in Different Environments

Ingress Controllers in CSP Environments

In cloud environments, ingress controllers are tightly integrated with cloud provider load balancing services.

# Example AWS ALB Ingress Controller configuration
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: example-ingress
  annotations:
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/scheme: internet-facing
spec:
  rules:
  - host: example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: example-service
            port:
              number: 80

Ingress Controllers in On-premise Environments

In on-premise environments, the LoadBalancer service type doesn’t automatically receive an external IP without additional components.

On-premise deployments typically use one of these approaches:

1. NodePort Services:
   • Expose the Ingress Controller on a port on each node (30000-32767 range)
   • Configure external load balancer to distribute traffic to these ports across nodes
2. MetalLB or similar solutions:
   • Enables LoadBalancer service type in on-premise environments
   • Provides automatic IP assignment from a configured pool

MetalLB for On-premise Load Balancing

MetalLB is a popular solution for implementing LoadBalancer services in on-premise Kubernetes clusters.

MetalLB Configuration

MetalLB works in two modes:

1. Layer 2 Mode: Uses ARP (IPv4) or NDP (IPv6) to claim IP addresses on the local network
2. BGP Mode: Establishes BGP peering sessions with network routers for more advanced configurations

# Example MetalLB configuration
apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: production-pool
  namespace: metallb-system
spec:
  addresses:
  - 192.168.1.100-192.168.1.150

---
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
  name: l2-advertisement
  namespace: metallb-system
spec:
  ipAddressPools:
  - production-pool

When a LoadBalancer service is deployed with MetalLB:

1. MetalLB assigns an IP from the configured pool
2. The speaker component advertises the IP on the local network
3. Traffic sent to that IP is routed to the appropriate service
4. The service distributes traffic to the pods

Component Communication in Kubernetes

Kubernetes networking enables several types of communication between different components in a cluster.

Container-to-Container Communication

Containers within the same Pod share the same network namespace, enabling localhost communication.

Key Characteristics:

• Containers in the same pod share the same IP address
• They can communicate via localhost
• Shared volumes allow file-based communication
• Container ports must be unique within a pod
• This enables sidecar, ambassador, and adapter patterns

Pod-to-Pod Communication

Kubernetes ensures all pods can communicate directly with each other regardless of their node location.

Key Characteristics:

• Every pod has a unique IP address
• All pods can reach each other without NAT
• CNI (Container Network Interface) plugins implement this functionality
• Different CNI implementations (Calico, Flannel, Cilium, etc.) provide various features and performance characteristics
• Cross-node communication typically uses overlay networks or direct routing

Pod-to-Service Communication

Services provide stable endpoints for pods, handling internal load balancing and service discovery.

Key Components:

• Pod: Source of traffic, with its own IP address
• Virtual Interface: Each pod has an eth0 interface connecting to the node’s network
• Node: Hosts multiple pods and manages network traffic
• kube-proxy: Runs on each node, managing service routing using iptables, IPVS, or userspace proxying
• Service: Provides a stable ClusterIP that routes to a dynamic set of pod endpoints
• Cluster Network: Enables all communication across the cluster

Internet-to-Service Communication

External traffic enters the cluster through several different mechanisms.

Access Types:

1. LoadBalancer Service:
   • Automatically provisions a cloud load balancer in CSP environments
   • Requires MetalLB or similar solution in on-premise deployments
   • Provides stable external IP address
2. NodePort Service:
   • Opens a specific port on all nodes
   • Allows access via any node’s IP address
   • Port range is limited to 30000-32767 by default
3. Ingress:
   • HTTP/HTTPS-based routing layer
   • Enables hostname and path-based routing
   • Can provide SSL termination, authentication, and more sophisticated traffic management

Network Components and Features

Network Components Table

Key Network Features

Advanced Networking Concepts

Network Policies

Network Policies provide firewall-like rules for pod-to-pod communication, enabling micro-segmentation.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-frontend-to-backend
spec:
  podSelector:
    matchLabels:
      app: backend
  policyTypes:
  - Ingress
  ingress:
  - from:
    - podSelector:
        matchLabels:
          app: frontend
    ports:
    - protocol: TCP
      port: 8080

Service Mesh

Service meshes like Istio, Linkerd, and Consul provide advanced networking capabilities for microservices:

DNS in Kubernetes

CoreDNS provides service discovery within Kubernetes clusters:

• Each Service gets a DNS entry: <service-name>.<namespace>.svc.cluster.local
• Pods can have DNS entries (if enabled): <pod-ip-with-dashes>.<namespace>.pod.cluster.local
• DNS resolution is handled by CoreDNS pods in the kube-system namespace

Troubleshooting Kubernetes Networking

Common networking issues and diagnostic commands:

# Check if pods can reach each other
kubectl exec -it <pod-name> -- ping <other-pod-ip>

# Check DNS resolution from a pod
kubectl exec -it <pod-name> -- nslookup <service-name>

# Examine service configuration
kubectl describe service <service-name>

# Check endpoints for a service
kubectl get endpoints <service-name>

# Test connectivity to a service
kubectl exec -it <pod-name> -- curl <service-name>:<port>

# Check ingress configuration
kubectl describe ingress <ingress-name>

# View network policies
kubectl get networkpolicies

Key Points

References

• Kubernetes Networking Fundamentals
• Kubernetes Networking Guide
• Kubernetes Network Policies
• MetalLB Documentation
• Service Networking in Kubernetes
• DNS for Services and Pods
• Container Network Interface (CNI) Specification