January 12, 2025 8 min to read

Kubernetes Network Traffic Flow

• image reference link

Overview

Understanding Kubernetes network traffic flow is crucial for both CSP (Cloud Service Provider) and on-premise environments.

Network Environments Traffic Flow

CSP Environment Traffic Flow

• External Load Balancer: The cloud provider provides a Load Balancer for routing external traffic into the cluster. It is located outside the cluster and forwards requests from outside to the service or the ingress controller.
• Ingress Controller/Service: Ingress Controller handles HTTP/HTTPS traffic, and the service of type: LoadBalancer handles other protocols. They receive traffic from the cloud load balancer and route it to the appropriate pod.
• Pods: Pod is the minimum unit that runs the container of an application. Traffic is delivered to them through Service or Ingress.

On-premise Environment Traffic Flow

• External Load Balancer/Router: In an on-premises environment, a self-configured load balancer or router routes external traffic into the cluster. It is located within the on-premises data center.
• Ingress Controller/Service: This step is similar to the CSP environment. The Ingress Controller handles HTTP/HTTPS traffic, and the service of type: LoadBalancer handles other protocols. They receive traffic from an external Load Balancer/Router and route it to an appropriate Pod.
• Pods: Even in an on-premises environment, Pods are the same as the minimum unit that runs the container of the application.

Ingress Controllers in CSP Environments

In CSP environments, ingress controllers are often integrated with load balancer services from cloud providers.

When the ingress controller service type is set to LoadBalancer, the cloud provider automatically assigns an external IP address (usually a public IP), which becomes the IP address of the cloud load balancer.

Through this allocated IP address, external traffic is routed to the ingress controller through the cloud load balancer, and finally delivered to the appropriate path.

I describe the relationship between an ingress controller and a load balancer controller in a cloud service provider (CSP) environment.

1. Ingress Controller
   • The ingress controller is software that runs within the Kubernetes cluster. Observe the ingress resource and implement the routing rules defined in this resource.
   • The ingress controller serves to deliver traffic to appropriate services and pads.
2. Load Balancer Controller
   • In a cloud environment, load balancer controllers are usually separate services managed by cloud providers.
   • The service serves to route external traffic to the Kubernetes cluster.
3. Integration operations
   • When a Kubernetes service is created as a LoadBalancer type (for example, for an ingress controller), the cloud provider automatically allocates an external IP address and configures a load balancer.
   • This load balancer takes external traffic and passes it to the ingress controller. The ingress controller then routes the traffic to the appropriate service or pad within the Kubernetes.
4. CSP environment features
   • In CSP environments, the ingress controller and load balancer are typically tightly integrated, and users do not need to manage the load balancer separately.
   • Instead, by configuring Kubernetes services and defining ingress resources, you can automatically take advantage of the cloud provider’s load balancing capabilities.

Ingress Controllers in On-premise Environments

• In an on-premises environment, LoadBalancer type services are not automatically assigned external IPs. Instead, you should use an external load balancer or router that is typically managed by an on-premises network. In this case, you manually set the IP address to the external load balancer or router and configure that IP to route traffic to the ingress controller or service.

• In an on-premises environment, you can access the ingress controller using the NodePort or ClusterIP service type, and an external load balancer routes traffic to that service type. Therefore, when the ingress controller has an external load balancer IP (IP) in the on-premise server, it is usually the result of a network configuration or manual setup. Unlike a CSP environment, automatic IP allocation is not common in an on-premise environment.

MetalLB(Onpremise LoadBalancer)

For example, in a Metal LB setting, you can specify an IP address range as follows.

apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: example-pool
spec:
  addresses:
  - 192.168.1.100-192.168.1.150

• MetalLB can assign IP addresses between 192.168.1.100 and 192.168.1.150 to LoadBalancer services.
• When the ingress controller is deployed as a LoadBalancer service, MetalLB automatically allocates the IP address from this IP pool and provides it to the ingress controller. This IP address is used to route traffic from the outside to the ingress controller.
• In conclusion, MetalLB enables the use of the LoadBalancer service in an on-premises environment in a similar way to the cloud, allowing an IP address to be automatically assigned to the ingress controller.

Component Communication (Container, Pod, Service)

Kubernetes networking is designed to allow various types of entities within Kubernetes to communicate.

• Container-to-container networking
• Pod-to-Pod networking
• Pod-to-Service networking
• Internet-to-Service networking

Container-to-Container

• Kubernetes within a single Pod allows multiple containers to share the same network namespace. This means that containers within the same Pod share the same IP address and port space, so they can communicate with each other using localhost.
• These intra-Pod communications are critical for containers that must work closely together (e.g., primary application containers and sidecar containers that record or monitor primary applications).

Pod-to-Pod

• With Kubernetes, all nodes have a range of CIDR IPs designated for the pods. This way, all the pods will receive a unique IP address that can be seen in other pods in the cluster.
• Kubernetes ensures that the pads can communicate with each other between nodes without Network Address Translation (NAT). Each pad gets a unique IP address, so this IP address can be used to communicate with other pads in the cluster regardless of which node all the pads are at. -This is achieved through basic network plug-ins implementing container network interface (CNI) standards, and various implementations (e.g., Calico, Flannel, or Cilium) provide a variety of ways to enable this node-to-node inter-pad communication.

Pod-to-Service

• Pod: Each pod is an instance of an application running within a Kubernetes cluster. A pod has its own IP address and can communicate with other pods.
• Virtual bridge: It is connected through eth0 and veth pairs in the pod. This bridge serves as an intermediate communication that allows the pod to communicate with the node’s network.
• Node: A physical or virtual machine in a cluster. Each node hosts several pad and manages network traffic through kube-proxy.
• kube-proxy: Runs within a node, uses iptables or IPVS to route traffic to the service’s ClusterIP, and delivers traffic to the appropriate pad according to the service’s pad selection algorithm (e.g., round robin).
• Service: represents a logical set of pods and allows clients to access the pods in a consistent way without having to know the individual IPs of the pods. ClusterIP can be used to access the pods inside the cluster and may be exposed to the outside as needed (e.g., NodePort, LoadBalancer).
• Cluster Network: A network that enables communication throughout a cluster. Node-to-node communication and pad-to-pad communication are performed through this network.

Internet-to-Service

• Egress: outbound traffic
• Ingress: inbound traffic

Network Components Table

Key Network Features

Reference

• Kubernetes Networking Fundamentals
• Azure Kubernetes Network Policies
• Kubernetes Network Routing
• Kubernetes Networking Guide