January 15, 2025 22 min to read

What is AWS Ingress Annotations?

image reference link

Overview

Learn about AWS Load Balancer Controller Ingress Annotations and how they manage traffic routing to Kubernetes services using Application Load Balancers (ALB). These annotations provide a powerful way to customize and configure how external traffic is routed to your applications running on EKS or other Kubernetes deployments on AWS.

What is Ingress Annotations?

In AWS, the AWS load balancer controller configures and manages the behavior of the Application Load Balancer (ALB) that routes traffic to the Kubernetes service using annotations for incoming resources.

Annotations allow you to specify settings such as SSL, routing behavior, and security groups directly within the Kubernetes Ingress resource.

How Load Balancer Controller Works

• Resource Creation: Create or update ALBs, their listeners, and target groups based on the specified annotations.
• Management: Manage status checks, security settings, routing rules, SSL certificates, and other defined configurations.
• Dynamic Updates: When annotations change, the controller updates ALB with new settings without manual intervention, allowing Kubernetes native resources to manage complex configurations.
• Integration with AWS: The controller integrates directly with AWS services, automatically provisioning AWS resources and connecting them to your Kubernetes resources.

Ingress Groups

• In AWS, ingress group is a feature that allows multiple ingress resources to be integrated into a single load balancer (ALB) for more efficient management.
• This allows applications from different Kubernetes namespaces or different teams to share the same ALB while setting different Ingress rules.
• Ingress groups can significantly reduce costs and increase management efficiency by consolidating multiple ALBs into a single one.
• By default, ingress does not belong to an ingress group (it exists as an “implicit ingress group”), functioning as an independent entity with its own ALB.

Ingress Group Name Settings

• Use the alb.ingress.kubernetes.io/group.name annotation to define the group name.
• Ingress resources with the same group name will share one ALB, consolidating resources.
• The ALB assigned to an ingress group is tagged with ingress.k8s.aws/stack and the group name as the value.
• For resources that do not use ingress groups, the tag value is set in the format of namespace/ingressname.
• If you change the groupName assigned to an ingress resource, the ingress moves from the existing group to a new ingress Group, managed by the ALB of the new group.
• If there is no ALB in the new IngressGroup, a new ALB is automatically created.
• Example: alb.ingress.kubernetes.io/group.name: my-team.awesome-group

Ingress Group Ranking

• You can specify the order using the alb.ingress.kubernetes.io/group.order annotation.
• The ingress rule with a lower order value is applied first, allowing you to prioritize specific rules when multiple ingresses share the same path or host.
• This is particularly useful when you have overlapping rule definitions and need to ensure specific traffic patterns are handled correctly.
• Example: alb.ingress.kubernetes.io/group.order: 10

Ingress Group Example

In the example below, two ingress resources share a group called my-shared-group, so one ALB processes the request.

# Ingress for Service A
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: service-a-ingress
  annotations:
    alb.ingress.kubernetes.io/group.name: "my-shared-group"
    alb.ingress.kubernetes.io/group.order: "10"
spec:
  rules:
    - host: "service-a.example.com"
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: service-a
                port:
                  number: 80

# Ingress for Service B
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: service-b-ingress
  annotations:
    alb.ingress.kubernetes.io/group.name: "my-shared-group"
    alb.ingress.kubernetes.io/group.order: "20"
spec:
  rules:
    - host: "service-b.example.com"
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: service-b
                port:
                  number: 80

• In this example, service-a and service-b both belong to the ingress group my-shared-group, with service-a having priority (order 10) over service-b (order 20).
• This means both services will be exposed through the same ALB, reducing costs and simplifying management.

Ingress Traffic Listening

listen-ports: Specify ports for ALB

[{"HTTP": 80}, {"HTTPS": 443}]

ssl-redirect: Enable SSL redirection

'443'

ip-address-type: IPv4 or dualstack support

ipv4 or dualstack

waf-acl-id: Associate AWS WAF WebACL

arn:aws:wafv2:region:account:webacl/my-web-acl/id

listen-ports

• Specifies the port(s) that the ALB listens on for incoming traffic.
• Example: alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS": 443}, {"HTTP": 8080}, {"HTTPS": 8443}]'
• Since it applies to all Ingress within IngressGroup, each Ingress can define its own port, and the rule applies only to the ports specified in each Ingress.
• If the same receiving port is defined by multiple ingress within the IngressGroup, it is applied in the order of the IngressGroup.
• Default behavior:
  • If listen-ports is not specified and no certificate is configured, the default is {"HTTP": 80}
  • If a certificate is provided, the default becomes {"HTTPS": 443}
• Warning: You cannot have duplicate load balancer ports in the same group (Exception: when alb.ingress.kubernetes.io/group.order is specified)
• Since ALB integrates these settings for all Ingress in the group, defining listen-ports in one Ingress in the IngressGroup is sufficient.

ssl-redirect

• Enables SSL redirection by specifying the port to which HTTP traffic is redirected, typically HTTPS port 443.
• Example: alb.ingress.kubernetes.io/ssl-redirect: '443'
• SSL redirection affects all ingress in the group if defined in the ingressGroup.
• When SSL redirection is enabled, all HTTP listeners are configured by default to redirect to HTTPS, ignoring other HTTP rules.
• This is recommended for production environments to ensure secure communication.

ip-address-type

• Specifies whether ALB should use ipv4 or dualstack for IPv4 and IPv6 traffic support.
• Example: alb.ingress.kubernetes.io/ip-address-type: ipv4 or alb.ingress.kubernetes.io/ip-address-type: dualstack
• ipv4: Configure ALB to use only IPv4 addresses, i.e., handle IPv4 traffic exclusively.
• dualstack: ALB supports both IPv4 and IPv6 traffic. When set to dualstack, ALB has both IPv4 and IPv6 addresses, allowing traffic to be routed in both address types.
• Note: For dualstack to work properly, your VPC and subnets must be configured to support IPv6.

waf-acl-id

• Associates a AWS WAF WebACL with your ALB for additional security protections.
• Example: alb.ingress.kubernetes.io/waf-acl-id: arn:aws:wafv2:region:account:webacl/my-web-acl/id
• The WebACL provides protection against common web exploits like SQL injection, cross-site scripting, and more.
• For this to work, the appropriate IAM permissions must be granted to the load balancer controller.

customer-owned-ipv4-pool

• Specifies the customer-owned IPv4 address pool for ALB when using the Outpost environment.
• Warning: This annotation cannot be changed after creation, and you will need to recreate the Ingress to change it.
• Example: alb.ingress.kubernetes.io/customer-owned-ipv4-pool: ipv4pool-coip-xxxxxxx
• This is specifically useful for AWS Outposts deployments where you might want to use your on-premises IP addresses.

Full Example

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: secure-app-ingress
  annotations:
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS": 443}]'
    alb.ingress.kubernetes.io/ssl-redirect: '443'
    alb.ingress.kubernetes.io/certificate-arn: 'arn:aws:acm:region:account:certificate/cert-id'
    alb.ingress.kubernetes.io/ip-address-type: 'dualstack'
    alb.ingress.kubernetes.io/waf-acl-id: 'arn:aws:wafv2:region:account:webacl/my-web-acl/id'
spec:
  ingressClassName: alb
  rules:
    - host: secure-app.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: secure-app-service
                port:
                  number: 80

Ingress Traffic Routing

load-balancer-name: Specify custom ALB name

my-awesome-alb

target-type: Routing to instances or IP addresses

instance or ip

backend-protocol: Protocol for backend traffic

HTTP, HTTPS

target-group-attributes: Target group configuration

stickiness.enabled=true

load-balancer-name

• Allows you to specify a custom name for the ALB created by the controller.
• Example: alb.ingress.kubernetes.io/load-balancer-name: my-app-alb
• All Ingress resources within an IngressGroup must use the same load balancer name.
• The name cannot exceed 32 characters, and an error occurs when this limit is exceeded.
• This is useful for easier identification and management of resources in the AWS console.

target-type

• Defines how traffic from the load balancer should be routed to your application.
• Example: alb.ingress.kubernetes.io/target-type: instance or alb.ingress.kubernetes.io/target-type: ip
• instance mode:
  • Routes traffic to all EC2 instances in the cluster via NodePort that is open for the service.
  • When using instance mode, the service must use NodePort or LoadBalancer type.
  • This is the default and works well for most standard deployments.
• ip mode:
  • Routes traffic directly to the pod IPs.
  • The network plugin must use the secondary IP address of the ENI for pod IPs to use ip mode (e.g., amazon-vpc-cni-k8s).
  • IP mode is required for sticky sessions to work with application load balancer.
  • Service type is not important when using ip mode.
  • This mode offers more direct routing but has specific network requirements.

target-node-labels

• Specifies which nodes should be included in the target group registration when using the instance target type.
• Example: alb.ingress.kubernetes.io/target-node-labels: label1=value1, label2=value2
• This is useful for directing traffic to specific node groups (e.g., sending certain traffic only to GPU nodes).
• Only nodes with matching labels will be registered as targets in the ALB’s target group.

backend-protocol

• Specifies the protocol used to route traffic to your service backends.
• Example: alb.ingress.kubernetes.io/backend-protocol: HTTPS
• Supported values: HTTP, HTTPS, TCP, TLS, UDP, TCP_UDP
• Default is HTTP if not specified.
• Should match the protocol your application is listening on.
• Using HTTPS or TLS requires proper SSL configuration on your application.

backend-protocol-version

• Specifies the application protocol version used to route traffic to the pods.
• Example:
  • alb.ingress.kubernetes.io/backend-protocol-version: HTTP2
  • alb.ingress.kubernetes.io/backend-protocol-version: GRPC
• Only valid when HTTP or HTTPS is used as a backend protocol.
• Useful for modern applications that leverage HTTP/2 or gRPC for more efficient communication.

target-group-attributes

• Allows you to set attributes for the target groups created by the controller.
• Example: alb.ingress.kubernetes.io/target-group-attributes: stickiness.enabled=true
• Common use cases include:
  • Setting up sticky sessions
  • Configuring deregistration delay
  • Adjusting health check settings
  • Load balancing algorithm tuning

health-check-* annotations

• A set of annotations to configure the health check behavior for your service.
• Examples:
  • alb.ingress.kubernetes.io/healthcheck-protocol: HTTP
  • alb.ingress.kubernetes.io/healthcheck-port: traffic-port
  • alb.ingress.kubernetes.io/healthcheck-path: /health
  • alb.ingress.kubernetes.io/healthcheck-interval-seconds: '15'
  • alb.ingress.kubernetes.io/healthcheck-timeout-seconds: '5'
  • alb.ingress.kubernetes.io/success-codes: '200'
  • alb.ingress.kubernetes.io/healthy-threshold-count: '2'
  • alb.ingress.kubernetes.io/unhealthy-threshold-count: '2'
• These settings determine how the ALB checks if your application is healthy and ready to receive traffic.

actions.${action-name} and conditions.${conditions-name}

• Provides a method for configuring advanced load balancer listener rules beyond what is possible with standard Ingress specifications.
• actions.${action-name}: Defines what should happen for a specific rule, like redirects or fixed responses.
• conditions.${conditions-name}: Defines additional matching conditions beyond host/path.
• The serviceName in these annotations should match the serviceName in the Ingress rule, and the servicePort should be set to “use-annotation”.

Here’s an example showing how to use advanced actions and conditions:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  namespace: default
  name: ingress
  annotations:
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/actions.rule-path1: >
      {"type":"fixed-response","fixedResponseConfig":{"contentType":"text/plain","statusCode":"200","messageBody":"Host is www.example.com OR anno.example.com"}}
    alb.ingress.kubernetes.io/conditions.rule-path1: >
      [{"field":"host-header","hostHeaderConfig":{"values":["anno.example.com"]}}]
    # More actions and conditions...
spec:
  ingressClassName: alb
  rules:
    - host: www.example.com
      http:
        paths:
          - path: /path1
            pathType: Exact
            backend:
              service:
                name: rule-path1
                port:
                  name: use-annotation
          # More paths...

These advanced rules allow for complex traffic routing scenarios including:

• Multiple host conditions
• Path pattern matching
• HTTP header-based routing
• HTTP method-based routing
• Query string parameter routing
• Source IP filtering
• Combinations of multiple conditions

Ingress Access Control

Scheme: Define ALB as internal or internet-facing

alb.ingress.kubernetes.io/scheme

CIDR: Restrict IP ranges

alb.ingress.kubernetes.io/inbound-cidrs

Security Groups: Specify security groups

alb.ingress.kubernetes.io/security-groups

Subnet Discovery: Specify subnets for ALB

alb.ingress.kubernetes.io/subnets

Auth: Configure authentication

alb.ingress.kubernetes.io/auth-*

scheme

• Used to define whether the ALB should be accessible from the internet or only within your VPC.
• Example: alb.ingress.kubernetes.io/scheme: internal or alb.ingress.kubernetes.io/scheme: internet-facing
• internet-facing: Makes ALB accessible publicly via the internet. The ALB will have a public IP and DNS name.
• internal: Makes ALB private by restricting it to be accessible only within a VPC or connected networks.
• Use internal for services that should only be accessible within your private network, and internet-facing for public-facing applications.
• This setting affects all Ingress resources within an IngressGroup.

inbound-cidrs

• Restricts access to the ALB by allowing only the specified IP ranges (CIDRs) to connect.
• Example: alb.ingress.kubernetes.io/inbound-cidrs: 10.0.0.0/24, 192.168.1.0/24
• Particularly useful for controlling access to private applications or internal resources by limiting the range of IPs that can access the ALB.
• If Ingress is part of an IngressGroup, the inbound-cidrs annotation applies to all Ingress resources in the group.
• However, inbound-cidrs applies only to ports defined in that specific Ingress.
• If multiple ingresses share the same ingress port, you must define inbound-cidrs on only one of those ingresses to prevent collisions.
• Specifying alb.ingress.kubernetes.io/security-groups in Ingress will cause inbound-cidrs annotations to be ignored.
• Default behavior if not specified:
  • 0.0.0.0/0 (all IPv4 addresses) will be allowed if the ALB’s IPAddressType is set to ipv4.
  • 0.0.0.0/0 and ::/0 (all IPv4 and IPv6 addresses) will be allowed if the ALB’s IPAddressType is set to dualstack.

security-groups

• Specifies the security groups to attach to the load balancer.
• Example: alb.ingress.kubernetes.io/security-groups: sg-xxxx, nameOfSg1, nameOfSg2
• You can specify security groups by ID or by name.
• Without this annotation, the controller automatically creates a security group, attaches it to the load balancer, and configures it to allow traffic on the listen-ports from the inbound-cidrs.
• The controller also modifies this security group to allow inbound traffic to the Node/Pod security groups.
• Use this annotation when you want to apply your own custom security group policies.

manage-backend-security-group-rules

• Controls whether the controller should manage security group rules for traffic from the ALB to backend targets.
• Example: alb.ingress.kubernetes.io/manage-backend-security-group-rules: "true"
• This only applies when you specify custom security groups using the alb.ingress.kubernetes.io/security-groups annotation.
• Setting this to "false" means you’ll need to manually configure the rules to allow traffic from the ALB to your backend pods/nodes.

subnets

• Specifies which subnets the ALB should be created in.
• Example: alb.ingress.kubernetes.io/subnets: subnet-xxxx, subnet-yyyy, subnet-zzzz
• You can specify subnets by ID or by tags.
• By default, the controller automatically discovers subnets with the appropriate tags.
• For internet-facing ALBs, the controller looks for subnets tagged with kubernetes.io/role/elb: 1.
• For internal ALBs, it looks for subnets tagged with kubernetes.io/role/internal-elb: 1.
• This annotation is useful when you need to override the automatic subnet discovery.

auth-type and related annotations

The AWS Load Balancer Controller supports various authentication mechanisms through annotations:

OIDC Authentication

alb.ingress.kubernetes.io/auth-type: oidc
alb.ingress.kubernetes.io/auth-idp-oidc-config: '{"issuer":"https://example.com","secretName":"oidc-secret"}'
alb.ingress.kubernetes.io/auth-on-unauthenticated-request: authenticate
alb.ingress.kubernetes.io/auth-scope: openid

Cognito Authentication

alb.ingress.kubernetes.io/auth-type: cognito
alb.ingress.kubernetes.io/auth-idp-cognito: '{"userPoolARN":"arn:aws:cognito-idp:region:account:userpool/id","userPoolClientID":"client-id"}'
alb.ingress.kubernetes.io/auth-on-unauthenticated-request: authenticate
alb.ingress.kubernetes.io/auth-scope: openid

These authentication mechanisms allow you to secure your applications by ensuring users are authenticated before accessing them.

Example: Securing an Internal Application

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: secure-internal-app
  annotations:
    alb.ingress.kubernetes.io/scheme: internal
    alb.ingress.kubernetes.io/inbound-cidrs: 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16
    alb.ingress.kubernetes.io/security-groups: sg-private-apps
    alb.ingress.kubernetes.io/auth-type: cognito
    alb.ingress.kubernetes.io/auth-idp-cognito: '{"userPoolARN":"arn:aws:cognito-idp:region:account:userpool/id","userPoolClientID":"client-id"}'
spec:
  ingressClassName: alb
  rules:
    - host: internal-app.company.local
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: internal-app
                port:
                  number: 80

This example creates an internal ALB that:

• Is only accessible from private IP ranges
• Uses a specific security group
• Requires Cognito authentication
• Routes traffic to an internal application

SSL/TLS Configuration

certificate-arn: ARN of ACM certificate

arn:aws:acm:region:account:certificate/cert-id

ssl-policy: Security policy for TLS

ELBSecurityPolicy-TLS-1-2-2017-01

ssl-redirect: Redirect HTTP to HTTPS

'443'

certificate-arn

• Specifies the ARN of an AWS Certificate Manager (ACM) certificate to use for TLS termination.
• Example: alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:region:account:certificate/cert-id
• You can specify multiple certificate ARNs by separating them with commas.
• The first certificate in the list will be the default certificate.
• When multiple certificates are provided, the ALB will use Server Name Indication (SNI) to select the appropriate certificate based on the client’s requested hostname.
• If you’re using IngressGroups, all ingresses in the group should use the same certificates to avoid conflicts.

ssl-policy

• Defines the security policy that determines which ciphers and protocols are supported by the ALB.
• Example: alb.ingress.kubernetes.io/ssl-policy: ELBSecurityPolicy-TLS-1-2-2017-01
• AWS provides several predefined policies:
  • ELBSecurityPolicy-2016-08: Recommended for general use
  • ELBSecurityPolicy-TLS-1-2-2017-01: Supports TLS 1.2 only
  • ELBSecurityPolicy-TLS-1-1-2017-01: Supports TLS 1.1 and later
  • ELBSecurityPolicy-2015-05: Older policy with broader compatibility
  • ELBSecurityPolicy-FS-2018-06: Forward secrecy policy
• Choose a policy that balances security and compatibility with your users’ browsers/clients.
• For maximum security, use ELBSecurityPolicy-FS-1-2-Res-2020-10 which supports only secure ciphers and TLS 1.2+.

Example: Configuring HTTPS with Multiple Certificates

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: multi-domain-app
  annotations:
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS": 443}]'
    alb.ingress.kubernetes.io/certificate-arn: >-
      arn:aws:acm:region:account:certificate/cert1,
      arn:aws:acm:region:account:certificate/cert2
    alb.ingress.kubernetes.io/ssl-policy: ELBSecurityPolicy-FS-2018-06
    alb.ingress.kubernetes.io/ssl-redirect: '443'
spec:
  ingressClassName: alb
  rules:
    - host: app.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: app-service
                port:
                  number: 80
    - host: app.another-domain.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: app-service
                port:
                  number: 80

This example:

• Configures the ALB to listen on both HTTP (80) and HTTPS (443)
• Uses two different SSL certificates (the ALB will select the right one based on the domain)
• Enforces a strong SSL policy with forward secrecy
• Redirects all HTTP traffic to HTTPS

Advanced Features and Use Cases

Cross-Zone Load Balancing: Even distribution across AZs

load_balancing.cross_zone.enabled=true

Idle Timeout: Connection idle time

idle_timeout.timeout_seconds=60

HTTP/2 Support: Enable HTTP/2 protocol

routing.http2.enabled=true

Access Logging: Log traffic to S3

access_logs.s3.enabled=true

Cross-Zone Load Balancing

• Controls whether load balancing is distributed across all availability zones.
• Example: alb.ingress.kubernetes.io/load-balancer-attributes: load_balancing.cross_zone.enabled=true
• Ensures even distribution of traffic across all registered targets regardless of AZ.

Idle Timeout

• Configures how long connections can remain idle before being closed.
• Example: alb.ingress.kubernetes.io/load-balancer-attributes: idle_timeout.timeout_seconds=60
• Default is 60 seconds, but can be increased for applications with longer processing times.

HTTP/2 Support

• Enables HTTP/2 support on the ALB.
• Example: alb.ingress.kubernetes.io/load-balancer-attributes: routing.http2.enabled=true
• HTTP/2 offers improved performance through binary framing, multiplexing, and header compression.

Access Logging

• Configures ALB access logs to be stored in an S3 bucket.
• Example: alb.ingress.kubernetes.io/load-balancer-attributes: access_logs.s3.enabled=true,access_logs.s3.bucket=my-log-bucket,access_logs.s3.prefix=my-app-logs
• Useful for troubleshooting, security analysis, and traffic monitoring.

Best Practices

Troubleshooting Tips

Reference

• AWS Load Balancer Controller Documentation
• AWS Blog: Exposing Kubernetes Applications
• AWS Load Balancer Controller Annotations
• AWS EKS Documentation
• AWS ALB Documentation