Table Of Contents

• Introduction
• Installation
• Network Load Balancer
  • How To Provision
  • Subnet Discovery
  • Instance Mode
  • IP Mode
  • Annotations
  • Security Groups
    • Gotchas
  • Traffic Flow
    • Instance Mode
  • Integration Patterns
    • API Gateway
• Classic Load Balancer
  • How To Provision
  • Subnet Discovery
  • Annotations
  • Load Balancer Security Group
• References

Introduction

AWS Load Balancer Controller is a controller to help manage Elastic Load Balancers for a Kubernetes cluster.

• It satisfies Kubernetes Ingress resources by provisioning Application Load Balancers.

• It satisfies Kubernetes Service resources by provisioning Network Load Balancers.

Installation

You can install aws load balancer controller by following the instructions in below repo:

Pre-requisites -

• AWS LoadBalancer Controller >= v2.3.0

• Kubernetes >= v1.20 or EKS >= 1.16 or the following patch releases for Service type LoadBalancer 1.18.18+ for 1.18 or 1.19.10+ for 1.19

• Pods have native AWS VPC networking configured

  

  helm-aws-load-balancer-controller

  Helm chart for setting up AWS load balancer controller in your EKS cluster to provision LB

  • 0
    Stars
  • 0
    Fork

Network Load Balancer

How To Provision

To provision a Network Load Balancer, you need to create a service of type LoadBalancer.

apiVersion: v1
kind: Service
metadata:
  name: web-app
  namespace: platform
  labels:
    app: web-app
spec:
  ports:
    - name: https
      protocol: TCP
      port: 443
      targetPort: 443
  selector:
    app: web-app
type: LoadBalancer

Following actions are performed by the AWS Load balancer controller, when it sees a service object of type LoadBalancer:

1. NLB is created in AWS for the service. Depending on the annotations it’s either internal or external.

2. Target groups are created in either instance or ip mode depending on the annotations.

3. Listeners are created for each port detailed in the service definition.

4. Health checks are configured.

Subnet Discovery

AWS Load Balancer Controller auto discovers network subnets by default.

To be able to successfully do that you need to tag your subnets as follows:

Instance Mode

Instance target mode supports pods running on AWS EC2 instances. In this mode, AWS NLB sends traffic to the instances and the kube-proxy on the individual worker nodes forward it to the pods through one or more worker nodes in the Kubernetes cluster.

IP Mode

IP target mode supports pods running on AWS EC2 instances and AWS Fargate. In this mode, the AWS NLB targets traffic directly to the Kubernetes pods behind the service, eliminating the need for an extra network hop through the worker nodes in the Kubernetes cluster.

Annotations

Let’s look at some of the annotations that you can configure and their behaviors.

Security Groups

Network Load Balancers do not have associated security groups.

However, you can add any CIDR rules to the security group of your worker nodes.

This is achieved by making use of spec.loadBalancerSourceRanges property.

apiVersion: v1
kind: Service
metadata:
  name: web-app
  namespace: platform
  labels:
    app: web-app
spec:
  ports:
    - name: https
      protocol: TCP
      port: 443
      targetPort: 443
  selector:
    app: web-app
  type: LoadBalancer
  loadBalancerSourceRanges:
    - 10.1.0.0/16

Here, CIDR 10.1.0.0/16 is added to the worker nodes security group.

It’s critical to understand the importance of loadBalancerSourceRanges.

Gotchas

When you create a NLB for an application it adds below rules to the security group of the worker nodes:

Consider this setup - EKS having worker nodes across 3 AZ’s and you add one loadBalancerSourceRange.

This will result in 4 rules added to the worker node security group - 3 health check rules (1 per subnet) and 1 loadBalancerSourceRange rule.

So, for one NLB you end up with 4 rules.

Security groups have a default limit of 60 rules that can be added.

As you create many NLB’s/add more source ranges for your apps running in the same cluster, you will soon hit this limit and you can’t create any more NLB’s.

You can increase the quota but that multiplied by the quota for security groups per network interface cannot exceed 1,000.

Complete service file with the annotations will look as follows:

apiVersion: v1
kind: Service
metadata:
  name: web-app
  namespace: platform
  labels:
    app: web-app
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-type: "external"
    service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: "instance"
    service.beta.kubernetes.io/aws-load-balancer-scheme: "internal"
    service.beta.kubernetes.io/aws-load-balancer-additional-resource-tags: 'Stage=prod,App=web-app'
    service.beta.kubernetes.io/aws-load-balancer-attributes: access_logs.s3.enabled=true,access_logs.s3.bucket=prod-bucket,access_logs.s3.prefix=loadbalancing/web-app,load_balancing.cross_zone.enabled=true
spec:
  ports:
    - name: https
      protocol: TCP
      port: 443
      targetPort: 443
  selector:
    app: web-app
  type: LoadBalancer
  loadBalancerSourceRanges:
    - 10.1.0.0/16

Above file results in below configuration:

1. Uses AWS Load Balancer Controller instead of the in-tree kubernetes controller

2. Provisions an internal NLB with instance mode

3. Sets the tags provided

4. Enables access logs to the provided S3 path

5. Enables Cross Zone load balancing

Traffic Flow

Instance Mode

When a request is sent, it reaches the NLB which will load balance the traffic to the target backends (worker nodes).

In instance mode, the traffic gets sent to the NodePort of the instance resulting in an additional hop.

After which the kube-proxy running in the node, sends the request to the desired pod.

EKS by default runs in iptables proxy mode, which means that the kube-proxy will make use of iptables to route traffic to the pods. iptables mode chooses a backend at random.

Another factor is externalTrafficPolicy which is by default set to Cluster mode. Here, the traffic may randomly be routed to a pod on another host to ensure equal distribution.

Integration Patterns

API Gateway

Classic Load Balancer

How To Provision

To provision a Classic Load Balancer, you need to create a service of type LoadBalancer.

apiVersion: v1
kind: Service
metadata:
  name: web-app
  namespace: platform
  labels:
    app: web-app
spec:
  ports:
    - name: https
      protocol: TCP
      port: 443
      targetPort: 443
  selector:
    app: web-app
type: LoadBalancer

Following actions are performed by the In-Tree Legacy Cloud Provider in Kubernetes, when it sees a service object of type LoadBalancer:

1. ELB is created in AWS for the service. Depending on the annotations it’s either internal or external.

2. Listeners are created for each port detailed in the service definition.

3. Health checks are configured.

Subnet Discovery

In-Tree Legacy Cloud Provider auto discovers network subnets by default.

To be able to successfully do that you need to tag your subnets as follows:

Annotations

Let’s look at some of the annotations that you can configure and their behaviors.

Load Balancer Security Group

You can configure the security groups to be attached to an ELB for restricting traffic.

Complete service file with the annotations will look as follows:

apiVersion: v1
kind: Service
metadata:
  name: web-app
  namespace: platform
  labels:
    app: web-app
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-access-log-emit-interval: '60'
    service.beta.kubernetes.io/aws-load-balancer-access-log-enabled: 'true'
    service.beta.kubernetes.io/aws-load-balancer-access-log-s3-bucket-name: prod-bucket
    service.beta.kubernetes.io/aws-load-balancer-access-log-s3-bucket-prefix: loadbalancing/web-app
    service.beta.kubernetes.io/aws-load-balancer-additional-resource-tags: Stage=prod,App=web-app
    service.beta.kubernetes.io/aws-load-balancer-connection-draining-enabled: 'true'
    service.beta.kubernetes.io/aws-load-balancer-connection-draining-timeout: '300'
    service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: 'true'
    service.beta.kubernetes.io/aws-load-balancer-extra-security-groups: sg-53fae93f
spec:
  ports:
    - name: https
      protocol: TCP
      port: 443
      targetPort: 443
  selector:
    app: web-app
  type: LoadBalancer
  loadBalancerSourceRanges:
    - 10.1.0.0/16

Above file results in below configuration:

1. Provisions an ELB in the desired subnet

2. Sets the tags provided

3. Enables access logs to the provided S3 path

4. Enables Cross Zone load balancing

5. Adds .spec.loadBalancerSourceRanges to the uniquely generated security group for this ELB

6. Adds sg-53fae93f as an additional security group to the ELB

7. Adds a matching permit line to allow traffic from the ELB to the target worker nodes

References

https://kubernetes.io/docs/concepts/services-networking/service/

https://kubernetes-sigs.github.io/aws-load-balancer-controller/

https://docs.aws.amazon.com/elasticloadbalancing/latest/network/load-balancer-target-groups.html#client-ip-preservation

https://docs.aws.amazon.com/elasticloadbalancing/latest/network/target-group-register-targets.html#target-security-groups

https://kubernetes.io/docs/concepts/services-networking/_print/#pg-374e5c954990aec58a0797adc70a5039

https://github.com/kubernetes/legacy-cloud-providers/blob/master/aws/aws.go