Efficient Kubernetes Autoscaling With Karpenter and KEDA: A Comprehensive Guide
In modern cloud environments, efficient resource management is essential to maintain performance while minimizing costs. This documentation provides a step-by-step guide to configuring pods and cluster autoscaling in a Kubernetes environment using Karpenter and KEDA.
Karpenter dynamically provisions and removes nodes based on real-time demand, ensuring that the cluster has just the right amount of capacity. KEDA, on the other hand, scales workloads by adjusting pod replicas based on key metrics, such as pipeline activity and user requests.
By implementing this approach, the cluster remains responsive to workload fluctuations, preventing over-provisioning and reducing unnecessary resource usage. This results in a cost-effective, scalable, and efficient infrastructure that adapts to real-time demands without manual intervention.
Prerequisitesβ
Before setting up AutoScaling, ensure the following requirements are met:
- Prometheus is installed and exporting cluster metrics.
- Nginx Ingress is installed and configured to export its metrics to Prometheus.
Karpenterβ
The configuration and installation of Karpenter involves several steps, including:
- Setting up resources in the AWS provider β configuring IAM roles, permissions, and networking.
- Installing the Karpenter Helm chart β deploying Karpenter controller in the Kubernetes cluster.
- Configuring essential components β setting up
Node Pools
andNode Class
, and integrating with cluster resources.
AWS Configurationβ
To ensure Karpenter can properly manage node provisioning, it is essential to configure AWS resources correctly. This includes:
- Creating IAM roles and policies β Karpenter requires specific permissions to provision and manage EC2 instances.
- Tagging VPC Subnets and Security Groups β Karpenter uses these tags to determine which network subnets and security groups to apply when creating new nodes.
Karpenter does not use Auto Scaling Groups (ASG) when creating new nodes. Instead, it provisions EC2 instances directly and registers them with the cluster.
To prepare your AWS cluster for Karpenter, follow the steps below:
-
Configure IRSA:
To allow Karpenter to interact with AWS services securely, configure IAM Roles for Service Accounts (IRSA). This configuration can be done automatically during cluster installation using a terraform-aws-platform template or manually after the cluster is deployed.
-
Configure Network and Security Groups:
Ensure that the required tags are added to VPC
subnets
andsecurity group
so Karpenter can use them for provisioning new nodes.module "vpc" {
source = "terraform-aws-modules/vpc/aws"
version = "5.1.2"
private_subnet_tags = merge(var.tags, tomap({ "karpenter.sh/discovery" = "${var.cluster_name}" }))
default_security_group_tags = merge(var.tags, tomap({ "karpenter.sh/discovery" = "${var.cluster_name}" }))
tags = var.tags
}infoThis method will add tags to all private subnets created during cluster initialization. If your cluster operates in a single-subnet zone, you must manually add the required tags to the appropriate subnets, as Terraform does not support tagging individual subnets.
Install Karpenterβ
Install and configure Karpenter using the add-ons approach or manually. Specify the controller role that was created in AWS, and configure tolerations and nodeSelector if necessary:
karpenter:
# tolerations:
# - key: "type"
# operator: "Equal"
# value: "system"
# effect: "NoSchedule"
# nodeSelector:
# type: system
# -- Karpenter IAM role to manage cluster nodes
serviceAccount:
annotations:
eks.amazonaws.com/role-arn: arn:aws:iam::0123456789:role/KarpenterControllerRole-eks
Since Karpenter is installed in a separate namespace, you need to update the configuration of its CRD used for Webhook Validation resources:
Node Pools:
kubectl patch crd nodepools.karpenter.sh --type=merge -p '{
"spec": {
"conversion": {
"webhook": {
"clientConfig": {
"service": {
"namespace": "karpenter"
}
}
}
}
}
}'
Node Class:
kubectl patch crd ec2nodeclasses.karpenter.k8s.aws --type=merge -p '{
"spec": {
"conversion": {
"webhook": {
"clientConfig": {
"service": {
"namespace": "karpenter"
}
}
}
}
}
}'