Auto-scaling nodes on EKS

2 minute read

For more information about auto-scaling managed nodes on EKS, refer to Amazon EKS managed node groups.

You can install the Kubernetes cluster autoscaler to auto-scale nodes.

Auto-scaling considerations for EKS

Managed controller and operations center workload

By assigning managed controller and operations center workload to a dedicated pool, the scaling down of nodes can be prevented by restricting eviction of managed controller or operations center deployments. Scale up will happen normally when resources need to be increased in order to deploy additional managed controllers, but scale down will only happen when the nodes are free of operations center or managed controller workload. This might be acceptable since controllers are meant to be stable and permanent, meaning that they are not ephemeral but long lived.

This is achieved by adding the following annotation to operations center and managed controllers: "cluster-autoscaler.kubernetes.io/safe-to-evict": "false"

For operations center, the annotation is added to the cloudbees-core.yml in the CJOC "StatefulSet" definition under "spec - template - metadata - annotations"

apiVersion: "apps/v1beta1" kind: "StatefulSet" metadata: name: cjoc labels: com.cloudbees.cje.type: cjoc com.cloudbees.cje.tenant: cjoc spec: serviceName: cjoc replicas: 1 updateStrategy: type: RollingUpdate template: metadata: annotations: cluster-autoscaler.kubernetes.io/safe-to-evict: "false"

For managed controller, the annotation is added in the configuration page under the 'Advanced Configuration - YAML' parameter. The YAML snippet to add would look like:

apiVersion: apps/v1 kind: StatefulSet spec: template: metadata: annotations: cluster-autoscaler.kubernetes.io/safe-to-evict: "false"

Agent workload

For Pipelines, 'toleration' can be added to podTemplate using the yaml parameter as follows:

podTemplate(yaml: ''' apiVersion: v1 kind: Pod spec: tolerations: - key: nodeType operator: Equal value: build effect: NoSchedule ''', nodeSelector: 'workload=build', containers: [ containerTemplate(name: 'maven', image: 'maven:3.8.5-jdk-11', command: 'sleep', args: '99d') ]) { node(POD_LABEL) { stage('Run maven') { container('maven') { sh 'mvn --version' } } } }

IAM policy

The worker running the cluster autoscaler needs access to certain resources and actions. To create an IAM policy, refer to Create an IAM policy and role.

Install cluster autoscaler

To install the cluster autoscaler, refer to Deploy the Cluster Autoscaler.

In August 2020, the Jenkins project voted to replace the term master with controller. We have taken a pragmatic approach to cleaning these up, ensuring the least amount of downstream impact as possible. CloudBees is committed to ensuring a culture and environment of inclusiveness and acceptance - this includes ensuring the changes are not just cosmetic ones, but pervasive. As this change happens, please note that the term master has been replaced through the latest versions of the CloudBees documentation with controller (as in managed controller, client controller, team controller) except when still used in the UI or in code.