Scale resources using Custom Autoscaling

Custom Autoscaling is a Preview feature.

A Preview feature:

Has not undergone end-to-end testing with CloudBees products.
Is provided without service-level agreements (SLA), and therefore does not include CloudBees' commitment to functionality or performance.
May impact other stable areas of the product when used.
May have limited documentation.
May not be feature-complete during the Preview period.
May graduate from Preview to a supported feature or be removed from the product.
May introduce breaking changes that prevent upgrades due to incompatibility with future development.

Product features and documentation are frequently updated. If you find an issue or have a suggestion, please contact CloudBees Support.

Pipelines and procedures typically rely on resource pools, either assigned directly to them or inherited from the project, to execute jobs. When a job step runs in one of these objects, if there are no available build resources (agents) in the resource pool, the job must wait until one becomes available. This can greatly increase the time needed for jobs to start (and complete) during times of high demand, such as when running multiple releases.

This page describes how to scale up and down resources using Custom Autoscaling resource pools. This approach provides an automatic method to scale your environment based on your system demands using the CloudBees CD/RO bound agent to create and remove resource pools and resources (agent replicas) within your Kubernetes cluster.

CloudBees CD/RO v2025.06.0 or later is required to configure Custom Autoscaling resources. If you attempt the following steps in v2025.03.1 or earlier, your Helm installation will fail.

Before you start

This section provides important information you need to understand configuring and running Custom Autoscaling resources.

Perform installation or upgrade first

Before updating the configurations for Custom Autoscaling, CloudBees recommends installing or upgrading to v2025.06.0 or later following standard practices. This helps prevent Custom Autoscaling configuration changes from interfering with the installation or upgrade and makes troubleshooting easier.

For more information, refer to:

Custom Autoscaling overview

When configuring Custom Autoscaling, the CloudBees CD/RO bound agent is assigned a service account with the required RBAC permissions to create resource pools as StatefulSets and automatically scale cloudbees-flow-agent replicas within them. This process is repeated for each agent configured with Custom Autoscaling and assigned to the CloudBees CD/RO instance. CloudBees CD/RO then registers the resource pools and their resources (agent replicas) for job execution.

Because the CloudBees CD/RO bound agent assigned to the server handles agent scaling, Custom Autoscaling agents must be deployed in the same Kubernetes namespace as their assigned server. For more information refer to Configure Custom Autoscaling in Helm charts.

In CloudBees CD/RO, pipelines and procedures can target these Custom Autoscaling resource pools for job execution. If jobs are queued due to a lack of available resources (agents), CloudBees CD/RO automatically initiates a scale-up procedure, which incrementally increases resources until the system load reduces or the defined maximum resource limit is reached. Similarly, when idle agents are detected (agents inactive for a set duration), CloudBees CD/RO triggers a scale-down procedure, reducing the pool size by one per run until the defined minimum is reached.

When scaling down resources, CloudBees CD/RO scales down one resource per run based on the idle resources check interval configured in the agent settings.

For instance, if there are 4 idle resources, with the minimum resource pool size defined as 1, then three agents need to be scaled down to leave one available in the pool. If you have the idle check interval set for 20 minutes, it takes 60 minutes for the three idle agents to be scaled down.

Configuring Custom Autoscaling involves the following steps, which should be performed in order:

Upgrade to v2025.06.0 or later.
Configure Custom Autoscaling in CloudBees CD/RO.
Configure Custom Autoscaling in Helm charts.
Configure Custom Autoscaling resource pools.
(Optional as needed) Scale down resources.

Custom Autoscaling can only be performed using the methods described on this page. There are several UI and API restrictions in place for Custom Autoscaling. For more information, refer to Custom Autoscaling restrictions.

Additionally, scaling up resources is meant to only be performed by the bound agent calling a specific procedure. Although it is technically possible to manually run this procedure, CloudBees strongly advises against do so. For more information, refer to Do not manually scale up resources.

Do not manually scale up resources

By design, only the CloudBees CD/RO bound agent should perform resource Custom Autoscaling. When resources are scheduled to be scaled up, the bound agent runs the EC-Utilities ScaleUpElasticResourcePool procedure, which registers the resources with the scheduler to check for and remove idle resources.

Although it is technically possible to manually run the ScaleUpElasticResourcePool procedure, CloudBees strongly advises not to do so. This is because, when manually running the ScaleUpElasticResourcePool procedure:

There are no inherent limits on the number of resources that may be created by a user with adequate permissions.
Resources are not registered with the scheduler, and therefore are not scaled down when they are idle.

Due to these factors, if the ScaleUpElasticResourcePool procedure is manually run, situations may arise where excessive amounts of environmental/system resources are consumed, resulting in any of the following:

Resource starvation across the cluster.
Node pressure leading to pod eviction.
Out-of-memory (OOM) kills and CPU throttling.
Degraded control plane performance.
Cluster instability or crash.
Difficulty recovering from overload or unresponsive state.

For cloud-hosted environments, additional risks include:

Spiraling cloud infrastructure costs.
Cloud quota exhaustion or throttling, resulting in failed resource creation or provisioning delays.
Sudden operational instability.

For these reasons, CloudBees strongly advises not to manually run the ScaleUpElasticResourcePool procedure. Instead, allow the CloudBees CD/RO bound agent to automatically create resources based on your system demand.

Custom Autoscaling restrictions

Custom Autoscaling resource pools and resources have restrictions in place to help ensure they are only created, modified, or removed in a controlled manner. For instance:

Custom Autoscaling resource pools can only be created by the CloudBees CD/RO bound agent via Kubernetes operations based on your Helm chart configurations.
- Custom Autoscaling resource pools can not be created or deleted in the UI (DevOps essentials Resource pools) or via APIs (createResourcePool or deleteResourcePool).
Once a Custom Autoscaling resource pool is created, its details are immutable, and its name nor type can be changed.
- Custom Autoscaling resource pools can not be modified in the UI (via Resource pools Edit Resource Pool) or via APIs (modifyResourcePool).
Custom Autoscaling resources (agents) are only created in or deleted from Custom Autoscaling resource pools based on Helm chart settings and automated processes.
- Resources can not be created or deleted in the UI (DevOps essentials Resources) or via APIs (createResource or deleteResource).
Custom Autoscaling resources (agents) can only be added to or removed from Custom Autoscaling resource pools based on Helm chart settings and automated processes.
- Resources can not be added or removed in the UI (DevOps essentials Resources) or via APIs (addResourcesToPool or removeResourcesFromPool).

To enforce this behavior, the corresponding APIs prevent unintended usage. If you attempt to use these APIs to create, delete, or modify Custom Autoscaling resource pools, or resources within them, you will receive an error message, and the operation fails. The following table provides a list of restricted APIs, the error message you will receive when attempting them and an explanation of how to carry out the operation:

Autoscaling API restrictions

API Error code Explanation

API	Error code	Explanation
`createResource`	`CustomAutoScalingResourceCreationNotAllowed`	You cannot add resources to a Custom Autoscaling resource pool using the `createResource` API. Additional resources can only be added based on the agent values file configuration, or the Custom Autoscaling resource pool configuration. For more information, refer to: Configure Custom Autoscaling in Helm charts Configure Custom Autoscaling resource pools
`deleteResource`	`CustomAutoScalingResourceDeletionNotAllowed`	You cannot delete resources from a Custom Autoscaling resource pool using the `deleteResource` API. Resources are either automatically deleted based on your Agent settings, or can be manually scaled down using the ScaleDownElasticResourcePool procedure. For more information, refer to Scale down resources.
`createResourcePool`	`CustomAutoScalingPoolCreationNotAllowed`	You cannot create a Custom Autoscaling resource pool using the `createResourcePool` API. An Autoscaling resource pool is created, based on your Helm chart values, when running the ScaleUpElasticResourcePool. For more information, refer to Configure Custom Autoscaling in Helm charts.
`deleteResourcePool`	`CustomAutoScalingPoolDeletionNotAllowed`	You cannot delete a Custom Autoscaling resource pool using the `deleteResourcePool` API. To delete the Custom Autoscaling resource pool, you must update its values file with `customAutoScaling.enabled=false` and `helm` uninstall and reinstall the agent. For more information, refer to Configure Custom Autoscaling in Helm charts.
`modifyResourcePool`	`CustomAutoScalingPoolModificationNotAllowed`	Custom Autoscaling resource pools' type or name cannot be modified. To edit the name of a Custom Autoscaling resource pool, which is based on the CloudBees CD/RO agent Kubernetes release name, you must update the actual CloudBees CD/RO agent Kubernetes release name by uninstalling and reinstalling the agent Helm chart.
`removeResourcesFromPool`	`CustomAutoScalingPoolRemoveResourcesNotAllowed`	Custom Autoscaling resources cannot be removed using this method. To remove resources from a resource pool, you can either reconfigure the resource pool’s Custom Autoscaling Idle size configuration, or run the manual resource scale-down procedure. For more information, refer to: Configure Custom Autoscaling resource pools Scale down resources.
`addResourcesToPool`	`CustomAutoScalingPoolAddResourcesNotAllowed`	Custom Autoscaling resources cannot be added using this method. However, you can reconfigure the resource pool’s Custom Autoscaling Maximum size configuration. For more information, refer to Configure Custom Autoscaling resource pools.

createResource

CustomAutoScalingResourceCreationNotAllowed

You cannot add resources to a Custom Autoscaling resource pool using the createResource API. Additional resources can only be added based on the agent values file configuration, or the Custom Autoscaling resource pool configuration.

For more information, refer to:

Configure Custom Autoscaling in Helm charts
Configure Custom Autoscaling resource pools

deleteResource

CustomAutoScalingResourceDeletionNotAllowed

You cannot delete resources from a Custom Autoscaling resource pool using the deleteResource API. Resources are either automatically deleted based on your Agent settings, or can be manually scaled down using the ScaleDownElasticResourcePool procedure.

For more information, refer to Scale down resources.

createResourcePool

CustomAutoScalingPoolCreationNotAllowed

You cannot create a Custom Autoscaling resource pool using the createResourcePool API. An Autoscaling resource pool is created, based on your Helm chart values, when running the ScaleUpElasticResourcePool.

For more information, refer to Configure Custom Autoscaling in Helm charts.

deleteResourcePool

CustomAutoScalingPoolDeletionNotAllowed

You cannot delete a Custom Autoscaling resource pool using the deleteResourcePool API. To delete the Custom Autoscaling resource pool, you must update its values file with customAutoScaling.enabled=false and helm uninstall and reinstall the agent.

For more information, refer to Configure Custom Autoscaling in Helm charts.

modifyResourcePool

CustomAutoScalingPoolModificationNotAllowed

Custom Autoscaling resource pools' type or name cannot be modified. To edit the name of a Custom Autoscaling resource pool, which is based on the CloudBees CD/RO agent Kubernetes release name, you must update the actual CloudBees CD/RO agent Kubernetes release name by uninstalling and reinstalling the agent Helm chart.

removeResourcesFromPool

CustomAutoScalingPoolRemoveResourcesNotAllowed

Custom Autoscaling resources cannot be removed using this method. To remove resources from a resource pool, you can either reconfigure the resource pool’s Custom Autoscaling Idle size configuration, or run the manual resource scale-down procedure. For more information, refer to:

Configure Custom Autoscaling resource pools
Scale down resources.

addResourcesToPool

CustomAutoScalingPoolAddResourcesNotAllowed

Custom Autoscaling resources cannot be added using this method. However, you can reconfigure the resource pool’s Custom Autoscaling Maximum size configuration. For more information, refer to Configure Custom Autoscaling resource pools.

Configure Custom Autoscaling in CloudBees CD/RO

After you have updated your CloudBees CD/RO deployment to v2025.06.0 or later, the first step for Custom Autoscaling is to configure the agent settings in CloudBees CD/RO. These settings can be found on Administration Server settings Agent settings.

To configure the Custom Autoscaling agent settings:

Log in to your CloudBees CD/RO instance.
Navigate to Administration Server settings Agent settings.
Configure the following settings:
- For Enable Custom Autoscaling resource pools, select Enabled.
- For Interval to check for idle Custom Autoscaling resources (minutes), specify how frequent in minutes CloudBees CD/RO should check for idle resources.
  
  Expand idle interval check explanation
  
  To automatically scale down idle resources, CloudBees CD/RO runs a check on Custom Autoscaling resource pools based on the value in Interval to check for idle Custom Autoscaling resources to identify idle resources. When CloudBees CD/RO discovers idle resources, it automatically attempts to scale down the resources (one per run) to the minimum configured value according to cloudbees-flow-agent.customAutoscaling.minReplicas.
  
  For instance, if you have Interval to check for idle Custom Autoscaling resources set to 20, idle resources are checked for every 20 minutes. If any are identified in a resource pool, a scale down procedure runs automatically to scale down resources in the affected pool by 1. This keeps occurring every 20 minutes until no idle resources are detected or the minimum configured resources are reached.
  
  For instance, if you have 1 configured as the minimum number of resources to keep in the pool, and the check detects there are 4 idle resources, then three resources need to be scaled down, leaving one available. So when the check runs, it scales down one resource per run every 20 minutes until it reaches the minimum, which takes 60 minutes total (3 * 20).
  
  Alternatively, if you want to scale all resources down to the minimum immediately, you can manually scale down using the scale down procedure.

Configure Custom Autoscaling in Helm charts

Now that you have configured the agent settings in CloudBees CD/RO, the next step is to configure your values files to support Custom Autoscaling. This is a two-step process:

Configure bound agent permissions.
Configure agent Custom Autoscaling values.

Configure bound agent permissions

To allow the bound agent to create Custom Autoscaling resources:

Open your v2025.6.0 or later cloudbees-flow values file.
Search for the Flow bound agent configuration section.

Navigate to boundAgent.rbac, and update the following:

rbac:
  create: true (1)▼

create: true - Generates the necessary Kubernetes Role and RoleBinding resources. All permissions required for the bound agent are configured by default.

Changing the default values for boundAgent.rbac.roles.rules, may result in the bound agent not having the correct permission to perform autoscaling tasks, and may result in unexpected behavior as the bound agent tries to scale up and down resources.

Ensure the values file is properly formatted and save your values file.
Run your helm install or helm upgrade command using the updated cloudbees-flow values file.

Configure agent Custom Autoscaling values

To configure an agent as a Custom Autoscaling resource:

If you want to configure multiple resource pools as Custom Autoscaling, follow these instructions for each agent.

Open the cloudbees-flow-agent values file (v2025.6.0 or later).

Add the following Custom Autoscaling configuration:

During Helm install or upgrade, a Custom Autoscaling resource pool is created only if it doesn’t already exist. Once created, the pool retains its initial customAutoScaling settings (min, max, idleTimeout). Updating these values in the Helm chart later will not modify the existing pool.

To change an existing resource pool, refer to Configure Custom Autoscaling resource pools or use ectool modifyResourcePool. Future Helm upgrades do not affect resource pool settings. Only stepLimit continues to apply during agent bootstrap, as it controls agent behavior—not pool configuration.

customAutoScaling:
  enabled: true (1)
  minReplicas: 0 (2)
  maxReplicas: 3 (3)
  inactivityTimeoutSeconds: 600 (4)
  ## `stepLimit` is the number of steps that can be run concurrently on a single resource.`
  stepLimit: 20 (5)▼

Expand values explanation:

1 enabled: true - Enables Custom Autoscaling for the resource pool.

2 minReplicas: 1 - Defines the minimum number of active resources.

3 maxReplicas: 3 - Defines the maximum number of resources allowed.

4 inactivityTimeoutSeconds: 600 - Sets the time (in seconds) before an idle resource is scaled down.

stepLimit - Defines the number of steps that can be run concurrently on a single resource.

The stepLimit significantly affects how often resources are created and removed. A high value allows more steps to run on existing resources in the pool, reducing the need to provision new ones. However, this can impact performance if resources become overloaded. Conversely, a lower stepLimit may trigger the creation of additional resources more frequently, which can cause the system to reach the maxReplicas limit faster, and lead to degraded performance.

Ensure the values file is properly formatted and save your values file.

Run your helm install or helm upgrade command using the updated cloudbees-flow-agent values file.

When deploying your agents, you must deploy them to the same Kubernetes namespace as the server they are assigned to.

Failure to do so will result in Custom Autoscaling not working as expected, because the CloudBees CD/RO bound agent can only create and remove resources in the same namespace where it is deployed.

After the agent is up and running, to confirm your Custom Autoscaling resource pools were created, navigate to DevOps essentials Resource pools. There should be a resource pool of type Custom Autoscaling for each agent you deployed in the namespace with customAutoScaling enabled.

To configure these resource pools, refer to Configure Custom Autoscaling resource pools.

Configure Custom Autoscaling resource pools

Once you have your Agent settings configured, you can now update the Custom Autoscaling configuration for resource pools.

The resource pool name cannot be changed in the UI. Because it is autogenerated from CloudBees CD/RO agent deployment name, to change the name, you must change the agent deployment name.

To update the Custom Autoscaling configuration for a resource pool (Type Custom Autoscaling):

Navigate to DevOps essentials Resource pools.
Select the desired Custom Autoscaling resource pool. The Edit Resource Pool window opens.

In the Custom Autoscaling panel, update the desired configurations for:

Idle size: The minimum number of resources to leave present in the resource pool. This value must either be 0 or a positive integer.
Maximum size: Specify the maximum number of resources to allow for the resource pool. This value must be a positive integer.

Allowing a significantly high number of resources to be created may greatly increase resource consumption in your environment.

Inactivity timeout: Specify the time (in seconds) after which an idle resource is removed due to inactivity.

This is only the duration of inactivity in which a resource is designated as idle. It may not be removed until the scale down job runs on the configured interval for Agent settings Interval to check for idle Custom Autoscaling resources.

For more information, refer to Configure Custom Autoscaling in CloudBees CD/RO.

Select OK to save the changes.

Your Custom Autoscaling resource pool is now updated with the new configuration.

Scale down resources

If Custom Autoscaling does not scale down resources as expected, you can manually scale down the resource pool using the ScaleDownElasticResourcePool procedure:

Navigate to DevOps essentials Procedures.
In the search field, enter ScaleDownElasticResourcePool.
Select New run.
Fill the following fields:
- poolName: Provide the resource pool name to scale down.
  
  This resource pool must be of type Custom Autoscaling.
- resourcePoolIdleSize: Enter the minimum number of resources that should remain in the pool, even if idle. This setting is similar to the customAutoScaling.minReplicas in the agent values file.
- resourceScaleDownCount: Enter the number of resources you want to remove from resource pool.
(Optional) If needed, select the Force checkbox:
- If selected, the resource pool is scaled down to the specified resourcePoolIdleSize, regardless of the resourceScaleDownCount.
- If left unselected, only the number of resources specified in resourceScaleDownCount are removed, or until resourcePoolIdleSize is reached.
Select OK to start the procedure.

The procedure runs and scales the specified resource pool down to the configured number of resources.