Pipelines and procedures must specify a resource pool for execution. When you run a job for one of these objects, if there are no available build resources (agents) in the resource pool, the job must wait until one becomes available. This can greatly increase the time needed for jobs to start (and complete) during times of high demand, such as when running multiple releases.
This page describes how to scale up and down build resources using procedures. This approach provides an efficient method to scale your environment based on job demands by using the CloudBees CD/RO bound agent to create and remove agent replicas within your Kubernetes cluster.
The processes and procedures described on this page are only applicable for Kubernetes environments. If you attempt them in a non-Kubernetes environment, these procedures fail. |
These processes are used to scale resources using procedures:
Configure bound agent permissions
To configure the boundAgent
permission to scale agents in the cloudbees-flow
values file:
-
Open your v2024.12.0 or later
cloudbees-flow
values file. -
Search for the
Flow bound agent configuration section
. -
Scroll down to
boundAgent.rbac
.To include this configuration in your current
boundAgent
Helm chart, add the following:View the
boundAgent.rbac
configuration:## The following configuration enables Role-Based Access Control (RBAC) for managing ## bound agents with support for ephemeral scaling. Enable `rbac.create=true` ## to create the required service account. rbac: create: false role: ## RBAC rules to create. rules: - apiGroups: [ "apps" ] resources: [ "statefulsets/scale" ] verbs: [ "update", "patch", "get" ]
-
Change
boundAgent.rbac.create
fromfalse
totrue
. -
Save your changes.
-
Run your Helm
install
command for thecloudbees-flow
values file to update the cluster.
You have configured your Helm values to allow the bound agent to create additional agents. The next step is to create the agent resource source pool, which is described in Add agent resource pool.
Add agent resource pool
After configuring your Helm charts, you must include the --set resourcePools
parameter in your CloudBees CD/RO agent Helm install
command. This parameter specifies the resource pool to associate with the flow-agent
installation. The value of resourcePools
must follow the format <release-name>-flow-agent
.
For example, to install a flow-agent
with the release name flow-agent-test
, the command would be similar to:
helm install flow-agent-test cloudbees/cloudbees-flow-agent \ -f values-agent.yaml \ --namespace flow \ --set resourcePools="flow-agent-test-flow-agent"
You have configured your Helm values and created the agent resource pool. Once the CloudBees CD/RO agent is installed and initialized, you can move on to Create the manual scaling procedure.
Create the manual scaling procedure
This section describes creating a procedure that can be used to both scale up and scale down build resources (agents) as required.
To create the procedure:
-
In CloudBees CD/RO, select
. -
On the Procedures page, select Add procedure.
-
Select Create New….
-
For the new procedure, enter the following:
-
Name:
ManualResourcePoolScaling
-
Project:
CloudBees
The DSL procedure included below creates the procedure in the
CloudBees
project. To use it as is, you must have permission to create the procedure there. However, it is not required to create the procedure inCloudBees
. If you use a different project, ensure you updateprojectName
in the DSL to the project you use.
-
-
Select OK.
-
In the
ManualResourcePoolScaling
procedure screen, select DSL editor. -
In the DSL editor window, paste the following:
If you used a different project than
CloudBees
, ensure you updateprojectName
in the following DSL.ManualResourcePoolScaling
procedure DSLprocedure 'ManualResourcePoolScaling', { description = '''Scale build resources (agents) in a specified resource pool. ''' jobNameTemplate = '' projectName = 'CloudBees' resourceName = 'local' timeLimit = '0' timeLimitUnits = 'minutes' workspaceName = '' formalParameter 'RESOURCE_POOL', defaultValue: '', { description = '''Provide the name of the resource pool to scale. <b>IMPORTANT:</b> This resource pool must already exist. If it does not, the procedure fails. ''' expansionDeferred = '0' label = 'Resource pool name' orderIndex = '1' renderCondition = null required = '1' type = 'entry' } formalParameter 'MAX_SIZE', defaultValue: '', { description = '''Specify the maximum number of resources to allow for the resource pool. This value <b>must</b> be a positive integer. <b>IMPORTANT:</b> Allowing a significantly high number of resources to be created may greatly increase resource consumption in your environment. ''' expansionDeferred = '0' label = 'Maximum number of resources' orderIndex = '2' renderCondition = null required = '1' type = 'entry' } formalParameter 'DESIRED_SIZE', defaultValue: '', { description = '''Specify the desired number of resources for the resource pool. This value <b>must</b> be: • A positive integer. • Not equal to the current resource count. • Less than or equal to <b>Maximum number of resources</b>. • Greater than or equal to <b>Minimum number of resources</b>. <b>IMPORTANT</b>: • If the number of resources in the pool is <b><u>less</u></b> than <b>Desired number of resources</b>, new resources are created until the resource count reaches <b>Desired number of resources</b> or <b>Maximum number of resources</b>, whichever is less. • If the number of resources in the pool is <b><u>more</u></b> than <b>Desired number of resources</b>, resources are removed until the pool reaches <b>Desired number of resources</b> or <b>Minimum number of resources</b>, whichever is greater. ''' expansionDeferred = '0' label = 'Desired number of resources' orderIndex = '3' renderCondition = null required = '1' type = 'entry' } formalParameter 'MIN_SIZE', defaultValue: '', { description = 'The minimum number of resources to leave present in the resource pool. This value <b>must</b> be a positive integer.' expansionDeferred = '0' label = 'Minimum number of resources' orderIndex = '4' renderCondition = null required = '1' type = 'entry' } step 'scale', { description = 'Scales the number of resources within a resource pool.' alwaysRun = '0' broadcast = '0' command = '''APISERVER=https://kubernetes.default.svc TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token) NAMESPACE=$(cat /var/run/secrets/kubernetes.io/serviceaccount/namespace) REPLICA_COUNT=$(curl -k -X GET $APISERVER/apis/apps/v1/namespaces/$NAMESPACE/statefulsets/$[RESOURCE_POOL]/scale -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" | grep -o \'"replicas": [0-9]*\' | head -n 1 | grep -o \'[0-9]*\') if [ "$[DESIRED_SIZE]" -gt "$[MAX_SIZE]" ]; then echo "Desired size ($[DESIRED_SIZE]) is greater than the maximum allowed size ($[MAX_SIZE]). No new resources were created." exit 1 elif [ "$[DESIRED_SIZE]" -lt "$[MIN_SIZE]" ]; then echo "Desired size ($[DESIRED_SIZE]) is less than the minimum allowed size ($[MIN_SIZE]). No resources were removed." exit 1 elif [ "$[DESIRED_SIZE]" -eq "$REPLICA_COUNT" ]; then echo "Desired size ($[DESIRED_SIZE]) is equal to the current replica count ($REPLICA_COUNT). No resources were created or removed." exit 1 else # Set replica count to the desired size REPLICA_COUNT=$[DESIRED_SIZE] # Update the replica count with PATCH response=$(curl -k -s -o /dev/null -w "%{http_code}" -X PATCH "$APISERVER/apis/apps/v1/namespaces/$NAMESPACE/statefulsets/$[RESOURCE_POOL]/scale" -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/merge-patch+json" -d "{\\"spec\\": {\\"replicas\\": $REPLICA_COUNT}}") # Check the HTTP response code if [ "$response" -ne 200 ]; then echo "Error: Failed to scale the statefulset. HTTP response code: $response" exit 1 fi echo "Resource in Resource Pool updated to $REPLICA_COUNT." fi ''' condition = null errorHandling = 'failProcedure' exclusiveMode = 'none' logFileName = '' parallel = '0' postProcessor = '' precondition = null procedureName = 'ManualResourcePoolScaling' releaseMode = 'none' resourceName = '' shell = 'bash' subprocedure = null subproject = null timeLimit = '0' timeLimitUnits = 'seconds' workingDirectory = '' workspaceName = '' } }
-
Select the Save icon.
When you run this procedure:
Procedure explanation
The following explanation uses the names of the DSL formal parameters. You can open the procedure by selecting New run on the procedure, and compare the UI with the explanation. The following list is the corresponding UI field labels for these parameters:
|
-
It checks the current number of resources (
REPLICA_COUNT
) in the resource pool (RESOURCE_POOL
). -
It then checks
REPLICA_COUNT
against the desired size you are requesting (DESIRED_SIZE
).-
If
REPLICA_COUNT
equalsDESIRED_SIZE
, the procedure fails. -
If
DESIRED_SIZE
is higher thanREPLICA_COUNT
, it checks the requested size against the maximum number of allowed resources (MAX_SIZE
).If DESIRED_SIZE
is more than theMAX_SIZE
of allowed resources, the procedure fails. -
If
DESIRED_SIZE
is less thanREPLICA_COUNT
, it checks the requested size against the minimum number of allowed agents (MIN_SIZE
).If DESIRED_SIZE
is less than theMIN_SIZE
of allowed resources, the procedure fails.
-
-
The procedure then scales the resources up or down based on whether
DESIRED_SIZE
is more or less thanREPLICA_COUNT
.
Scale build resources manually using a procedure
This section describes scaling resources by running the ManualResourcePoolScaling
procedure created in Create the manual scaling procedure.
-
Check the number of resources currently in the resource pool:
-
Navigate to
. -
In the Find… search box, enter the name of the resource pool you want to scale.
-
Check the number of current build resources in the Resources column.
When creating resources, if you do not scale the resource count back down, they remain active in your environment, and may affect your overall system performance.
When you run the
ManualResourcePoolScaling
to scale down resources, if a resource is nominated for removal that is currently running a job, it is removed regardless of the status of the running job. The job is then paused until it can be reassigned to another available resource.When scaling resources down, CloudBees recommends taking note of the number of resources in the pool that are currently idle and removing this number of resources to help avoid long job reassignment times.
-
-
Navigate to
, and search forManualResourcePoolScaling
. -
Select
ManualResourcePoolScaling
from the list. -
On the
ManualResourcePoolScaling
procedure page, select New run. -
Provide the following parameter values:
-
For Resource pool name, provide the name of a resource pool you configured in Add agent resource pool.
This resource pool must already exist. If it does not, the procedure fails. -
For Maximum number of resources, provide the maximum number of resources allowed for the resource pool. This value must be a positive integer.
If you allow a significantly high number of resources to be created, it will greatly increase resource consumption in your environment. -
For Desired number of resources, provide the number of resources you want the pool to scale to. This value must be:
-
A positive integer.
-
Not equal to the current resource count.
If you provide a number equal to the current resource count, the procedure fails. -
Less than or equal to Maximum number of resources.
If you provide a number higher than Maximum number of resources, the procedure fails. -
Greater than or equal to Minimum number of resources.
If you provide a number less than Minimum number of resources, the procedure fails.
-
-
For Minimum number of resources, provide the minimum number of resources you want to leave available in the pool. This value must be a positive integer.
-
-
Select OK to run the procedure.
The procedure runs and checks the number of resources in the resource pool. It then creates or removes resources until the count requested in Desired number of resources is reached.