New Shared Agents / Clouds are not being provisioned for my jobs in the queue when I have agents that are suspended

Jenkins maintains a NodeProvisioner for every label. The NodeProvisioner maintains statistics about:

The NodeProvisioner is tasked with going and provisioning new agents when there are "none available", with the aim in such cases of provisioning enough nodes that the queue will tend to zero.

The NodeProvisioner implementation is essentially unchanged from when this was introduced. This is mentioned because the reason for the initial implementation was EC2. The concern was that the end-users want to spin up lots of nodes just for a queue spike as then they would have them running for the rest of the hour and costing money.

So what happens is that the NodeProvisioner uses exponentially weighted averages of the statistics, and the limits that are used against those statistics are biased to inhibit provisioning as the number of agents provisioned against the label increases. There are a number of system properties that control this calculation, and they can be can tweak them to alter provisioning behaviour.

Now, one of the things that end-users don’t want to happen is that the first run of the NodeProvisioner goes and requests a load of agents be provisioning, they get provisioned, they are in the middle of connecting to Jenkins and the NodeProvisioner runs again, sees that there are no available agents and provisions some more. For reasons that are lost in the depths of time, Jenkins thus counts agents that are in the process of connecting as being available (i.e. it assumes they will be available shortly) but they are not counted amongst the total number of agents…​ the net result is that when you look at the load statistics graph for a label, you can…​ and do…​ see a negative number of busy executors:

More specifically, play close attention to the part of the graph near zero:

The reason for this weird behaviour is that (at least prior to 1.607) Jenkins decided that the number of busy executors could be derived from the total number of on-line executors by subtracting the number of available executors (which from before was the number of on-line idle executors plus the number of connecting executors). So this negative busy count means that there are some executors in the process of connecting.

A little further digging reveals that in essence these executors coming on line end up poisoning the statistics that are used for provisioning. If you use the default LoadStatistics.CLOCK the poisoning takes about 5-6 minutes to clear out…​ (On the other hand there is a special case that ignores the statistics if you have absolutely no nodes matching the label).

So what you end up seeing is that the nodes get provisioned in batches, the provisioning poisons the statistic and blocks provisioning until the spike decays away, at which point the provisioning can allocate another bunch of nodes.

This behaviour is really only evident when using a non-monopolising retention strategy as with a monopolising strategy the provisioned agents will keep building jobs. But as other plugins implementing the Jenkins Cloud API have found (e.g. the Docker plugin) it is really a bug in Jenkins' core provisioning logic.

A number of fixes has been committed in this area:

The load graphs seen when running on 1.607 (though beware JENKINS-27708 and JENKINS-27871 which I fixed towards 1.610) result in much better provisioning behaviour.

So, there is no additional provisioning because you have more than zero executors which means that provisioning is based on the average number of available executors being close to zero and the provisioning of the executor you do have resulting in poisoning of that statistic which will take approximately 6 minutes to clear - or if the node is released and you then have zero executors available then the provisioning logic will switch back to the special case handling.