Performance issues due to build loading in CloudBees CI 2.401.3.3 up to 2.414.2.2

Issue

After upgrading CloudBees CI to a version between 2.401.3.3 and 2.414.3.7 (excluded), the controller is experiencing slowness in the UI and job execution

When collecting a thread dump, we see many threads blocked on jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber:

[...]
   java.lang.Thread.State: BLOCKED (on object monitor)
  at jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:527)
  - waiting to lock <0x00000004e02b5c08> (a hudson.model.RunMap)
[...]

And the thread holding the lock is spending time on AbstractLazyLoadRunMap.loadNumberOnDisk:

[...]
    jenkins.model.lazy.AbstractLazyLoadRunMap.loadNumberOnDisk(AbstractLazyLoadRunMap.java:336)
    jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:539)
[...]

Environment

CloudBees CI on modern cloud platforms - operations center >= 2.401.3.3 and < 2.414.3.7
CloudBees CI on modern cloud platforms - managed controller >= 2.401.3.3 and < 2.414.3.7
CloudBees CI on traditional platforms - operations center >= 2.401.3.3 and < 2.414.3.7
CloudBees CI on traditional platforms - client controller >= 2.401.3.3 and < 2.414.3.7

Related Issue(s)

BEE-39952: Replace load-on-demand system for builds

Explanation

A change in build loading from 2.401.3.3 through 2.414.3.7 (excluded) is causing expensive filesystem operations that may considerably slow down navigation and background processes. This is particularly impactful in environment with a large number of items / builds as well as environment over a shared file storage solution such as NFS.

Resolution

Upgrade CloudBees CI to version 2.414.3.7 or later.