Issue
-
After upgrading CloudBees CI to a version between 2.401.3.3 and 2.414.3.7 (excluded), the controller is experiencing slowness in the UI and job execution
-
When collecting a thread dump, we see many threads blocked on
jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber:[...] java.lang.Thread.State: BLOCKED (on object monitor) at jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:527) - waiting to lock <0x00000004e02b5c08> (a hudson.model.RunMap) [...]
And the thread holding the lock is spending time on
AbstractLazyLoadRunMap.loadNumberOnDisk:[...] jenkins.model.lazy.AbstractLazyLoadRunMap.loadNumberOnDisk(AbstractLazyLoadRunMap.java:336) jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:539) [...]
Environment
-
CloudBees CI on modern cloud platforms - operations center >= 2.401.3.3 and < 2.414.3.7
-
CloudBees CI on modern cloud platforms - managed controller >= 2.401.3.3 and < 2.414.3.7
-
CloudBees CI on traditional platforms - operations center >= 2.401.3.3 and < 2.414.3.7
-
CloudBees CI on traditional platforms - client controller >= 2.401.3.3 and < 2.414.3.7
Explanation
A change in build loading from 2.401.3.3 through 2.414.3.7 (excluded) is causing expensive filesystem operations that may considerably slow down navigation and background processes. This is particularly impactful in environment with a large number of items / builds as well as environment over a shared file storage solution such as NFS.
Resolution
Upgrade CloudBees CI to version 2.414.3.7 or later.