User Lookup Performances Issues due to hudson.model.User#getOrCreate

Issue

Jenkins is getting slow and eventually freezes
Many slow-requests and / or threads that are WAITING trying to retrieve user information from the file system

"Handling GET /m2/securityRealm/finishLogin from X.X.X.X : RequestHandlerThread[#4449]" [...]
    [...]
	at hudson.model.User.getOrCreate(User.java:429)
	at hudson.model.User.getById(User.java:529)
	at hudson.model.User$UserIDCanonicalIdResolver.resolveCanonicalId(User.java:1071)
	at hudson.model.User.get(User.java:399)
	at hudson.model.User.get(User.java:368)
	at hudson.model.User.get(User.java:488)
	[...]

Many slow-requests and / or threads that are WAITING trying to get user information from the file system (Pipeline Stage View)

"Handling GET /job/myJob/job/develop/1/wfapi/changesets from X.X.X.X : RequestHandlerThread[#13]" [...]

    [...]
    hudson.model.User.getOrCreate(User.java:425)
    hudson.model.User.get(User.java:403)
    hudson.model.User.get(User.java:362)
    hudson.plugins.git.GitChangeSet.findOrCreateUser(GitChangeSet.java:374)
    hudson.plugins.git.GitChangeSet.getAuthor(GitChangeSet.java:435)
    com.cloudbees.workflow.rest.external.ChangeSetExt.mapFields(ChangeSetExt.java:176)
    com.cloudbees.workflow.rest.external.ChangeSetExt.create(ChangeSetExt.java:151)
    com.cloudbees.workflow.rest.endpoints.RunAPI.doChangesets(RunAPI.java:112)
    [...]

Many slow-requests and / or threads that are WAITING trying to get user information from the file system (variant JENKINS-47429)

"Handling GET /job/myJob/job/develop/job/develop/3/wfapi/changesets from X.X.X.X : [...]

    [...]
    at hudson.model.User.getOrCreate(User.java:431)
    at hudson.model.User.getById(User.java:529)
    at hudson.model.User$UserIDCanonicalIdResolver.resolveCanonicalId(User.java:1071)
    at hudson.model.User.get(User.java:399)
    at hudson.model.User.get(User.java:368)
    at hudson.plugins.git.GitChangeSet.findOrCreateUser(GitChangeSet.java:379)
    [...]

Environment

CloudBees Jenkins Enterprise - Managed controller (CJE-MM) < 2.89.3.x (CJE 1.11.2)
CloudBees Jenkins Platform - Client controller (CJP-CM) < 2.89.3.x
CloudBees Jenkins Team (CJT) < 2.89.3.x
Jenkins LTS < 2.89.3

Related Issue(s)

Explanation

First fix in version 2.46.3.x

This is caused by a known issue JENKINS-43936. Jenkins always looks for user information in the configuration files (file system) even though it already has the information in memory. This results in lots of I/O operations which could lead to performance issues and Jenkins becoming unresponsive.

This can cause performance issues, especially for large instances. Symptoms are a large number of threads requesting user information from the file system, instance unresponsive, users unable to login. When the server and/or the remote storage is overloaded, Jenkins eventually freezes and needs to be restarted.

Second fix in version 2.89.3.x

The first fix was followed by JENKINS-47429. Jenkins was always running a poorly optimized legacy scan for users. This call was removed as the migration of legacy user was done in other place of the code, therefore rendering this code useless.

Possible workaround

processes and requests that retrieve User information may be impacted. In particular, Pipeline jobs may exacerbate the situation since Pipeline Stage View Plugin attempts to resolve authors of ChangeLog entries of the ChangeSet to specific users, and this API is invoked with each page view, so it is hit frequently. For more details and a potential workaround, please have look at JENKINS-35484.

Resolution

This issue has been fixed and released in Jenkins 2.93. The issue has been backported to the Jenkins LTS 2.89.3.

Therefore the solution is to upgrade:

CJE: upgrade to version 1.11.2 or later
CJP / CJT: upgrade to version 2.89.3.x or later
Jenkins LTS: upgrade to version 2.89.3 later