User Lookup Performances Issues due to SECURITY-243

Article ID:230163927
2 minute readKnowledge base

Issue

  • Jenkins is getting slow and eventually freezes

  • Many slow-requests and / or threads that are WAITING trying to get user information from the Security Realm

"Handling GET /job/myJob/job/develop/changes from X.X.X.X : RequestHandlerThread[#13]" [...]
    [...]
    .*.loadUserByUsername(.*)
    hudson.model.User$UserIDCanonicalIdResolver.resolveCanonicalId(User.java:1049)
    hudson.model.User.get(User.java:395)
    hudson.model.User.get(User.java:364)
    [...]

  • Many slow-requests and / or threads that are WAITING trying to get user information from Security Realm (Pipeline Stage View)

"Handling GET /job/myJob/job/develop/1/wfapi/changesets from X.X.X.X : RequestHandlerThread[#13]" [...]
    [...]
    .*.loadUserByUsername(.*)
    hudson.model.User$UserIDCanonicalIdResolver.resolveCanonicalId(User.java:1049)
    hudson.model.User.get(User.java:395)
    hudson.model.User.get(User.java:364)
    hudson.plugins.git.GitChangeSet.findOrCreateUser(GitChangeSet.java:374)
    hudson.plugins.git.GitChangeSet.getAuthor(GitChangeSet.java:435)
    com.cloudbees.workflow.rest.external.ChangeSetExt.mapFields(ChangeSetExt.java:176)
    com.cloudbees.workflow.rest.external.ChangeSetExt.create(ChangeSetExt.java:151)
    com.cloudbees.workflow.rest.endpoints.RunAPI.doChangesets(RunAPI.java:112)
    [...]

Environment

  • CloudBees Jenkins Enterprise - Managed controller (CJEMM) versions between 1.642.18.x and 2.7.19.x (before CJE 1.3.0)

  • Jenkins LTS versions between 1.651.2 and 2.7.3

Related Issue(s)

Explanation

Since the resolution of SECURITY-243, all the users in the commits of an SCM changeset need to be looked up in the Security Realm to see if they are valid authentication "users". For more details, please have look at JENKINS-35493.

This can cause performance issues, especially for large instances. Symptoms are a large number of threads requesting user information to the Security Realm, instance unresponsive and users unable to login. When the Security Realm server is overloaded, Jenkins eventually freezes and needs to be restarted.

In general, any process interacting with SCM changesets is impacted (for example viewing a page with changelog). In particular, Pipeline jobs are impacted as the Pipeline Stage View Plugin attempts to resolve authors of ChangeLog entries of the ChangeSet to specific users, and this API is invoked with each page view, so it is hit frequently. For more details and a potential workaround, please have look at JENKINS-35484.

Resolution

Therefore the solution is to upgrade: A cache has been implemented for User/Groups in the version 2.7.3 of Jenkins Core. For more details, please have a look at the pull request of Jenkins core PR#2446

  • CJE: upgrade to version 1.3.0 or later

  • CJP / CJT: upgrade to version 2.7.19.x or later

  • Jenkins LTS: upgrade to version 2.7.3 or later

Before Jenkins 2.7.3 / CJP 2.7.19.x

If upgrading Jenkins is not an option, the workaround is to disable the user lookup behavior using the following system properties:

-Dhudson.model.User.SECURITY_243_FULL_DEFENSE=false