GitHub events not Processed in Time due to Lock held by 'GitHubClient.checkApiUrlValidity'

Article ID:4402585116315
2 minute readKnowledge base

Issue

  • GitHub Events are not processed by the Organization / Mutlibranch Event mechanism on time

  • Thread Dumps show many BLOCKED threads waiting for a lock on org.jenkinsci.plugins.github_branch_source.Connector.connect similar to:

      "Executor #-1 for master : executing <itemFullName> #1" [...] waiting for monitor entry [...]
          java.lang.Thread.State: BLOCKED (on object monitor)
          at org.jenkinsci.plugins.github_branch_source.Connector.connect(Connector.java:345)
          [...]

    held by a single thread, similar to:

      "class org.jenkinsci.plugins.github_branch_source.PullRequestGHEventSubscriber$SCMHeadEventImpl Thu Jun 10 16:50:16 PDT 2021 / SCMEvent [#3]" #609 [...] waiting on condition [...]
          [...]
          at org.jenkinsci.plugins.github_branch_source.ApiRateLimitChecker$LocalChecker.waitUntilRateLimit(ApiRateLimitChecker.java:287)
          [...]
          at org.kohsuke.github.GitHub.checkApiUrlValidity(GitHub.java:1195)
          at org.jenkinsci.plugins.github_branch_source.Connector$GitHubConnection.verifyConnection(Connector.java:678)
          - locked <0x000000055d407510> (a org.jenkinsci.plugins.github_branch_source.Connector$GitHubConnection)
          [...]

Explanation

This is caused by a known issue JENKINS-65262: High lock contention in Queue causes builds to not trigger introduced in both release line 2.9.7 and 2.10.2 of the [GitHub Branch Source plugin]. The problem occurs when hitting the GitHub rate limits. One call wait for the rate limit quota to resets while holding a lock on the Connector. This blocks threads to even lookup for a connection.

This issue is fixed in version 2.10.3 of the GitHub Branch Source plugin. Also backported to version 2.9.9.

Resolution

The solution is to upgrade CloudBees CI to version 2.289.1.2 or later.

Workaround

A workaround is to upgrade the GitHub Branch Source plugin:

  • to version 2.10.4, if running CloudBees CI 2.277.x or later

  • to version 2.9.9, if running CloudBees CI from version 2.222.x to 2.263.x