Issue
-
GitHub Events are not processed by the Organization / Mutlibranch Event mechanism on time
-
Thread Dumps show many BLOCKED threads waiting for a lock on
org.jenkinsci.plugins.github_branch_source.Connector.connect
similar to:"Executor #-1 for master : executing <itemFullName> #1" [...] waiting for monitor entry [...] java.lang.Thread.State: BLOCKED (on object monitor) at org.jenkinsci.plugins.github_branch_source.Connector.connect(Connector.java:345) [...]
held by a single thread, similar to:
"class org.jenkinsci.plugins.github_branch_source.PullRequestGHEventSubscriber$SCMHeadEventImpl Thu Jun 10 16:50:16 PDT 2021 / SCMEvent [#3]" #609 [...] waiting on condition [...] [...] at org.jenkinsci.plugins.github_branch_source.ApiRateLimitChecker$LocalChecker.waitUntilRateLimit(ApiRateLimitChecker.java:287) [...] at org.kohsuke.github.GitHub.checkApiUrlValidity(GitHub.java:1195) at org.jenkinsci.plugins.github_branch_source.Connector$GitHubConnection.verifyConnection(Connector.java:678) - locked <0x000000055d407510> (a org.jenkinsci.plugins.github_branch_source.Connector$GitHubConnection) [...]
Environment
-
CloudBees CI (CloudBees Core) < 2.289.1.2
-
CloudBees CI (CloudBees Core) on modern cloud platforms - Managed controller < 2.289.1.2
-
CloudBees CI (CloudBees Core) on modern cloud platforms - Operations Center < 2.289.1.2
-
CloudBees CI (CloudBees Core) on traditional platforms - Client controller < 2.289.1.2
-
CloudBees CI (CloudBees Core) on traditional platforms - Operations Center < 2.289.1.2
-
CloudBees Jenkins Platform - Client controller < 2.289.1.2
-
CloudBees Jenkins Platform - Operations Center < 2.289.1.2
-
GitHub Branch Source plugin 2.9.7 and 2.10.2
Explanation
This is caused by a known issue JENKINS-65262: High lock contention in Queue causes builds to not trigger introduced in both release line 2.9.7 and 2.10.2 of the [GitHub Branch Source plugin]. The problem occurs when hitting the GitHub rate limits. One call wait for the rate limit quota to resets while holding a lock on the Connector. This blocks threads to even lookup for a connection.