Stash pullrequest builder plugin makes scheduled job builds are not triggering on time

Article ID:115003358408
2 minute readKnowledge base

Issue

  • Jenkins instance and users are in the same time zone but Jenkins System keeps triggering builds randomly.

  • On logs can be found a similar trace to the following:

2017-02-06 12:16:53.815-0600 [id=5089154]	SEVERE	s.s.stash.StashApiClient$1#call: Failing to get response from Stash PR GET/rest/api/1.0/projects/INNODELOC/repos/InNode-location-performance-ci/pull-requests
2017-02-06 12:16:53.816-0600 [id=37]	WARNING	hudson.triggers.Trigger#checkTriggers: stashpullrequestbuilder.stashpullrequestbuilder.StashBuildTrigger.run() failed for hudson.maven.MavenModuleSet@6d32c80b[Walmart-Technology-Reston/Enterprise-Location-Service-Performance-Test-CI]java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.RuntimeException: Didn't get a 200 response from Stash PR GET! Response; 'Not Found' with message; null
	at stashpullrequestbuilder.stashpullrequestbuilder.stash.StashApiClient.getRequest(StashApiClient.java:262)
	at stashpullrequestbuilder.stashpullrequestbuilder.stash.StashApiClient.getPullRequests(StashApiClient.java:86)
	at stashpullrequestbuilder.stashpullrequestbuilder.StashRepository.getTargetPullRequests(StashRepository.java:72)
	at stashpullrequestbuilder.stashpullrequestbuilder.StashPullRequestsBuilder.run(StashPullRequestsBuilder.java:32)
	at stashpullrequestbuilder.stashpullrequestbuilder.StashBuildTrigger.run(StashBuildTrigger.java:299)
	at hudson.triggers.Trigger.checkTriggers(Trigger.java:273)
	at hudson.triggers.Trigger$Cron.doRun(Trigger.java:222)
	at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:50)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.util.concurrent.ExecutionException: java.lang.RuntimeException: Didn't get a 200 response from Stash PR GET! Response; 'Not Found' with message; null
	at java.util.concurrent.FutureTask.report(FutureTask.java:122)
	at java.util.concurrent.FutureTask.get(FutureTask.java:206)
	at stashpullrequestbuilder.stashpullrequestbuilder.stash.StashApiClient.getRequest(StashApiClient.java:254)
	... 14 more
Caused by: java.lang.RuntimeException: Didn't get a 200 response from Stash PR GET! Response; 'Not Found' with message; null
	at stashpullrequestbuilder.stashpullrequestbuilder.stash.StashApiClient$1.call(StashApiClient.java:234)
	at stashpullrequestbuilder.stashpullrequestbuilder.stash.StashApiClient$1.call(StashApiClient.java:221)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	... 1 more

  • On thread dumps can be found a similar trace to the following which proves that Stash pullrequest builder plugin is doing too much work in the timer trigger thread, delaying all subsequent executions.

"jenkins.util.Timer [#1]" id=37 (0x25) state=TIMED_WAITING cpu=40%
    - waiting on <0x1c6062eb> (a java.util.concurrent.FutureTask)
    - locked <0x1c6062eb> (a java.util.concurrent.FutureTask)
    at sun.misc.Unsafe.park(Native Method)
    at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
    at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:426)
    at java.util.concurrent.FutureTask.get(FutureTask.java:204)
    at stashpullrequestbuilder.stashpullrequestbuilder.stash.StashApiClient.getRequest(StashApiClient.java:254)
    at stashpullrequestbuilder.stashpullrequestbuilder.stash.StashApiClient.getPullRequestMergeStatus(StashApiClient.java:150)
    at stashpullrequestbuilder.stashpullrequestbuilder.StashRepository.isPullRequestMergable(StashRepository.java:209)
    at stashpullrequestbuilder.stashpullrequestbuilder.StashRepository.isBuildTarget(StashRepository.java:261)
    at stashpullrequestbuilder.stashpullrequestbuilder.StashRepository.getTargetPullRequests(StashRepository.java:75)
    at stashpullrequestbuilder.stashpullrequestbuilder.StashPullRequestsBuilder.run(StashPullRequestsBuilder.java:32)
    at stashpullrequestbuilder.stashpullrequestbuilder.StashBuildTrigger.run(StashBuildTrigger.java:299)
    at hudson.triggers.Trigger.checkTriggers(Trigger.java:273)
    at hudson.triggers.Trigger$Cron.doRun(Trigger.java:222)
    at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:50)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

    Locked synchronizers: count = 1
      - java.util.concurrent.ThreadPoolExecutor$Worker@2b0a3762

Environment

JENKINS

JENKINS-41336

Resolution

First, disable the Stash pullrequest builder plugin because it is blocking the scheduler and preventing all timer triggers.

Then, think about your jobs making use of other plugins which enable to build pull requests from an Atlassian Stash instance.