Older versions of Micro Focus Application Automation Tools plugin can cause deadlocks

Article ID:360027915232
2 minute readKnowledge base

Issue

Obtaining a thread dump shows, for example:

Found one Java-level deadlock:
=============================
"AbridgedConnectivityWorker-67":
  waiting to lock monitor 0x00007f90c2459368 (object 0x00000007a7fe37c0, a hudson.DescriptorExtensionList),
  which is held by "PreventRefreshFilter.initAutoRefreshFilter"
"PreventRefreshFilter.initAutoRefreshFilter":
  waiting to lock monitor 0x00007f90b81ab2e8 (object 0x00000006c1339400, a hudson.ExtensionList$Lock),
  which is held by "JSyncArtifactManagerFactory.upgrade"
"JSyncArtifactManagerFactory.upgrade":
  waiting to lock monitor 0x00007f90c2459368 (object 0x00000007a7fe37c0, a hudson.DescriptorExtensionList),
  which is held by "PreventRefreshFilter.initAutoRefreshFilter"

Java stack information for the threads listed above:
===================================================
"AbridgedConnectivityWorker-67":
	at hudson.ExtensionList.ensureLoaded(ExtensionList.java:317)
	- waiting to lock <0x00000007a7fe37c0> (a hudson.DescriptorExtensionList)
	at hudson.ExtensionList.iterator(ExtensionList.java:172)
	at hudson.ExtensionList.getInstance(ExtensionList.java:162)
	at jenkins.model.JenkinsLocationConfiguration.get(JenkinsLocationConfiguration.java:47)
	at jenkins.model.Jenkins.getRootUrl(Jenkins.java:2255)
	at com.hpe.application.automation.tools.octane.CIJenkinsServicesImpl.getServerInfo(CIJenkinsServicesImpl.java:102)
	at com.hp.octane.integrations.services.bridge.BridgeServiceImpl$1.run(BridgeServiceImpl.java:82)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
"PreventRefreshFilter.initAutoRefreshFilter":
	at hudson.ExtensionList.ensureLoaded(ExtensionList.java:317)
	- waiting to lock <0x00000006c1339400> (a hudson.ExtensionList$Lock)
	at hudson.ExtensionList.getComponents(ExtensionList.java:183)
	at hudson.DescriptorExtensionList.load(DescriptorExtensionList.java:192)
	at hudson.ExtensionList.ensureLoaded(ExtensionList.java:318)
	- locked <0x00000007a7fe37c0> (a hudson.DescriptorExtensionList)
	at hudson.ExtensionList.iterator(ExtensionList.java:172)
	at hudson.ExtensionList.get(ExtensionList.java:149)
	at hudson.plugins.claim.ClaimConfig.get(ClaimConfig.java:202)
	at hudson.plugins.claim.http.PreventRefreshFilter.initAutoRefreshFilter(PreventRefreshFilter.java:43)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at hudson.init.TaskMethodFinder.invoke(TaskMethodFinder.java:104)
	at hudson.init.TaskMethodFinder$TaskImpl.run(TaskMethodFinder.java:175)
	at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:296)
	at jenkins.model.Jenkins$5.runTask(Jenkins.java:1083)
	at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:214)
	at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
"JSyncArtifactManagerFactory.upgrade":
	at hudson.ExtensionList.ensureLoaded(ExtensionList.java:316)
	- waiting to lock <0x00000007a7fe37c0> (a hudson.DescriptorExtensionList)
	at hudson.ExtensionList.iterator(ExtensionList.java:172)
	at hudson.ExtensionList.get(ExtensionList.java:149)
	at jenkins.security.apitoken.ApiTokenPropertyConfiguration.get(ApiTokenPropertyConfiguration.java:58)
	at jenkins.security.ApiTokenProperty$DescriptorImpl.newInstance(ApiTokenProperty.java:386)
	at jenkins.security.ApiTokenProperty$DescriptorImpl.newInstance(ApiTokenProperty.java:359)
	at hudson.model.User.allocateDefaultPropertyInstancesAsNeeded(User.java:211)
	at hudson.model.User.load(User.java:198)
	at hudson.model.User.<init>(User.java:191)
	at hudson.model.User.getOrCreateById(User.java:523)
	at hudson.model.User.getById(User.java:619)
	at hudson.model.User.get(User.java:603)
	at hudson.model.User.current(User.java:586)
	at hudson.plugins.jobConfigHistory.PluginUtils.isUserExcluded(PluginUtils.java:121)
	at hudson.plugins.jobConfigHistory.JobConfigHistorySaveableListener.onChange(JobConfigHistorySaveableListener.java:55)
	at hudson.model.listeners.SaveableListener.fireOnChange(SaveableListener.java:81)
	at hudson.model.Descriptor.save(Descriptor.java:874)
	- locked <0x00000007a87ba378> (a jenkins.telemetry.Correlator)
	at jenkins.telemetry.Correlator.<init>(Correlator.java:51)
	at jenkins.telemetry.Correlator$$FastClassByGuice$$5678e8d6.newInstance(<generated>)
	at com.google.inject.internal.cglib.reflect.$FastConstructor.newInstance(FastConstructor.java:40)
	at com.google.inject.internal.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:61)
	at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:105)
	at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:85)
	at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:267)
	at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)
	at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1103)
	at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
	at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:145)
	at hudson.ExtensionFinder$GuiceFinder$FaultTolerantScope$1.get(ExtensionFinder.java:432)
	at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41)
	at com.google.inject.internal.InjectorImpl$2$1.call(InjectorImpl.java:1016)
	at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1092)
	at com.google.inject.internal.InjectorImpl$2.get(InjectorImpl.java:1012)
	at hudson.ExtensionFinder$GuiceFinder._find(ExtensionFinder.java:394)
	at hudson.ExtensionFinder$GuiceFinder.find(ExtensionFinder.java:385)
	at hudson.ClassicPluginStrategy.findComponents(ClassicPluginStrategy.java:493)
	at hudson.ExtensionList.load(ExtensionList.java:380)
	at hudson.ExtensionList.ensureLoaded(ExtensionList.java:318)
	- locked <0x00000006c1339400> (a hudson.ExtensionList$Lock)
	at hudson.ExtensionList.iterator(ExtensionList.java:172)
	at jenkins.model.Jenkins.getDescriptorByType(Jenkins.java:1539)
	at com.cloudbees.jenkins.plugins.jsync.archiver.JSyncArtifactManagerFactory.upgrade(JSyncArtifactManagerFactory.java:47)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at hudson.init.TaskMethodFinder.invoke(TaskMethodFinder.java:104)
	at hudson.init.TaskMethodFinder$TaskImpl.run(TaskMethodFinder.java:175)
	at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:296)
	at jenkins.model.Jenkins$5.runTask(Jenkins.java:1083)
	at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:214)
	at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

Found 1 deadlock.

We see the plugin mentioned:

	at com.hpe.application.automation.tools.octane.CIJenkinsServicesImpl.getServerInfo(CIJenkinsServicesImpl.java:102)

Resolution

A thread dump can be obtained using the following guides:

Upgrading hp-application-automation-tools-plugin to >= 5.6.2 should resolve the issue.

Uninstalling hp-application-automation-tools-plugin should also resolve the issue. See https://www.jenkins.io/doc/book/managing/plugins/#uninstalling-a-plugin

This article is part of our Knowledge Base and is provided for guidance-based purposes only. The solutions or workarounds described here are not officially supported by CloudBees and may not be applicable in all environments. Use at your own discretion, and test changes in a safe environment before applying them to production systems.