Issue
HA can fail with many java.lang.NoClassDefFoundError related exceptions:
Exception in thread "ConnectionMap.Acceptor [10.0.0.1:56736],null,null" java.lang.NoClassDefFoundError: org/jgroups/blocks/TCPConnectionMap$TCPConnection$Receiver at org.jgroups.blocks.TCPConnectionMap$TCPConnection.start(TCPConnectionMap.java:424) at org.jgroups.blocks.TCPConnectionMap$Acceptor.handleAccept(TCPConnectionMap.java:310) at org.jgroups.blocks.TCPConnectionMap$Acceptor.run(TCPConnectionMap.java:283) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.ClassNotFoundException: org.jgroups.blocks.TCPConnectionMap$TCPConnection$Receiver at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at winstone.classLoader.WebappClassLoader.loadClass(WebappClassLoader.java:83) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ... 4 more Apr 01, 2016 3:13:35 PM org.jgroups.util.TimeScheduler2$RecurringTask run SEVERE: failed running task STABLE: StableTask java.lang.NoClassDefFoundError: org/jgroups/protocols/pbcast/STABLE$1 at org.jgroups.protocols.pbcast.STABLE.sendStableMessage(STABLE.java:687) at org.jgroups.protocols.pbcast.STABLE$StableTask.run(STABLE.java:829) at org.jgroups.util.TimeScheduler2$RecurringTask.run(TimeScheduler2.java:561) at org.jgroups.util.TimeScheduler2$MyTask.run(TimeScheduler2.java:493) at org.jgroups.util.TimeScheduler2$Entry.execute(TimeScheduler2.java:398) at org.jgroups.util.TimeScheduler2$1.run(TimeScheduler2.java:258) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.ClassNotFoundException: org.jgroups.protocols.pbcast.STABLE$1 at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at winstone.classLoader.WebappClassLoader.loadClass(WebappClassLoader.java:83) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ... 9 more
The problem impacts older versions of CJP and customers who use custom init scripts that probably don’t take the --webroot switch into account. This problem is happening because the default webroot data resides in the ${JENKINS_HOME}/war directory which is a default setting for single node installations of Jenkins.
Newer versions of CJP include a --webroot switch, but older customers may need to make this adjustment manually if they want to use the HA features properly.
Resolution
Adjust or add the webroot switch and point it ouside of ${JENKINS_HOME}
To resolve the problem, we advise adding --webroot to your init scripts or Jenkins arguments and point it to a folder that exists outside of your shared ${JENKINS_HOME} data directory. Note that the webroot data needs to be stored and accessible on each individual node on it’s local filesystem, so avoid sharing it over NFS.
Quick example:
java -server -jar jenkins.war --webroot=/var/cache/jenkins/war
Make sure that the new folder you create for the webroot data has read/write permissions for your jenkins service account on each HA node.