Websocket session memory leak causing controller restart - JENKINS-70103

3 minute readKnowledge base

Issue

When running a controller with Java 11 and connecting an agent via websocket transport which is running with Java 8, the agent will fail to connect repeatedly with the error:

Nov 19, 2022 10:42:19 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Connected
Nov 19, 2022 10:42:19 AM hudson.remoting.UserRequest perform
WARNING: LinkageError while performing UserRequest:hudson.slaves.SlaveComputer$SlaveVersion@299c90a9
java.lang.UnsupportedClassVersionError: Failed to load hudson.slaves.SlaveComputer$SlaveVersion
        at hudson.remoting.RemoteClassLoader.loadClassFile(RemoteClassLoader.java:460)
        at hudson.remoting.RemoteClassLoader.loadRemoteClass(RemoteClassLoader.java:292)
        at hudson.remoting.RemoteClassLoader.loadWithMultiClassLoader(RemoteClassLoader.java:269)
        at hudson.remoting.RemoteClassLoader.findClass(RemoteClassLoader.java:228)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:419)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:352)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:348)
        at hudson.remoting.MultiClassLoaderSerializer$Input.resolveClass(MultiClassLoaderSerializer.java:132)
        at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1986)
        at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1850)
        at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2160)
        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1667)
        at java.io.ObjectInputStream.readObject(ObjectInputStream.java:503)
        at java.io.ObjectInputStream.readObject(ObjectInputStream.java:461)
        at hudson.remoting.UserRequest.deserialize(UserRequest.java:289)
        at hudson.remoting.UserRequest.perform(UserRequest.java:189)
        at hudson.remoting.UserRequest.perform(UserRequest.java:54)
        at hudson.remoting.Request$2.run(Request.java:376)
        at hudson.remoting.InterceptingExecutorService.lambda$wrap$0(InterceptingExecutorService.java:78)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:126)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.UnsupportedClassVersionError: hudson/slaves/SlaveComputer$SlaveVersion has been compiled by a more recent version of the Java Runtime (class file version 55.0), this version of the Java Runtime only recognizes class file versions up to 52.0
        at java.lang.ClassLoader.defineClass1(Native Method)
        at java.lang.ClassLoader.defineClass(ClassLoader.java:757)
        at java.lang.ClassLoader.defineClass(ClassLoader.java:636)
        at hudson.remoting.RemoteClassLoader.loadClassFile(RemoteClassLoader.java:458)
        ... 24 more

Nov 19, 2022 10:42:19 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Write side closed

There is no delay between the connection failure and retry due to: https://issues.jenkins.io/browse/JENKINS-70105 (Fix: https://github.com/jenkinsci/remoting/pull/603)

Due to the lack of delay, this error can happen hundreds or thousands of times per minute per agent, and each connection attempt is leaking a websocket session. This memory leak eventually causes your controller to crash.

The only evidence of the problem in your controller logs would be continuous messages showing:

INFO  j.s.DefaultJnlpSlaveReceiver#channelClosed: Jetty (winstone)-1599 for a terminated: java.nio.channels.ClosedChannelException

The memory leak fix is tracked by: https://issues.jenkins.io/browse/JENKINS-70103 https://github.com/jenkinsci/jenkins/pull/7412

Introducing a 10 second delay for the retry is tracked by: https://issues.jenkins.io/browse/JENKINS-70105 https://github.com/jenkinsci/remoting/pull/603

Environment

Resolution

The solution to this issue is to upgrade to CloudBees CI version 2.375.1.1. The release notes can be found at CloudBees CI release notes.

We can help plan your next upgrade, see how at Required Data: Assisted Upgrade.

Workaround

There are multiple workarounds possible for this issue:

  • Upgrade the agents to Java 11

  • Shut down the Java 8 agents temporarily until they are upgraded to Java 11

  • If you upgraded and started to encounter this issue, revert the CloudBees CI version and restore the JENKINS_HOME from the backup from before the upgrade, and then run the controller with Java 8 (and wait for a new product version with the fix)

  • Disable using websocket transport for your agents until the issue is fixed (instead use inbound TCP, or ssh to connect your agents)

If disabling websocket support for Kubernetes shared cloud configuration or any other Kubernetes cloud configurations in your environment:

  • when disabling websockets for the agents running in the same cluster as your controller, that should require nothing more than un-checking the WebSocket checkmark for your Kubernetes shared cloud definition, which will cause the agent pods to connect to the controller via TCP using the controller’s internal Kubernetes service (the default for in-cluster Kubernetes agents)

  • when disabling websockets for agents running in a separate Kubernetes cluster as your controller, first you would have to open a TCP port for the remoting connection on your nginx ingress controller, by following Connecting inbound agents using the values.yaml file, next you would uncheck the WebSocket checkmark for your Kubernetes cloud definition, and fill in the hostname and port opened up on your ingress controller under the Jenkins tunnel option.

Tested product/plugin versions