SSH Agents leak "Computer.threadPoolForRemoting" threads

Issue

Jenkins is unresponsive and/or the instance shows a growing number of BLOCKED threads like the following:

  "Computer.threadPoolForRemoting [#...]" #XXXXX daemon [...] waiting for monitor entry [...]
     java.lang.Thread.State: BLOCKED (on object monitor)
      at hudson.plugins.sshslaves.SSHLauncher.afterDisconnect(SSHLauncher.java:1330)
      - waiting to lock <0x00000003c6da7ed0> (a hudson.plugins.sshslaves.SSHLauncher)
      at hudson.slaves.SlaveComputer$3.run(SlaveComputer.java:627)
      at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      at java.lang.Thread.run(Thread.java:745)

Environment

CloudBees Jenkins Enterprise - Managed controller (CJE-MM)
Jenkins LTS
SSH Build Agents plugin

Related Issues

Resolution

The threadPoolForRemoting is an executor pool which is widely used. Such a threads burst is most likely caused by underlying bugs and issues that create threads intensively, so intensively that it clogs the executors service.

For this particular stacktrace - that shows it is related to the SSH Launcher - there is an epic that is currently under investigation: JENKINS-27514. Until this is resolved, here are possible solutions and workarounds to avoid this:

Solution

Upgrade SSH Agents plugin to version 1.24 or later

Many improvements are made to the SSH Build Agents plugin so it is important to keep ip up to date. In particular, version 1.24 contains fixes that prevent the thread spikes caused by SSH connections.

Workarounds

Configure SSH Agents with a launch timeout

By default, the SSH Launcher has no connection launch timeout. One workaround for that issue is to specify a launch timeout in the Advanced configuration of your SSH Agents:

Use the NIO SSH Build Agents plugin

We recommend to use the NIO SSH Build Agents plugin which provides a much better throughput than the SSH Build Agents plugin.