The Kubernetes Plugin leaks client connection threads

Article ID:360022194592
1 minute readKnowledge base

Issue

  • The controller thread dump shows hundreds / thousands of OkHttp ConnectionPool threads similar to:

"OkHttp ConnectionPool" id=679 (0x2a7) state=TIMED_WAITING cpu=0%
    - waiting on <0x09293902> (a okhttp3.ConnectionPool)
    - locked <0x09293902> (a okhttp3.ConnectionPool)
    at java.lang.Object.wait(Native Method)
    at java.lang.Object.wait(Object.java:460)
    at okhttp3.ConnectionPool$1.run(ConnectionPool.java:67)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)

    Locked synchronizers: count = 1
      - java.util.concurrent.ThreadPoolExecutor$Worker@2b7a6107

  • The controller Jenkins logs shows socket exception like the following:

Dec 12, 2018 1:22:08 AM io.fabric8.kubernetes.client.dsl.internal.ExecWebSocketListener onFailure
SEVERE: Exec Failure: HTTP:0. Message:No response
java.net.SocketException: Socket closed
  at java.net.SocketInputStream.socketRead0(Native Method)
  at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
  at java.net.SocketInputStream.read(SocketInputStream.java:171)
  at java.net.SocketInputStream.read(SocketInputStream.java:141)
  at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
  at sun.security.ssl.InputRecord.read(InputRecord.java:503)
  at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:983)
  at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:940)
  at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
  at okio.Okio$2.read(Okio.java:139)
  [...]

Environment

Related Issue(s)

Resolution

This is caused by a thread leak in the Kubernetes Plugin. Every time plugin needs to connect to the kubernetes server, a new client object is created.

The issue is captured by JENKINS-54770 that has been fixed in version 1.13.6 of the Kubernetes Plugin.

Solution

The solution is to upgrade the Kubernetes Plugin to version 1.13.6 or later.

This article is part of our Knowledge Base and is provided for guidance-based purposes only. The solutions or workarounds described here are not officially supported by CloudBees and may not be applicable in all environments. Use at your own discretion, and test changes in a safe environment before applying them to production systems.