The Kubernetes Plugin leaks client connection threads

Article ID:360022194592
1 minute readKnowledge base

Issue

  • The controller thread dump shows hundreds / thousands of OkHttp ConnectionPool threads similar to:

"OkHttp ConnectionPool" id=679 (0x2a7) state=TIMED_WAITING cpu=0%
    - waiting on <0x09293902> (a okhttp3.ConnectionPool)
    - locked <0x09293902> (a okhttp3.ConnectionPool)
    at java.lang.Object.wait(Native Method)
    at java.lang.Object.wait(Object.java:460)
    at okhttp3.ConnectionPool$1.run(ConnectionPool.java:67)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)

    Locked synchronizers: count = 1
      - java.util.concurrent.ThreadPoolExecutor$Worker@2b7a6107

  • The controller Jenkins logs shows socket exception like the following:

Dec 12, 2018 1:22:08 AM io.fabric8.kubernetes.client.dsl.internal.ExecWebSocketListener onFailure
SEVERE: Exec Failure: HTTP:0. Message:No response
java.net.SocketException: Socket closed
  at java.net.SocketInputStream.socketRead0(Native Method)
  at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
  at java.net.SocketInputStream.read(SocketInputStream.java:171)
  at java.net.SocketInputStream.read(SocketInputStream.java:141)
  at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
  at sun.security.ssl.InputRecord.read(InputRecord.java:503)
  at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:983)
  at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:940)
  at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
  at okio.Okio$2.read(Okio.java:139)
  [...]

Environment

Related Issue(s)

Resolution

This is caused by a thread leak in the Kubernetes Plugin. Every time plugin needs to connect to the kubernetes server, a new client object is created.

The issue is captured by JENKINS-54770 that has been fixed in version 1.13.6 of the Kubernetes Plugin.

Solution

The solution is to upgrade the Kubernetes Plugin to version 1.13.6 or later.