Issue
-
The controller thread dump shows hundreds / thousands of
OkHttp ConnectionPool
threads similar to:
"OkHttp ConnectionPool" id=679 (0x2a7) state=TIMED_WAITING cpu=0% - waiting on <0x09293902> (a okhttp3.ConnectionPool) - locked <0x09293902> (a okhttp3.ConnectionPool) at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:460) at okhttp3.ConnectionPool$1.run(ConnectionPool.java:67) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Locked synchronizers: count = 1 - java.util.concurrent.ThreadPoolExecutor$Worker@2b7a6107
-
The controller Jenkins logs shows socket exception like the following:
Dec 12, 2018 1:22:08 AM io.fabric8.kubernetes.client.dsl.internal.ExecWebSocketListener onFailure SEVERE: Exec Failure: HTTP:0. Message:No response java.net.SocketException: Socket closed at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) at java.net.SocketInputStream.read(SocketInputStream.java:171) at java.net.SocketInputStream.read(SocketInputStream.java:141) at sun.security.ssl.InputRecord.readFully(InputRecord.java:465) at sun.security.ssl.InputRecord.read(InputRecord.java:503) at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:983) at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:940) at sun.security.ssl.AppInputStream.read(AppInputStream.java:105) at okio.Okio$2.read(Okio.java:139) [...]
Environment
-
CloudBees CI (CloudBees Core) on modern cloud platforms - Managed controller
-
CloudBees CI (CloudBees Core) on modern cloud platforms - Operations Center
-
CloudBees CI (CloudBees Core) on traditional platforms - Client controller
-
CloudBees CI (CloudBees Core) on traditional platforms - Operations Center
-
Kubernetes Plugin < 1.13.6
Resolution
This is caused by a thread leak in the Kubernetes Plugin. Every time plugin needs to connect to the kubernetes server, a new client object is created.
The issue is captured by JENKINS-54770 that has been fixed in version 1.13.6 of the Kubernetes Plugin.
Solution
The solution is to upgrade the Kubernetes Plugin to version 1.13.6 or later.