Agent / Controller provisioning fails and Jenkins logs show a stacktrace similar to the following:
okhttp3.internal.http2.ConnectionShutdownException at okhttp3.internal.http2.Http2Connection.newStream(Http2Connection.java:219) at okhttp3.internal.http2.Http2Connection.newStream(Http2Connection.java:205) [...] at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:404)
Or the following:
okhttp3.internal.http2.StreamResetException: stream was reset: PROTOCOL_ERROR at okhttp3.internal.http2.Http2Stream.takeHeaders(Http2Stream.java:158) at okhttp3.internal.http2.Http2Codec.readResponseHeaders(Http2Codec.java:131) at okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.java:88) [...] at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:411)
CPLT2-5621: Fix "`Is alpn-boot on the boot class path?`"
CPLT2-6044: revert CPLT2-5621
This is caused by Java - more precisely the okhttp library used by the kubernetes client - that chooses the wrong protocol HTTP/2 to communicate with the Kubernetes API Server although it does not support it. There a couple of changes meant to bring support for HTTP/2 to Java 8 and Java clients that could explain this problem. And also a change in Kubernetes:
In CloudBees CI 188.8.131.52, the
alpn-boot.jarwas added to the classpath in CloudBees images to fix CPLT2-5621. That brings support for HTTP/2 and cause
okhttpto use HTTP/2 in some circumstances.
In Kubernetes 1.17 and later,
okhttpseems to wrongly choose the HTTP/2 protocol to communicate with Kubernetes but the
kube-apiserverdoes not support it. Our experience is that this happens also with Openshift 4.2 and later.
Running version 184.108.40.206 of CloudBees CI in Kubernetes version older than 1.17 or Openshift version older than 4.2 is very risky. If Kubernetes/Openshift is upgraded, controller and agent provisioning would likely stop working and there is no workaround possible other than upgrading CloudBees CI. In public Cloud, clusters might be automatically upgraded by the Cloud operator (GKE, EKS, AKS, …).
Some solutions emerged to workaround that problem:
In kubernetes-client 4.4.0, there is a system property
http2.disablethat can be used to disable HTTP/2 and can workaround the problem.
In CloudBees CI 220.127.116.11, kubernetes-client 4.4.0 is available and the system property
http2.disablecan be used
In CloudBees CI 18.104.22.168, the
alpn-boot.jarwas removed from the classpath in CloudBees images to fix the problem CPLT2-6044. The system property
http2.disableshould not be needed as this should prevent
okhttpfrom choosing HTTP/2 if it is not supported.
Kubernetes Client maintainers addressed the problem directly as it is discovered that the same behavior is caused by JDK 8u252 that brings support for HTTP/2 with JEP-244:
The recommended solution is to upgrade CloudBees CI to version 22.214.171.124 or later. That version guarantee that the kubernetes client uses http1.1 when communicating with Kubernetes and prevent that
CloudBees CI >= 126.96.36.199
If impacted, add the System Property
http2.disable=true to the startup of Operations Center and Managed Controllers. See How to add Java arguments to Jenkins on CloudBees CI Modern ? for details.
CloudBees CI <= 188.8.131.52
If impacted, there is no workaround. CloudBees CI must be upgraded to a version that has a workaround (184.108.40.206 or later) or a version that contains the solution (220.127.116.11 or later).