Agent provisioning fails after Upgrading Kubernetes or CloudBees CI on Traditional Platforms or JDK

1 minute read

Issue

  • Agent provisioning fails and Jenkins logs show a stacktrace similar to the following:

okhttp3.internal.http2.ConnectionShutdownException
   at okhttp3.internal.http2.Http2Connection.newStream(Http2Connection.java:219)
   at okhttp3.internal.http2.Http2Connection.newStream(Http2Connection.java:205)
   [...]
   at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:404)

Or the following:

okhttp3.internal.http2.StreamResetException: stream was reset: PROTOCOL_ERROR
	at okhttp3.internal.http2.Http2Stream.takeHeaders(Http2Stream.java:158)
	at okhttp3.internal.http2.Http2Codec.readResponseHeaders(Http2Codec.java:131)
	at okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.java:88)
  [...]
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:411)

Environment

Related Issue

Explanation

This is caused by Java - more precisely the okhttp library used by the kubernetes client - that chooses the wrong protocol HTTP/2 to communicate with the Kubernetes API Server although it does not support it. There a couple of changes meant to bring support for HTTP/2 to Java 8 and Java clients that could explain this problem. But also a change in Kubernetes:

  • JDK 8u252 bring support for HTTP/2 with JEP-244, see Jetty, ALPN & Java 8u252. This may cause again okhttp to use HTTP/2 in some circumstances.

  • In Kubernetes 1.17 and later, okhttp seems to wrongly choose the HTTP/2 protocol to communicate with Kubernetes but the kube-apiserver does not support it. Our experience is that this happens also with Openshift 4.2 and later.

Some solutions emerged to workaround that problem:

  • In kubernetes-client 4.4.0, there is a system property http2.disable that can be used to disable HTTP/2 and can workaround the problem.

  • In CloudBees CI 2.190.2.2, kubernetes-client 4.4.0 is available and the system property http2.disable can be used

Kubernetes Client maintainers addressed the problem directly:

From our experience, JDK version 8u272 and later do not cause this http/2 problem anymore

Resolution

The recommended solution is to upgrade CloudBees CI to version 2.235.1.2 or later. That version guarantee that the kubernetes client uses http1.1 when communicating with Kubernetes and prevent this issue from happening.

Workaround

If an upgrade is not possible, there are potential workarounds.

CloudBees CI >= 2.190.2.2

If impacted, add the System Property http2.disable=true to the startup of the Controller. See How to add Java arguments to Jenkins? for details.

CloudBees CI < 2.190.2.2

If impacted, upgrade the JDK to a version greater than 8u272.