Kubernetes agents are failing with 'SocketTimeoutException: connect timed out'

Issue

My pods are getting created but some build are failing with an error similar to the following in the console output or controller logs:

java.net.SocketTimeoutException: connect timed out
     at java.net.PlainSocketImpl.socketConnect(Native Method)
     at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
     at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
     at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
     at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
     at java.net.Socket.connect(Socket.java:589)
     at okhttp3.internal.platform.Platform.connectSocket(Platform.java:129)
     at okhttp3.internal.connection.RealConnection.connectSocket(RealConnection.java:246)
     at okhttp3.internal.connection.RealConnection.connect(RealConnection.java:166)
     [...]

Environment

Related Issue(s)

JENKINS-53532

Explanation

The exception java.net.SocketTimeoutException: connect timed out is caused by the connection timeout being exceeded during the connection between the Jenkins controller and a kubernetes agent. This timeout applies after the connection has been established. It is set to 5s by the kubernetes plugin by default.

Before Kubernetes plugin version 1.22.3, a value of 0 results in a Connection Timeout of 10s: no timeout is explicitly set to the kubernetes client and the default timeout of the okhttp client is used.

Since Kubernetes plugin version 1.22.3, the minimum value possible for the Connection Timeout is 5s.

Resolution

If an instance is impacted by this problem, consider increasing the Connection Timeout in the Kubernetes Cloud configuration.

An issue in the kubernetes plugin prior to version 1.14.9 prevents from persisting the configured value of the connection timeout. The kubernetes plugin must first be upgraded to version 1.14.9 or later (CloudBees Core 2.164.3.2 or later).

CloudBees Core on Modern Platform

If using CloudBees Core on Modern Platform, this can be done from the Operations Center.

In the Operations Center, select the "All" view and configure the item "kubernetes shared cloud". Then adjust the value of the Connection Timeout. Once saved, it may take a few seconds for the change to be applied to all managed controllers.

Any Master

Go to Manage Jenkins Configure System Cloud and adjust the Connection Timeout of the Kubernetes cloud configuration.