Diagnosing if the TCP agent listener port thread has terminated

2 minute readKnowledge base

Issue

On operations center instances, controllers are failing to connect using inbound TCP.

On controllers, if you find that agents configured to connect using inbound TCP are failing to connect with the following error, it’s likely that the TCP agent listener port has terminated:

May 26, 2023 11:29:37 A.M. hudson.remoting.jnlp.Main$CuiListener status
INFO: Trying protocol: JNLP4-connect
May 26, 2023 11:29:37 A.M. org.jenkinsci.remoting.protocol.impl.BIONetworkLayer$Reader run
INFO: Waiting for ProtocolStack to start.
May 26, 2023 11:29:37 A.M. hudson.remoting.jnlp.Main$CuiListener status
INFO: Protocol JNLP4-connect encountered an unexpected exception
java.util.concurrent.ExecutionException: org.jenkinsci.remoting.protocol.impl.ConnectionRefusalException: Connection closed before acknowledgement sent
        at org.jenkinsci.remoting.util.SettableFuture.get(SettableFuture.java:223)
        at hudson.remoting.Engine.innerRun(Engine.java:809)
        at hudson.remoting.Engine.run(Engine.java:543)
Caused by: org.jenkinsci.remoting.protocol.impl.ConnectionRefusalException: Connection closed before acknowledgement sent
        at org.jenkinsci.remoting.protocol.impl.AckFilterLayer.onRecvClosed(AckFilterLayer.java:280)
        at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.onRecvClosed(ProtocolStack.java:825)
        at org.jenkinsci.remoting.protocol.NetworkLayer.onRecvClosed(NetworkLayer.java:155)
        at org.jenkinsci.remoting.protocol.impl.BIONetworkLayer.access$1000(BIONetworkLayer.java:51)
        at org.jenkinsci.remoting.protocol.impl.BIONetworkLayer$Reader.run(BIONetworkLayer.java:274)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:125)
        at java.base/java.lang.Thread.run(Thread.java:829)

May 26, 2023 11:29:37 A.M. hudson.remoting.jnlp.Main$CuiListener status
INFO: reconnect rejected, sleeping 10s:
java.lang.Exception: The server rejected the connection: None of the protocols were accepted
        at hudson.remoting.Engine.onConnectionRejected(Engine.java:888)
        at hudson.remoting.Engine.innerRun(Engine.java:835)
        at hudson.remoting.Engine.run(Engine.java:543)

Note that if you simply do not have the inbound TCP port enabled, you will encounter the message:

May 25, 2023 5:14:56 P.M. hudson.remoting.jnlp.Main$CuiListener status
INFO: Locating server among [JENKINS_URL]
May 25, 2023 5:14:56 P.M. hudson.remoting.jnlp.Main$CuiListener error
SEVERE: JENKINS_URL/tcpSlaveAgentListener/ is invalid: 404 Not Found
java.io.IOException: JENKINS_URL/tcpSlaveAgentListener/ is invalid: 404 Not Found
        at org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver.resolve(JnlpAgentEndpointResolver.java:221)
        at hudson.remoting.Engine.innerRun(Engine.java:755)
        at hudson.remoting.Engine.run(Engine.java:543)

Environment

Resolution

  1. First go to Manage Jenkins → Configure Global Security → TCP port for inbound agents and ensure it is enabled.

  2. If it is not enabled, set it to a fixed port, such as 50000 (or any available port that is open between the two machines), then try the connection again.

  3. If it is already enabled, it is possible that the TCP agent listener port thread terminated unexpectedly. To check if that is the case, go to JENKINS_URL/threadDump and search in your browser for the text TCP agent listener port, you should see that thread in RUNNABLE state.

  4. If you do not see that thread, you could be impacted by JENKINS-70334 or JENKINS-59910.

BEE-29155 JENKINS-70334 is fixed in Jenkins 2.388 and CloudBees CI 2.387.1.2.

Workaround

  1. Go to Manage Jenkins → Configure Global Security → TCP port for inbound agents and change it to Disable

  2. Click Apply to make the change active

  3. Set the TCP port for inbound agents to Fixed and choose a port number

  4. Click Apply to make the change active

  5. Try the connection again

Tested product/plugin versions

CloudBees Jenkins Platform - controller 2.346.4.1