Why is my ssh agent connection terminated with error: Server host key rejected

Issue

During the course of an ssh session with an agent either the server or the client can request a rekey, and openssh servers on linux have a setting for triggering a re-key event that is typically defaulted to every hour. This process typically goes on in the background unnoticed and uneventful:

SSH rekeying is the process of exchanging the session keys at a configured interval, based on a time limit or a data limit for the SSH session. SSH rekeying is triggered when the time limit (maximum minutes) or the data limit has been reached for the session. Rekey can be initiated by either the client or the server.

Citation from commscope.com

During the re-key process there is a key exchange (KEX) and the client and server negotiate on a set of ciphers and algorithms they support to continue the session. We traced, and reported, a bug in this procedure to a change in the Apache Mina-SSHd project - SSHD-1264 that CloudBees SSH Build Agents Plugin relies on.

This issue affects CloudBees SSH Build Agents Plugin version 2.16 and is fixed in version 2.19.

Symptoms

While using CloudBees SSH Build Agents Plugin, you encounter a disconnect every hour and see in your controller log files:

Apr 1, 2022 9:36:01 AM [ssh] Connection established in 3.238 seconds.
Apr 1, 2022 10:36:00 AM [ssh] Verifying server host key...
Apr 1, 2022 10:36:00 AM [ssh] RSA key fingerprint is SHA256:bjuNUryRnpHMO5qOFr9FfSHOv7gcaw62m+UorIxOZmU
Apr 1, 2022 10:36:00 AM [ssh] Server host key rejected
ERROR: Connection terminated
java.io.IOException: Pipe closed after 0 cycles
	at org.apache.sshd.common.channel.ChannelPipedInputStream.read(ChannelPipedInputStream.java:126)
	at org.apache.sshd.common.channel.ChannelPipedInputStream.read(ChannelPipedInputStream.java:105)
	at hudson.remoting.FlightRecorderInputStream.read(FlightRecorderInputStream.java:93)
	at hudson.remoting.ChunkedInputStream.readHeader(ChunkedInputStream.java:74)
	at hudson.remoting.ChunkedInputStream.readUntilBreak(ChunkedInputStream.java:104)
	at hudson.remoting.ChunkedCommandTransport.readBlock(ChunkedCommandTransport.java:39)
	at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34)
	at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:61)
ERROR: 4/1/22 10:36 AM agent was terminated
java.io.IOException: Pipe closed after 0 cycles
	at org.apache.sshd.common.channel.ChannelPipedInputStream.read(ChannelPipedInputStream.java:126)
	at org.apache.sshd.common.channel.ChannelPipedInputStream.read(ChannelPipedInputStream.java:105)
	at hudson.remoting.FlightRecorderInputStream.read(FlightRecorderInputStream.java:93)
	at hudson.remoting.ChunkedInputStream.readHeader(ChunkedInputStream.java:74)
	at hudson.remoting.ChunkedInputStream.readUntilBreak(ChunkedInputStream.java:104)
	at hudson.remoting.ChunkedCommandTransport.readBlock(ChunkedCommandTransport.java:39)
	at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34)
	at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:61)
Apr 1, 2022 10:37:58 AM [ssh] Opening connection to 10.227.198.157:22 as joel/****** (joel)
Apr 1, 2022 10:37:58 AM [ssh] Authenticating...
Apr 1, 2022 10:37:58 AM [ssh] Verifying server host key...
Apr 1, 2022 10:37:58 AM [ssh] ECDSA key fingerprint is SHA256:bhp3clzbDy1rglrEk/3KH9elIHRh3kaD4qD0KOY++fE
Apr 1, 2022 10:37:58 AM [ssh] Server host key verified
Apr 1, 2022 10:37:58 AM [ssh] Authentication successful.
Apr 1, 2022 10:37:58 AM [ssh] Checking for header junk...
Apr 1, 2022 10:37:58 AM [ssh] No header junk found.
Apr 1, 2022 10:37:58 AM [ssh] Inspecting operating system...
Apr 1, 2022 10:37:58 AM [ssh] Operating system: LINUX
Apr 1, 2022 10:37:58 AM [ssh] Remote SSH server: SSH-2.0-OpenSSH_7.4
Apr 1, 2022 10:37:58 AM [ssh] Verifying Java...
Apr 1, 2022 10:37:58 AM [ssh] Trying 'java' as a java command...
java -version returned 1.8.0.
Apr 1, 2022 10:37:58 AM [ssh] Verifying agent.jar...
Apr 1, 2022 10:37:59 AM [ssh] agent.jar verified, MD5: 0ecc112158d161e4017a98119d137d68
Apr 1, 2022 10:37:59 AM [ssh] $ cd '/cbci' && java  -jar agent.jar
<===[JENKINS REMOTING CAPACITY]===>channel started
Remoting version: 4.11.2
This is a Unix agent
Evacuated stdout
Agent successfully connected and online
Apr 1, 2022 10:38:02 AM [ssh] Connection established in 3.798 seconds.

Environment

Related Issue(s)

SSHD-1264: different host key algorithm used on rekey than used for the initial connection
BEE-18090: server re key not handled correctly, causing agents to disconnect

Resolution

The issue is fixed in Mina SSHD 2.9.0. That fix is consumed by CloudBees SSH Build Agents Plugin 2.19 that is available under the CloudBees Assurance Program in version 2.361.2.1 of CloudBees CI.

The solution is to upgrade CloudBees CI to version 2.361.2.1 or later

Workaround

The recommended resolution and workaround is to start using the SSH Build Agents Plugin to configure your ssh connected build agents.