Issue
-
I upgraded the SSH Build Agents plugin to the latest and I see the following warning in Manage Jenkins:
SSH Host Key Verifiers are not configured for all SSH agents on this Jenkins instance. This could leave these agents open to man-in-the-middle attacks. Update your agent configuration to resolve this -
My agents fail to connect to my Jenkins controller with one of the following stacktraces:
[04/03/17 10:03:51] [SSH] Opening SSH connection to [AGENT_HOSTNAME]:22. [04/03/17 10:03:51] [SSH] WARNING: No entry currently exists in the Known Hosts file for this host. Connections will be denied until this new host and its associated key is added to the Known Hosts file. Key exchange was not finished, connection is closed. java.io.IOException: There was a problem while connecting to [AGENT_HOSTNAME]:22
[04/04/17 08:45:39] [SSH] Opening SSH connection to [AGENT_HOSTNAME]:22. [04/04/17 08:45:39] [SSH] WARNING: The SSH key for this host does not match the key required in the connection configuration. Connections will be denied until until the host key matches the configuration key. Key exchange was not finished, connection is closed. java.io.IOException: There was a problem while connecting to [AGENT_HOSTNAME]:22
Resolution
Until now, SSH Agent were launched without using any Host Key verification which was a security concern. The release of SSH Build Agents plugin 1.15 fixes this by introducing a Host Key Verification strategy to SSH Agents. This new feature is designed to prevent man-in-the-middle attack as explained in the Jenkins Security Advisory 2017-03-20.
| The Man-in-the-middle attacks happens when a server pretend to be the remote Host, between you and the server you intend to connect to. In that case you are connecting to the "man-in-the-middle" that can intercept the information you transmit when you try to authenticate and use them to establish connection with the remote host. |
What is SSH Host Key Verification ?
Host key verification is a process verifying the remote host identity.
-
Host Keys are stored on the SSH Server under
/etc/ssh/and are used to identify the server (Jenkins agents acts as SSH Servers) -
The SSH Client keeps a list of Host keys that it trusts under
~/.ssh/known_hosts(Jenkins controller acts as the SSH Client)
The purpose of Host key verification is to ensure that you are connecting to the right remote host - the host you intend to connect to. Host keys are stored in the Known Hosts file (usually under ~/.ssh/known_hosts). When the SSH Client initiates a connection to a remote host, the remote host (SSH Server) sends its Host Key. The SSH Client then goes through this Known Hosts file and looks for existing host keys for this host. If one is found, it checks if the host key send by the remote host matches the Known host key.
The verification can fail for different reason. For more details, have a look at the Troubleshooting section below.
How to fix the Warning ?
It is recommended to use an Host Key Verification strategy when configuring an SSH Agent. If none is selected - because you just upgraded - you will see the warning SSH Host Key Verifiers are not configured for all SSH agents on this Jenkins instance. This could leave these agents open to man-in-the-middle attacks. Update your agent configuration to resolve this.
Simply re-saving your agents with the strategy Non verifying Verification Strategy fixes this, this is explained in JENKINS-43062.
We recommend however that you select an Host Verification Strategy to prevent man-in-the-middle attack. Please read the following section/s.
How to setup Host Key Verification for Jenkins ?
They are different verification strategies:
-
Know Hosts file Verification Strategy: use the Known Host file, configuration outside Jenkins
-
Manually provided Verification Strategy: provide the host key directly from Jenkins
-
Manually trusted Verification Strategy: trust the host key from Jenkins
-
Non verifying Verification Strategy: legacy behavior, no host key verification
Use the help tooltip to understand how each strategy works:
| When saving the configuration, the agent need to be relaunched to ensure the verification succeeded - in other words that Host Keys are correctly setup |
Know Hosts file Verification Strategy
The configuration happens outside Jenkins. You need to add an entry for the remote agent in the ~/.ssh/known_hosts on the controller’s machine
First remove the current host keys for the agent’s host:
ssh-keygen -R $AGENT_HOSTNAME_OR_IP
On the Agent’s host, run one of the following commands to get the agent’s Host Key.
Depending on the sshd version and configuration settings on your agent, the host key to use would be in either:
cat /etc/ssh/ssh_host_ecdsa_key.pub ecdsa-sha2-nistp256 AAAAE2VjZHN...
cat /etc/ssh/ssh_host_rsa_key.pub ssh-rsa AAAAB3NzaC1y...
cat /etc/ssh/ssh_host_dsa_key.pub ssh-dss AAAAB3NzaC1kc...
Append/Copy the hostname followed by this host key under ~/.ssh/known_hosts on the controller’s machine. For example:
10.1.2.5 ssh-rsa AAAAB3NzaC1yc2EAAAADAQAB…
Relaunch the agent in Jenkins.
Manually provided Verification Strategy
With this configuration, you can provide the host key in the Node’s configuration:
Ensure that you provide the key encoded with the right algorithm, either:
-
ecdsa-sha2-nistp256(from/etc/ssh/ssh_host_ecdsa_key.pub) -
ssh_rsa(from/etc/ssh/ssh_host_rsa_key.pub) -
ssh-dss(from/etc/ssh/ssh_host_dsa_key.pub)
If not, please have a look at the troubleshooting section about how to get the host key.
Manually trusted Verification Strategy
(This feature is stable starting from version 1.16 of the SSH Build Agents plugin)
With this configuration, an authorised Jenkins user with Computer.CONFIGURE need to manually approve/disapprove the remote host key in Jenkins if it has changed. Once approved, the host key will be marked as trusted.
The host key received on the initial connection will be automatically trusted. You have the option to enable Require manual verification of initial connection so that even the first connection requires approval from an authorised Jenkins user.
Example: Approval on initial connection:
Example: Approval after change of remote Host Key:
| You need to relaunch the agent manually after approving the host key |
Considerations for Shared Agents
Some strategy are not suitable for Shared Agents.
A Shared Agent is meant to be provisioned to Client controllers and therefore only connects when provisioned. The shared agent item on the CJOC is simply a definition/template of an agent used to create nodes on the Client controller that requests it. The connection - and therefore the Host Key verification - only happens between Client controller and Provisioned agent.
In this regards, some strategy might be more appealing and suitable than others:
-
Know Hosts file Verification Strategy: Requires to add the Host Key of the shared agent manually on each client controller’s Known Host File
-
Manually provided Verification Strategy: No configuration on the controller. Provide the Host Key once in the Shared Agent configuration.
-
Manually trusted Verification Strategy: No configuration on the controller. Note that if Require manual verification of initial connection is ticked, every time the node is provisioned an authorised user of the Client controller will have to manually hit Trust SSH Host Key. So this option is not really suitable for Shared Agent.
-
Non verifying Verification Strategy: No host key verification (unsafe)
Troubleshooting
If the agent logs shows the IOException The server hostkey was not accepted by the verifier callback, then the Host Key verification fails when connecting the agent, and you need to fix it. You can troubleshoot this from the controller’s machine logged in as the Jenkins user with the command:
ssh -o UserKnownHostsFile=~/.ssh/known_hosts -o StrictHostKeyChecking=yes -o HostKeyAlgorithms='ssh-rsa,ssh-dss' -i $PRIVATE_KEYFILE $AGENT_USER@$AGENT_HOSTNAME_OR_IP
The following examples show SSH commands outputs with / without Host Key Verification:
-
Without strict Host Key verification the client is prompted to automatically add an entry for the remote host:
jenkins@ubuntu-1:~$ ssh -i ~/.ssh/jenkins_node_rsa jenkins@10.1.2.5 The authenticity of host '10.1.2.5 (10.1.2.5)' can't be established. ECDSA key fingerprint is SHA256:w+lsfQRQZFfXg3kPDcisR/2c9vM5jWnO3Zl9G0R7VkY. Are you sure you want to continue connecting (yes/no)?
This is the default behavior. Typing "yes" will automatically add the host key to the Known Hosts file.
-
With strict Host Key verification but no entry for the remote host, this would fail:
jenkins@ubuntu-1 jenkins:~$ ssh -o UserKnownHostsFile=~/.ssh/known_hosts -o StrictHostKeyChecking=yes -o HostKeyAlgorithms='ssh-rsa,ssh-dss' -i ~/.ssh/jenkins_node_rsa jenkins@10.1.2.5 No RSA host key is known for 10.1.2.5 and you have requested strict checking. Host key verification failed.
The SSH Client does not know the host and cannot verify its identity. The entry needs to be added manually to the Known Hosts file.
-
With strict Host Key verification but wrong/unexpected host key, the connection would fail with the following scary message:
jenkins@ubuntu-1 jenkins:~$ ssh -o UserKnownHostsFile=~/.ssh/known_hosts -o StrictHostKeyChecking=yes -o HostKeyAlgorithms='ssh-rsa,ssh-dss' -i ~/.ssh/jenkins_node_rsa jenkins@10.1.2.5 @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @ @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY! Someone could be eavesdropping on you right now (man-in-the-middle attack)! It is also possible that a host key has just been changed. The fingerprint for the RSA key sent by the remote host is SHA256:ajWjatHT78IPzOcR9pMG3bMRHVhprv39jCgQMypcMNY. Please contact your system administrator. Add correct host key in /Users/jenkins/.ssh/known_hosts to get rid of this message. Offending ECDSA key in /Users/jenkins/.ssh/known_hosts:39 RSA host key for 10.1.2.5 has changed and you have requested strict checking. Host key verification failed.
This means that the host key sent by the remote host does not match the one expected - i.e. the one stored in the Known Hosts file. There could be different reasons for receiving that message:
-
It could be a sign of an attack
-
This could be due to the host key entry being stored using the wrong Key Algorithm
-
This could be due to a change of Host Key on the server
In any case, the Known Hosts file needs to be fixed.
Get the Host Key
You can read the remote host key it from the remote host’s machine:
cat /etc/ssh/ssh_host_rsa_key.pub
You can ask for the remote host key from the controller:
ssh-keyscan -t 'ssh-rsa,ssh-dss' $AGENT_HOSTNAME_OR_IP
Get current Known Hosts keys
You can list the current stored host keys of a specific host with the following command:
ssh-keygen -F $AGENT_HOSTNAME_OR_IP
This outputs the host key(s) stored for this specific host in the format [hostname] [algorithm] [key]. Following is an example of a host key with format ecdsa-sha2-nistp256:
10.1.2.5 ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAI...
This example shows an algorithm that is not supported by Jenkins ecdsa-sha2-nistp256 and that would make the verification fail
|