Resolution
You can watch the process of connecting an SSH agent on CloudBees TV at: https://youtu.be/99DddJiH7lM?t=550
Ideally, we recommend that the same user account jenkins is created on each Server that will be connected to your Jenkins controller. However, it is possible to run agents with a different user. Indeed, each agent acts as an SSH server and the Jenkins controller acts as the SSH Client.
Requirements
The controller needs to be able to connect to the agent via ssh, with the Private key known by the controller, and the corresponding public key is put in the agent’s ~/.ssh/authorized_keys file.
So, a SSH server needs to be installed in the agent (For instance, on Ubuntu: openssh-server), and a supported Java version must be installed on the agent as well.
Before trying to connect an agent to a controller. Make sure that when you login to the console of the controller, you can ssh to the agent:
ssh $AGENT_HOSTNAME
Expected output would be similar to:
ECDSA key fingerprint is SHA256:rX96v9WGIabHecNJusEYhXvENKyKe6GzQ/B/5ozRH9Q. Are you sure you want to continue connecting (yes/no)?
Steps
In order to connect an SSH agent with the user $AGENT_USER, follow these steps:
On the agent machine:
-
Log in to the agent machine as
$AGENT_USER:sudo su $AGENT_USER
-
Create private and public SSH keys. The following command creates the private key
jenkinsAgent_rsaand the public keyjenkinsAgent_rsa.pub. It is recommended to store your keys under ~/.ssh/ so we move to that directory before creating the key pair.mkdir ~/.ssh; cd ~/.ssh/ && ssh-keygen -t rsa -m PEM -C "Jenkins agent key" -f "jenkinsAgent_rsa"
To increase security, a passphraseis advisable to associate it to your Private Key. -
Add the public SSH key to the list of authorized keys on the agent machine
cat jenkinsAgent_rsa.pub >> ~/.ssh/authorized_keys
-
Ensure that the permissions of the
~/.sshdirectory is secure, as most ssh daemons will refuse to use keys that have file permissions that are considered insecure:chmod 700 ~/.ssh chmod 600 ~/.ssh/authorized_keys ~/.ssh/jenkinsAgent_rsa
-
Copy the private SSH key (
~/.ssh/jenkinsAgent_rsa) from the agent machine to your OS clipboard (eg:xclip,pbcopy, or ctrl-c).cat ~/.ssh/jenkinsAgent_rsa
The output should be similiar to this:
-----BEGIN RSA PRIVATE KEY----- ... -----END RSA PRIVATE KEY-----
In Jenkins:
-
Go to the
Build Executor Statuslink on the Jenkins dashboard:
-
Choose
New Node, andPermanent agenttype:
-
Fill in the
Remote root directorywith a path the user on the agent is allowed to write to, set theHostvalue to the hostname of the agent, and press theAddbutton forCredentials:
-
Choose
SSH Username with private key, theUsernameis the user account on the agent machine (usuallyjenkins), and choosePrivate Key->Enter directlyand paste the key from your OS clipboard, and give a usefulDescriptionfor this credential.
Note: If you are setting up multiple agents, you can use the same public/private key for all agents (the public key would be put into
~/.ssh/authorized_keyson every agent, and the private key added as a credential on the controller can be re-used for every new agent) -
For additional details about SSH Host Key verification, see Host Key Verification for SSH Agents
-
Save the configuration and the agent connection log should be similar to:
Oct 2, 2018 3:13:43 PM [ssh] Opening connection to ssh-agent:22 as jenkins (agent ssh key) Oct 2, 2018 3:13:52 PM [ssh] Authenticating... Oct 2, 2018 3:13:56 PM [ssh] Verifying server host key... Oct 2, 2018 3:13:56 PM [ssh] ECDSA key fingerprint is SHA256:... Oct 2, 2018 3:13:56 PM [ssh] Server host key verified Oct 2, 2018 3:14:00 PM [ssh] Authentication successful. Oct 2, 2018 3:14:00 PM [ssh] Checking for header junk... Oct 2, 2018 3:14:01 PM [ssh] No header junk found. Oct 2, 2018 3:14:01 PM [ssh] Inspecting operating system... Oct 2, 2018 3:14:02 PM [ssh] Operating system: UNKNOWN Oct 2, 2018 3:14:02 PM [ssh] Remote SSH server: SSH-2.0-OpenSSH_7.4p1 Debian-10+deb9u3 Oct 2, 2018 3:14:02 PM [ssh] Verifying Java... Oct 2, 2018 3:14:10 PM [ssh] Trying 'java' as a java command... java -version returned 1.8.0. Oct 2, 2018 3:14:11 PM [ssh] Verifying agent.jar... Oct 2, 2018 3:14:12 PM [ssh] Copying latest agent.jar... Oct 2, 2018 3:14:12 PM [ssh] Copied 32,768 bytes. ... Oct 2, 2018 3:14:14 PM [ssh] $ cd '/home/jenkins/' && java -jar agent.jar <===[JENKINS REMOTING CAPACITY]===>channel started Remoting version: 3.25 This is a Unix agent Evacuated stdout Agent successfully connected and online
On the agent machine:
Under the path specified in Remote root directory, the agent.jar would have been copied, and builds for jobs that run on this agent will allocate their workspace under ${remoteRootDirectory}/workspace.
NOTE:
Depending on the permissions of the home directory for the agent machine’s user account, it may be necessary to set the jar cache directory - due to the issue JENKINS-18578
The default jar cache location is hardcoded to ~/.jenkins/cache/jars
To change the default setting:
In the agent configuration, under the Advanced method of Launch Option,
Place the following in the Suffix Start Agent Command ` -jar-cache <path to jar cache directory>`
Make sure there’s a leading space so the parameters aren’t tacked directly onto the agent.jar itself.
Troubleshooting
If you recieve any errors in the logs when trying to connect to the agent, such as Authentication failed, you will likely need to gather the SSHD logs of the agent in order to find the real root cause. Where the logs are located my vary depending on the OS you are using, but some commands you can try running on the agent are:
grep sshd /var/log/secure journalctl -u sshd grep sshd /var/log/auth.log
Once you have the SSHD logs, look them over to find any errors. If you are unsure what the problem is please open a support ticket and attach the logs along with a support bundle for CloudBees Support to investigate.
I see the error Authentication refused: bad ownership or modes for file /home/<JENKINS_USER>/.ssh/authorized_keys
This is due to a security feature of SSHD. It will not allow use of the authorized_keys file unless only the owner of the file has read and write access. To fix this error:
-
Ensure that the file is owned by the user running Jenkins. If not run the command:
chown <JENKINS_USER>:<JENKINS_GROUP> /home/<JENKINS_USER>/.ssh/authorized_keys -
Make sure that the permissions on the file only allow the Jenkins user to read and write (
rw-------). If not, run the commandchmod 600 /home/<JENKINS_USER>/.ssh/authorized_keys
I see the error hudson.AbortException: Header junk detected
This error can be caused when the build agent you are trying to connect to is emitting unexpected messages during the ssh login, such as a message of the day (motd)
To resolve this, you may have to either modify /etc/motd, or disable it in the ~/.profile, ~/.bashrc, ~/.cshrc, or any other similar file for the user that Jenkins is trying to connect to (this change should be done on the build agent machine).
I see the error No entry currently exists in the Known Hosts file for this host
This happens when the ssh host key for the agent machine is not in the ~/.ssh/known_hosts file on the Jenkins controller. To fix this, you will need to configure SSH Host Key Verification, as documented in Host Key Verification for SSH Agents
I see the error Could not authenticate due to I/O issue
WARNING c.c.j.p.s.ClientSessionPublicKeyAuthenticator#doAuthenticate: Could not authenticate due to I/O issue java.io.IOException: Could not parse PEM, only key pairs, private keys, public keys and certificates are supported
This error happens because you may have not generated your key pair with the RSA format. Create public and private RSA SSH keys and continue with these steps on the agent machine.
References
-
See SSH Credentials Management with Jenkins to get a better understanding of Crendentials Management and Scopes.