How to Connect to Remote SSH Agents?

Article ID:222978868
5 minute readKnowledge base

Issue

  • How to do the initial connection of SSH agents to Jenkins, using SSH keys.

Environment

Resolution

You can watch the process of connecting an SSH agent on CloudBees TV at: https://youtu.be/99DddJiH7lM?t=550

Ideally, we recommend that the same user account jenkins is created on each Server that will be connected to your Jenkins controller. However, it is possible to run agents with a different user. Indeed, each agent acts as an SSH server and the Jenkins controller acts as the SSH Client.

Requirements

The controller needs to be able to connect to the agent via ssh, with the Private key known by the controller, and the corresponding public key is put in the agent’s ~/.ssh/authorized_keys file.

So, a SSH server needs to be installed in the agent (For instance, on Ubuntu: openssh-server), and a supported Java version must be installed on the agent as well.

Before trying to connect an agent to a controller. Make sure that when you login to the console of the controller, you can ssh to the agent:

ssh $AGENT_HOSTNAME

Expected output would be similar to:

ECDSA key fingerprint is SHA256:rX96v9WGIabHecNJusEYhXvENKyKe6GzQ/B/5ozRH9Q.
Are you sure you want to continue connecting (yes/no)?

Steps

In order to connect an SSH agent with the user $AGENT_USER, follow these steps:

On the agent machine:

  1. Log in to the agent machine as $AGENT_USER:

     sudo su $AGENT_USER
  2. Create private and public SSH keys. The following command creates the private key jenkinsAgent_rsa and the public key jenkinsAgent_rsa.pub. It is recommended to store your keys under ~/.ssh/ so we move to that directory before creating the key pair.

     mkdir ~/.ssh; cd ~/.ssh/ && ssh-keygen -t rsa -m PEM -C "Jenkins agent key" -f "jenkinsAgent_rsa"
    To increase security, a passphrase is advisable to associate it to your Private Key.
  3. Add the public SSH key to the list of authorized keys on the agent machine

     cat jenkinsAgent_rsa.pub >> ~/.ssh/authorized_keys
  4. Ensure that the permissions of the ~/.ssh directory is secure, as most ssh daemons will refuse to use keys that have file permissions that are considered insecure:

     chmod 700 ~/.ssh
     chmod 600 ~/.ssh/authorized_keys ~/.ssh/jenkinsAgent_rsa
  5. Copy the private SSH key (~/.ssh/jenkinsAgent_rsa) from the agent machine to your OS clipboard (eg: xclip, pbcopy, or ctrl-c).

     cat ~/.ssh/jenkinsAgent_rsa

    The output should be similiar to this:

     -----BEGIN RSA PRIVATE KEY-----
     ...
     -----END RSA PRIVATE KEY-----

In Jenkins:

  1. Go to the Build Executor Status link on the Jenkins dashboard:

    Build Executor Status
  2. Choose New Node, and Permanent agent type:

    New Node
    Permanent Agent
  3. Fill in the Remote root directory with a path the user on the agent is allowed to write to, set the Host value to the hostname of the agent, and press the Add button for Credentials:

    ssh agent configuration
  4. Choose SSH Username with private key, the Username is the user account on the agent machine (usually jenkins), and choose Private Key -> Enter directly and paste the key from your OS clipboard, and give a useful Description for this credential.

    ssh agent credentials

    Note: If you are setting up multiple agents, you can use the same public/private key for all agents (the public key would be put into ~/.ssh/authorized_keys on every agent, and the private key added as a credential on the controller can be re-used for every new agent)

  5. For additional details about SSH Host Key verification, see Host Key Verification for SSH Agents

  6. Save the configuration and the agent connection log should be similar to:

    Oct 2, 2018 3:13:43 PM [ssh] Opening connection to ssh-agent:22 as jenkins (agent ssh key)
    Oct 2, 2018 3:13:52 PM [ssh] Authenticating...
    Oct 2, 2018 3:13:56 PM [ssh] Verifying server host key...
    Oct 2, 2018 3:13:56 PM [ssh] ECDSA key fingerprint is SHA256:...
    Oct 2, 2018 3:13:56 PM [ssh] Server host key verified
    Oct 2, 2018 3:14:00 PM [ssh] Authentication successful.
    Oct 2, 2018 3:14:00 PM [ssh] Checking for header junk...
    Oct 2, 2018 3:14:01 PM [ssh] No header junk found.
    Oct 2, 2018 3:14:01 PM [ssh] Inspecting operating system...
    Oct 2, 2018 3:14:02 PM [ssh] Operating system: UNKNOWN
    Oct 2, 2018 3:14:02 PM [ssh] Remote SSH server: SSH-2.0-OpenSSH_7.4p1 Debian-10+deb9u3
    Oct 2, 2018 3:14:02 PM [ssh] Verifying Java...
    Oct 2, 2018 3:14:10 PM [ssh] Trying 'java' as a java command...
    java -version returned 1.8.0.
    Oct 2, 2018 3:14:11 PM [ssh] Verifying agent.jar...
    Oct 2, 2018 3:14:12 PM [ssh] Copying latest agent.jar...
    Oct 2, 2018 3:14:12 PM [ssh] Copied 32,768 bytes.
    ...
    Oct 2, 2018 3:14:14 PM [ssh] $ cd '/home/jenkins/' && java  -jar agent.jar
    <===[JENKINS REMOTING CAPACITY]===>channel started
    Remoting version: 3.25
    This is a Unix agent
    Evacuated stdout
    Agent successfully connected and online

On the agent machine:

Under the path specified in Remote root directory, the agent.jar would have been copied, and builds for jobs that run on this agent will allocate their workspace under ${remoteRootDirectory}/workspace.

NOTE: Depending on the permissions of the home directory for the agent machine’s user account, it may be necessary to set the jar cache directory - due to the issue JENKINS-18578 The default jar cache location is hardcoded to ~/.jenkins/cache/jars

To change the default setting:

In the agent configuration, under the Advanced method of Launch Option, Place the following in the Suffix Start Agent Command ` -jar-cache <path to jar cache directory>`

Make sure there’s a leading space so the parameters aren’t tacked directly onto the agent.jar itself.

Troubleshooting

If you recieve any errors in the logs when trying to connect to the agent, such as Authentication failed, you will likely need to gather the SSHD logs of the agent in order to find the real root cause. Where the logs are located my vary depending on the OS you are using, but some commands you can try running on the agent are:

grep sshd /var/log/secure
journalctl -u sshd
grep sshd /var/log/auth.log

Once you have the SSHD logs, look them over to find any errors. If you are unsure what the problem is please open a support ticket and attach the logs along with a support bundle for CloudBees Support to investigate.

I see the error Authentication refused: bad ownership or modes for file /home/<JENKINS_USER>/.ssh/authorized_keys

This is due to a security feature of SSHD. It will not allow use of the authorized_keys file unless only the owner of the file has read and write access. To fix this error:

  1. Ensure that the file is owned by the user running Jenkins. If not run the command: chown <JENKINS_USER>:<JENKINS_GROUP> /home/<JENKINS_USER>/.ssh/authorized_keys

  2. Make sure that the permissions on the file only allow the Jenkins user to read and write (rw-------). If not, run the command chmod 600 /home/<JENKINS_USER>/.ssh/authorized_keys

I see the error hudson.AbortException: Header junk detected

This error can be caused when the build agent you are trying to connect to is emitting unexpected messages during the ssh login, such as a message of the day (motd)

To resolve this, you may have to either modify /etc/motd, or disable it in the ~/.profile, ~/.bashrc, ~/.cshrc, or any other similar file for the user that Jenkins is trying to connect to (this change should be done on the build agent machine).

I see the error No entry currently exists in the Known Hosts file for this host

This happens when the ssh host key for the agent machine is not in the ~/.ssh/known_hosts file on the Jenkins controller. To fix this, you will need to configure SSH Host Key Verification, as documented in Host Key Verification for SSH Agents

I see the error Could not authenticate due to I/O issue
WARNING c.c.j.p.s.ClientSessionPublicKeyAuthenticator#doAuthenticate: Could not authenticate due to I/O issue
java.io.IOException: Could not parse PEM, only key pairs, private keys, public keys and certificates are supported

This error happens because you may have not generated your key pair with the RSA format. Create public and private RSA SSH keys and continue with these steps on the agent machine.

References