Managing agents

CloudBees CI allows administrators to standardize their build environments and share those environments with their teams. Standardized build environments are easier to manage, easier to upgrade, and provide consistency for users.

Agents are computers that handle the tasks of running builds in a CloudBees CI cluster, and within the context of a CloudBees CI cluster, they are shared resources for connected controllers.

Static agents

There may be times when the standardized CloudBees CI build environment cannot represent the required agent:

On operating systems such as Microsoft Windows, Oracle Solaris, FreeBSD, or OpenBSD.
When tool licenses permit the tool to run only on a dedicated agent.
In security configurations which require a specific agent (code signing certificates, hardware security devices, and so forth).

CloudBees CI provides advanced capabilities to manage platform-specific and purpose-built agents and to schedule work on agents.

For CloudBees CI on traditional platforms, CloudBees recommends cloudbees-core-agent when launching containerized agents.

CloudBees SSH Build Agents plugin

The CloudBees SSH Build Agents plugin is an alternative SSH agent connector that uses a non-blocking I/O architecture. This alternative connector shows different scalability properties from the standard SSH agent connector.

There are three main differences in scalability characteristics between the non-blocking I/O connector and the standard SSH agent connector:

The non-blocking I/O connector limits the number of threads that are used to maintain the SSH channel: when there are a large number of channels (that is, many SSH agents) the non-blocking connector uses fewer threads. This permits the UI to remain more responsive than with the standard SSH agent connector.
The non-blocking I/O connector balances CPU resource requirements. If it requires more CPU resources than are available, it applies back-pressure to the channels generating the load. This allows the system to remain responsive, but slightly increases build times, whereas the traditional SSH agent connector would typically lose the connection and register a corresponding build failure.
The non-blocking I/O connector is optimized for reduced connection time. For example, it avoids copying the agent JAR file unless necessary, and by default suppresses the logging of the agent environment.

There are four important technical notes regarding SSH keys:

The SSH client library used by the non-blocking I/O connector currently supports only RSA, DSA and (as of version 2.0) ECDSA keys.
The key must be in PEM key format. The CloudBees SSH Build Agents plugin currently does not support other key formats.
The maximum key size is limited by the Java Cryptography Extension (JCE) policy of the Jenkins controller’s JVM. Unless the unrestricted policy is installed, the plugin limits the RSA key size to 2048 bits. For more information, refer to "Java Cryptography Extension (JCE) Unlimited Strength Jurisdiction Policy Files" on the Oracle Java SE downloads page.
Versions of the plugin prior to 2.0 did not support connecting to CYGWIN or other Microsoft Windows-based SSH servers.

Use the SSH Build Agents plugin

The plugin adds a launch method: Launch build agents via SSH (Non-blocking I/O). The configuration options are almost identical to those in the traditional SSH agents connector.

The differences in the configuration parameters are:

If specified, a space character is appended to the Prefix Start Agent Command, and the entire string is prepended to the start agent command. The traditional SSH Agents connector requires that the user remembers to end the prefix with a space or a semicolon to avoid breaking the agent launch command.
If specified, a space is prepended to the Suffix Start Agent Command and the entire string is appended to the start agent command. The traditional SSH Agents connector requires that the user remembers to start the suffix with a space or a semicolon to avoid breaking the agent launch command.
The traditional SSH Agents connector always copies the agent environment variables to the agent launch log, whereas the non-blocking I/O SSH agents connector provides this as an option that defaults to off.

The logging of the agent environment is only of use when debugging initial connection issues and has a performance impact on the startup time. Once the agent channel has been established, the agent environment variables are accessible via the Node’s System Information screen.

For information on connecting remote SSH agents, refer to How to connect to remote SSH agents. If further assistance is needed, refer to CloudBees Support.

Microsoft Windows

The CloudBees SSH Build Agents Plugin version 2.0 and above supports agents running Microsoft Windows. There are some restrictions on connecting Microsoft Windows agents over SSH:

SSH server implementations on Microsoft Windows do not all behave the same when it comes to launching processes and managing the I/O streams from those processes. As a result, not every SSH server implementation works for launching Jenkins agents. At the time of writing, the recommended SSH server implementation is what provided the Cygwin project’s port of OpenSSH.

The Microsoft port of OpenSSH does not work correctly for connecting Jenkins agents. The way I/O streams are forwarded causes the connection to be terminated early. The plugin will log a warning if the Microsoft port of OpenSSH is detected, but will attempt to connect regardless.

The CloudBees recommended SSH server is Cygwin’s port of OpenSSH. Other SSH implementations on Windows may require special configuration and handling to function properly.
Microsoft Windows by default does not provide command line tools to verify the integrity of the agent JAR file.

As a result, when launching agents over SSH, the agent JAR file is always transferred.

Configure Windows agents

CloudBees has included agent installation instructions as a reference for Microsoft Windows administrators. These instructions have been validated by CloudBees to confirm that they are a minimal setup for Microsoft Windows Server 2012 using a GUI.

If no Microsoft Windows agents can connect over SSH, set up an agent using this procedure to determine whether it is either:

a network issue or
an issue with the configuration of the Microsoft Windows operating system images.

These instructions are provided as a working example. Additional steps and configuration settings may be required in your environment. The exact steps required depend on your own network risk profile and domain settings.

This procedure assumes the following prerequisites:

A server or virtual machine with at least 2GB of RAM and 60GB of disk space.
Installation of Microsoft Windows Server 2012 R2 x86_64 Standard (Server with a GUI).
All security updates from Microsoft have been applied to an operating system.
The server has not joined a domain, nor have any group policy files been applied to the server.

To configure the Windows Agent:

Install the Oracle Java Virtual Machine:
1. Go to https://java.com/en/download/.
2. Download and run the Java virtual machine installer for a supported version (refer to Supported platforms for CloudBees CI on traditional platforms).
3. Start a Command Prompt and verify that Java is correctly on the system PATH by running java -version. The output from the command should report the version of Java that has been installed.
Install Cygwin’s port of OpenSSH:
1. Go to https://cygwin.com/.
2. Download and run the setup-x86_64.exe installer.
  
  The default options should be enough, but it may be necessary to configure an HTTP proxy if the server cannot connect directly. It will be necessary to select a local mirror to download from.
3. From the Select Packages screen, select the OpenSSH package for installation:
4. Proceed to the next screen of the installer, and ensure that the Select required packages (RECOMMENDED) option is enabled.
5. After selecting Next the installation should proceed.
6. When the installation is completed, you are asked if you want to create an icon on the desktop and a start menu item. It is recommended that you select both options.
Validate that Cygwin and Java are correctly installed:
1. Launch a Cygwin64 Terminal.
2. Run java -version.
3. Verify that the output looks similar to the following (version numbers and usernames may differ):
4. Verify that the cygrunsrv program is correctly installed by running cygrunsrv -h. The output should be a help screen similar to the following:

Configure Cygwin’s port of OpenSSH:

From a Cygwin64 Terminal launched to run as an Administrator, run the ssh-host-config script.

Provide the following answers:

Should StrictModes be used: yes
Should privilege separation be used: yes
new local account 'sshd': yes
Do you want to install sshd as a service: yes

Enter the value of CYGWIN for the daemon:

A significant number of tutorials on the internet incorrectly advise you to enter binmode ntsec at this step. Those instructions are incorrect as of the Cygwin 1.7 release (2009).

Do you want to use a different name: no
Create new privileged user account 'WIN-…\cyg_server' (Cygwin name: 'cyg_server'): yes

Please enter the password: (you must provide a password)

*** Info: Generating missing SSH host keys
ssh-keygen: generating new host keys: RSA DSA ECDSA ED25519
*** Info: Creating default /etc/ssh_config file
*** Info: Creating default /etc/sshd_config file

*** Info: StrictModes is set to 'yes' by default.
*** Info: This is the recommended setting, but it requires that the POSIX
*** Info: permissions of the user's home directory, the user's .ssh
*** Info: directory, and the user's ssh key files are tight so that
*** Info: only the user has write permissions.
*** Info: On the other hand, StrictModes don't work well with default
*** Info: Windows permissions of a home directory mounted with the
*** Info: 'noacl' option, and they don't work at all if the home
*** Info: directory is on a FAT or FAT32 partition.
*** Query: Should StrictModes be used? (yes/no) yes

*** Info: Privilege separation is set to 'sandbox' by default since
*** Info: OpenSSH 6.1.  This is unsupported by Cygwin and has to be set
*** Info: to 'yes' or 'no'.
*** Info: However, using privilege separation requires a non-privileged account
*** Info: called 'sshd'.
*** Info: For more info on privilege separation read /usr/share/doc/openssh/README.privsep.
*** Query: Should privilege separation be used? (yes/no) yes
*** Info: Note that creating a new user requires that the current account have
*** Info: Administrator privileges.  Should this script attempt to create a
*** Query: new local account 'sshd'? (yes/no) yes
*** Info: Updating /etc/sshd_config file

*** Query: Do you want to install sshd as a service?
*** Query: (Say "no" if it is already installed as a service) (yes/no) yes
*** Query: Enter the value of CYGWIN for the daemon: []
*** Info: On Windows Server 2003, Windows Vista, and above, the
*** Info: SYSTEM account cannot setuid to other users -- a capability
*** Info: sshd requires.  You need to have or to create a privileged
*** Info: account.  This script will help you do so.

*** Info: It's not possible to use the LocalSystem account for services
*** Info: that can change the user id without an explicit password
*** Info: (such as passwordless logins [e.g. public key authentication]
*** Info: via sshd) when having to create the user token from scratch.
*** Info: For more information on this requirement, see
*** Info: https://cygwin.com/cygwin-ug-net/ntsec.html#ntsec-nopasswd1

*** Info: If you want to enable that functionality, it's required to create
*** Info: a new account with special privileges (unless such an account
*** Info: already exists). This account is then used to run these special
*** Info: servers.

*** Info: Note that creating a new user requires that the current account
*** Info: have Administrator privileges itself.

*** Info: No privileged account could be found.

*** Info: This script plans to use 'cyg_server'.
*** Info: 'cyg_server' will only be used by registered services.
*** Query: Do you want to use a different name? (yes/no) no
*** Query: Create new privileged user account 'WIN-...\cyg_server' (Cygwin name: 'cyg_server')? (yes/no) yes
*** Info: Please enter a password for new user cyg_server.  Please be sure
*** Info: that this password matches the password rules given on your system.
*** Info: Entering no password will exit the configuration.
*** Query: Please enter the password:
*** Query: Reenter:

*** Info: User 'cyg_server' has been created with password 'phish!998'.
*** Info: If you change the password, please remember also to change the
*** Info: password for the installed services which use (or will soon use)
*** Info: the 'cyg_server' account.


*** Info: The sshd service has been installed under the 'cyg_server'
*** Info: account.  To start the service now, call `net start sshd' or
*** Info: `cygrunsrv -S sshd'.  Otherwise, it will start automatically
*** Info: after the next reboot.

*** Info: Host configuration finished. Have fun!▼

Start the CYGWIN sshd service from the Services management panel.
Open and configure Windows Firewall:
1. Select Inbound rules in the left-hand pane:
2. Select New Rule in the right-hand pane, and select a Port rule as the type of rule to create:
3. For the Protocol and Ports, select TCP and port 22:
4. For the Action, select Allow the connection:
5. For the Profile, select all networks:
6. Give the rule the name SSH:
7. Verify that the inbound rule is created and enabled.

Configure the user account that Jenkins must use to sign in with to use SSH key-based authentication. There is no requirement to use key-based authentication, but it presents significant advantages in terms of security and is the recommended authentication mechanism when connecting agents using SSH.

Open a Cygwin64 Terminal and run the command ssh-keygen -t ecdsa (other key types are supported). Specifically, some plugins require that you use ssh-keygen -m pem -t ecdsa instead.

A 256-bit ECDSA key is stronger than a 2048-bit RSA key. This means that without applying the JCE Unlimited Strength Jurisdiction Policy Files, ECDSA keys are more secure than RSA keys as the JVM is limited to 2048-bit RSA keys.

$ ssh-keygen -t ecdsa
Generating public/private ecdsa key pair.
Enter file in which to save the key (/home/Jenkins/.ssh/id_ecdsa):
Created directory '/home/Jenkins/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/Jenkins/.ssh/id_ecdsa.
Your public key has been saved in /home/Jenkins/.ssh/id_ecdsa.pub.
The key fingerprint is:
SHA256:JfjsfW6pbmW/ENOe400/MIrmB1jxCE8Cy68/cUMoEik Jenkins@WIN-62JQBI8M63K
The key's randomart image is:
+---[ECDSA 256]---+
|    ...          |
| E o. .+ o       |
|  . .o. B =      |
|   . ..+ B . .   |
|    . ..S   o .  |
|      .+ =  o* . |
|     .  + =o+o* .|
|      .. o.=oo.=.|
|       .+++o. o.+|
+----[SHA256]-----+▼

Next change the working directory to ~/.ssh by running cd ~/.ssh.
Now create the authorized_keys by copying the public key that we have just generated: cp idecdsa.pub authorized_keys.
Extract the private key from the server. If the server is running as a virtual machine, this is typically just a question of displaying the public key on the screen using copy and paste to transfer to the host operating system. With a physical server it may be easier to use sftp to copy the file out.

Connect the Windows agent to Jenkins

Create the SSH key in Jenkins:
Configure the agent to use the SSH key credential:

If you select Manually trusted key Verification Strategy and leave Require manual verification of initial connection selected, Jenkins will refuse to connect until you approve the SSH Host key from the SSH Host Verification action. The SSH Host Verification action is only available after the first connection attempt.

Figure 1. SSH Host Verification
The agent should connect.