When users need to use SSH key pairs to access a remote server from the controller or nodes, Jenkins provides a set of tools to manage the credentials: the Credentials Plugin, the Folders Plus plugin, and for CloudBees CI, there is Restricting access and delegating administration with Role-Based Access Control.
For example, a user has to give Jenkins the credentials required to build jobs and connect the build agents. However, giving the credentials to the root context object is not necessary. Jenkins in combination with the Credentials Plugin uses the concept of contextualized credentials stores.
This can be done using Folders. Each Folder has its own credentials store. If each team/project has their own folder, then any credentials defined in the folder are only available to jobs within the folder.
This concept also applies to the CloudBees Operations center. With the operations center, you can give each team/project their own controller. Then any credentials defined in that controller are restricted to that controller.
Credentials available in the CloudBees operations center can be defined at the global level to be available to all controllers or at a folder level to scope them to a specific set of controllers.
Contextualized credentials stores is the key concept for Credentials Management in Jenkins.
Security
Compartmentalization
The Folders Plus plugin enables administrators to organize their instance into a Folder hierarchy.
If a higher level of compartmentalization and control are required, the job administrator should configure SSH keys at the Folder level. This will help mitigate the risks of the SSH private-key being compromised intentionally or unintentionally by a job administrator.
Access
A suitable Authorization Strategy such as the one provided by the Role-Based Access Control plugin should be used to manage users/groups access to Folders.
Passphrase
When generating SSH key pairs, it is recommended to protect the SSH private-key with a passphrase.
Due to how the Credentials plugin works, the SSH private key can be copied back out of the credential configuration - while the passphrase cannot. This provides an extra layer of protection.
| It is possible for a suitably motivated Jenkins administrator to recover the passphrase. |
Scopes
Next is the notion of credential scopes for credentials defined in the root of Jenkins:
-
SYSTEM: This credential is only available to the object on which the credential is associated. Typically, system-scoped credentials are used for things like email authentication, agent connection, etc., for example, where the Jenkins instance itself is using the credential. Unlike the global scope, this significantly restricts where the credential can be used, thereby providing a higher degree of confidentiality to the credential.
-
GLOBAL: This credential is available to the object on which the credential is associated and all objects that are children of that object. Typically, you would use global-scoped credentials for things that jobs need.
Credential scopes add a degree of confidentiality for build agents. It is recommended to use the SYSTEM scope for credentials used to connect nodes.
Defining SSH credentials
The SSH Agent plugin enables users to inject credentials - SSH private keys - into the build jobs using an SSH Agent. The build can run on any node and the Jenkins controller provides it with a set of credentials. This allows the user to securely provide access to remote resources (using SSH authentication) without exposing the SSH private-key material to the job.
Without this plugin, to use a node to connect to a remote server via SSH, that node would have to be provisioned with an SSH key required.
Notes:
-
The SSH Agent socket is unique to each job, so it should be mostly safe from parallel jobs running on the same node. For extra security, ensure that each build agent is one executor only.
-
While the SSH private-key content is not directly exposed to the job, this key (via the agent) can be used to authenticate to any remote resource while the agent connection is open, (such as, the job is running).
-
To SSH successfully, SSH Agents need the public key (matching with the SSH private keys which injects) included into the
~/.ssh/authorized_keysin the remote server to connect.
CloudBees recommends the following practices to best manage credentials in Jenkins:
-
Use Passphrase to protect SSH keys.
-
For nodes, use scope SYSTEM so that those credentials cannot be used by build jobs.
-
Compartmentalise the instance into Folders. Give each team/project their own folder. Give each folder the credentials that those folders require - those credentials cannot be used by other "team" folders (unless those folders are subfolders).
-
Use an authorization strategy such as Role-Based Access Control Plugin to restrict access to the folders.
-
Configure jobs with SSH Agent to inject the SSH private-key from the per-folder credentials store into the build jobs.
Example
There are two teams: TeamA and TeamB:
-
ProjectA of TeamA is meant to be deployed to Server1
-
ProjectB of TeamB is meant to be deployed to Server2
There are two SSH nodes already connected to Linux1 and Linux2. Both nodes have the label linux.
Following are two different approaches/architectures to address this scenario.
Architecture with Key Provisioning
To manage SSH credentials outside Jenkins, configure Nodes/Servers this way:
-
Linux1 contains the SSH private key (projectA_rsa) to access to Server1.
-
Linux2 contains the SSH private key (projectB_rsa) to access to Server2.
-
Server1 accepts SSH connection corresponding to projectA_rsa.pub.
-
Server2 accepts SSH connection corresponding to projectB_rsa.pub.
The problem with this architecture is Linux1 is dedicated to Server1. To deploy to Server1, There’s no choice but to use Linux1 or provision the other nodes and Server with SSH Keys.
Now, for example, if the requirements change, ProjectB needs to access and deploy to Server1. Solve this by creating a new connection between Server1 and Linux2 by either using the existing public key linux2_rsa.pub or create a new key pair.
At scale, this becomes hardly manageable. There are too many nodes having access to many servers. This results in a highly coupled architecture with tons of SSH key pairs that can neither be controlled nor traced.
With this solution:
-
The nodes are inefficiently used.
-
The nodes are not fungible: the same job cannot run on different nodes, and therefore the label is useless.
-
Adding new nodes would require lots of configuration: SSH key generation and key provisioning.
Architecture with Credential Management
To manage SSH credentials inside Jenkins, configure Jenkins/Jobs:
-
Global Store contains SSH Credentials for Server1(projectA_rsa).
-
Global Store contains SSH Credentials for Server2(projectB_rsa).
-
ProjectA configured with SSH Agent to inject Server1 credentials.
-
ProjectB configured with SSH Agent to inject Server2 credentials.
-
Server1 accepts SSH connection corresponding to projectA_rsa.pub.
-
Server2 accepts SSH connection corresponding to projectB_rsa.pub.
With this configuration, ProjectA and ProjectB can run on any node. Credentials are injected via SSH Agent plugin to the build node.
sshagent(['CREDENTIAL_ID']) { ... }
Now, for example, if the requirements change: ProjectB needs to access and deploy to Server1. To fulfill that requirement, no configuration is required on the node or the server but only on the ProjectB configuration. The Server2 credentials are available in the Global store. The CREDENTIAL_ID in the Jenkinsfile can be updated.
Still, the same job can run on any node.
| To add more nodes to that label, no special configuration is needed so that ProjectA and ProjectB can run on them. |
Architecture with Credential Management in Folders
In the previous example, credentials are stored globally, and anyone can access it and use it in Jenkins. A way to secure credentials is to use the Folders Plus plugin and the steps on Restricting access and delegating administration with Role-Based Access Control. This allows users to:
-
organize Jenkins teams/projects into Folders.
-
define team/project credentials inside Folder Credentials stores.
-
restrict Folder access using RBAC.
More advanced management can be achieved using Credentials Domains and/or Controlled agents, for example.
Performing secure copy
Use the SSH Agent Plugin to perform a Secure copy (scp) from an agent to another server. Recent versions of the plugin (greater than 1.14) should work out of the box on Linux. However, on Windows, the SSH agent requires manual installation.
An alternative to using the SSH Agent plugin is to use the withCredentials step demonstrated in Jenkins plugin content alternatives.
|
Set up the credentials
Create a new credential type: SSH Username with private key
| You can select the method of your choice to enter the Private Key. In case you have trouble creating credentials, please refer to Injecting Secrets into Jenkins Build Jobs. |
Troubleshooting
-
ERROR: Failed to run ssh-add: indicates something was wrong with the SSH based operations.-
Run the same operation manually from the agent running the build.
-
Example:
scp /Users/exampleUser/home/aws/listDProcessesNativeStacks.sh ubuntu@ip-172-31-69-105.ec2.internal:/home/ubuntu
-
-
Add the private key to the ssh-agent manually.
-
Example in linux:
ssh-add ~/.ssh/my-ssh-key)
-
-
-
If using versions older than indicated below, refer to CloudBees Support for further assistance.