SSH Credentials Management with Jenkins

Article ID:222838288
5 minute readKnowledge base

Issue

  • Users need to use SSH key pairs to access remote server from master/nodes. I am currently provisioning my nodes so that they can connect to remote servers…​

  • How to best manage these credentials?

Environment

  • CloudBees Jenkins Enterprise

  • Jenkins

  • Credentials Plugin

  • CloudBees Folders Plugin

  • SSH Agent Plugin

Resolution

Jenkins provides a set of tools to manage your Credentials: Credentials Plugin, Folders Plus plugin and for CloudBees CI there is Restricting access and delegating administration with Role-Based Access Control.

Concept

Fundamentally you have to give Jenkins the credentials required to build your jobs and connect your build agents. But you do not need to give the credentials to the root context object. Jenkins in combination with the Credentials Plugin has the concept of contextualized credentials stores.

A good way to understand this is if you use Folders. Each Folder has its own credentials store. If each team/project has their own folder, then any credentials defined in the folder are only available to jobs within the folder.

This concept also applies to CloudBees Operations Center. With CJOC, you can give each team/project their own CJE. Then any credentials defined in that CJE are restricted to that CJE. Credentials available in CJOC are available to all CJEs.

Contextualized credentials stores is the key concept for Credentials Management in Jenkins.

Security

Compartmentalisation

The Folders Plus plugin enables administrators to organize their instance into a Folder hierarchy.

For customers who require a higher level of compartmentalisation and control, the job administrator should configure SSH keys at a Folder level.

This will help mitigate risks of the SSH private-key being compromised intentionally or unintentionally by a job administrator.

Access

A suitable Authorization Strategy such as the one provided by the Role-Based Access Control plugin should be used to manage users/groups access to Folders.

Passphrase

When generating your SSH key pairs, it is recommended to protect your SSH private-key with a passphrase.

Due to how the Credentials plugin works, your SSH private key can be copied back out of the credential configuration - while the passphrase can not. This provides an extra layer of protection.

It is possible for a suitably motivated Jenkins administrator to recover the passphrase.

Scopes

Next we have the notion of credential scopes for credentials defined in the root of Jenkins:

  • SYSTEM: This credential is only available to the object on which the credential is associated. Typically you would use system-scoped credentials for things like email auth, agent connection, etc, i.e. where the Jenkins instance itself is using the credential. Unlike the global scope, this significantly restricts where the credential can be used, thereby providing a higher degree of confidentiality to the credential.

  • GLOBAL: This credential is available to the object on which the credential is associated and all objects that are children of that object. Typically you would use global-scoped credentials for things that are needed by jobs.

This adds a degree of confidentiality for build agents. It is recommended to use the SYSTEM scope for Credentials used to connect Nodes.

SSH Agent

The SSH Agent plugin enables you to inject credentials - SSH private keys - into the build jobs using an SSH Agent. The build can run on any node, the Jenkins controller will provide it with the set of credentials. This allows the user to securely provide access to remote resources (using SSH authentication) without exposing the SSH private-key material to the job.

Without this plugin, if you want to use a node to connect to a remote server via SSH you would have to provision that node with SSH key required.

Notes:

  1. The SSH Agent socket is unique to each job so it should be mostly safe from parallel jobs running on the same node. For extra security you should ensure that each build agent is 1 executor only.

  2. While the SSH private-key content is not directly exposed to the job, this key (via the agent) can be used to authenticate to any remote resource while the agent connection is open (i.e. the job is running)

  3. To ssh successfully, SSH Agents needs the public key (matching with the SSH private keys which injects) included into the `~/.ssh/authorized_keys ` in the remote server to connect.

Use case example: How to use the SSH Agent Plugin

Best Practice

To sum up, here are recommend practice to best manage your Credentials in Jenkins:

  • Passphrase protect your SSH Keys

  • For Nodes, use scope SYSTEM so that those credentials cannot be used by build jobs

  • Compartmentalise your instance into Folders. Give each team/project their own folder. Give each folder the Credentials that those folders require - those credentials cannot be used by other "team" folders (unless those folders are sub-folders)

  • Use an authorization strategy such as Role-Based Access Control Plugin to restrict access to the folders

  • Configure jobs with SSH Agent to inject the SSH private-key from the per-folder credentials store into the build jobs

Example

We have 2 teams TeamA and TeamB:

  • ProjectA of TeamA is meant to be deployed to Server1

  • ProjectB of TeamB is meant to be deployed to Server2

We have 2 SSH nodes already connected Linux1 and Linux2. Both nodes have the label linux.

Following are 2 different approaches/architectures to address this use-case.

Architecture with Key Provisioning

If we want to manage SSH credentials outside Jenkins, we need to configure Nodes/Servers:

  • Linux1 contains the SSH private key (projectA_rsa) to access to Server1

  • Linux2 contains the SSH private key (projectB_rsa) to access to Server2

  • Server1 accepts SSH connection corresponding to projectA_rsa.pub

  • Server2 accepts SSH connection corresponding to projectB_rsa.pub

cred provisioning 1

There are problems with this architecture: Linux1 is dedicated to Server1. If I want to deploy to Server1, I have no choice but to use Linux1 or provision my other nodes and Server with SSH Keys.

Now if for example the requirements change: TeamB needs to access and deploy to Server1. To solve this I would create a new connection between Server1 and Linux2 by either using the existing public key linux2_rsa.pub or create a new key pair.

cred provisioning 2

At scale this becomes hardly manageable. I end up with many nodes having access to many servers. This results in a highly coupled architecture with tons of SSH key pairs that I can neither control nor trace.

cred provisioning 3

With this solution:

  • My nodes are inefficiently used

  • My nodes are not fungible: the same job cannot run on different nodes and therefore my label is useless

  • Adding new nodes would requires lots of configuration: SSH keys generation + key provisioning

Architecture with Credential Management

If we want to manage SSH credentials inside Jenkins, we need to configure Jenkins/Jobs:

  • Global Store contains SSH Credentials for Server1(projectA_rsa)

  • Global Store contains SSH Credentials for Server2(projectB_rsa)

  • ProjectA configured with SSH Agent to inject Server1 credentials

  • ProjectB configured with SSH Agent to inject Server2 credentials

  • Server1 accepts SSH connection corresponding to projectA_rsa.pub

  • Server2 accepts SSH connection corresponding to projectB_rsa.pub

cred management 1

With such configuration, ProjectA and ProjectB can be run on any node. Credentials are injected via SSH Agent plugin to the build node.

cred management ssh agent 1

Now if for example the requirements change: TeamB needs to access and deploy to Server1. To fulfil that requirement, no configuration is required on the node or the server but only on the ProjectB configuration. The Server2 credentials are available in the Global store. Therefore I only need to inject it using SSH Agent:

cred management ssh agent 2

Still, the same job can run on any node.

cred management 2
If I want to add more nodes to that label, no special configuration is needed so that ProjectA and ProjectB can be run on them.

Architecture with Credential Management in Folders

In previous example, credentials are stored globally and anyone can access it and use it in Jenkins. A way to secure your credentials is to use the Folders Plus plugin and Restricting access and delegating administration with Role-Based Access Control:

  • organize Jenkins teams/projects into Folders

  • define team/project credentials inside Folder Credentials stores

  • restrict Folder access using RBAC

cred management 3

More advanced management can be achieved using for example Credentials Domains and/or Controlled agents.

Information and Links