How to address the vulnerabilities in the CloudBees Security Advisory 2021-11-04

Article ID:4409188496155
11 minute readKnowledge base

Issue

  • On November 4th, 2021, CloudBees published a security advisory that includes vulnerabilities related to the Jenkins security issues mentioned in Jenkins Security Advisory 2021-11-04. As a mitigation to the different security issues exposed by the vulnerabilities mentioned above, I want to be able to apply the security hardening measures suggested to address:

    • Remoting incomplete/improper role checks.

    • Incomplete/improper file path filtering.

Remoting role check improvements

Jenkins agents use remoting library to communicate with the controller. Individual messages (implementations of the Callable interface) instruct the other side of the bidirectional communication channel to perform operations and/or provide information.

The implementations are expected to restrict where they can be executed by their implementation of a role check. This role check is run after receiving the object through the communication channel and is expected to reject the execution of the message (Callable) in case that it is on an unexpected side of the channel.

A missing implementation of this role check is responsible for BEE-8354, DEVOPTICS-1698 and BEE-8591.

The new releases 2.303.3.3 and 2.277.43.0.2 now require that role checks are performed by Callable implementations, and failing to do so will cause the implementation to stop working. Any plugin failing to work in any of the above mentioned releases will need to be updated.

Agent to controller file path filter security fixes

Jenkins agents use remoting library to communicate with the controller. For to compatibility reasons, these agents are allowed a limited access to files on the controller file system. The new releases 2.303.3.3 and 2.277.43.0.2 fix multiple vulnerabilities that resulted in incomplete access control enforcement.

Any plugins that access files on the controller from code running on agents might be affected by these security fixes. See the Potential side effects section.

Agent to controller security fix related to build directory access

Jenkins agents use remoting library to communicate with the controller. Due to compatibility reasons, these agents are allowed a limited access to files on the controller file system. The directories agents are allowed to access include the directories storing build-related information, intended to allow agents to store build-related metadata during build execution. As a consequence, this allowed any agent to read and write the contents of any build store in Jenkins with few restrictions. The new releases 2.303.3.3 and 2.277.43.0.2 fix this behavior and agents are only allowed to access contents of builds directories related to builds currently running on the agent trying to access the directory.

Environment

Document overview

In this document, we are going to discuss different approaches to get your system protected, as well as the side effects that the security hardenings and security fixes can have on your system depending on the path that you follow.

  • Option A: You might need to follow the steps described in the Resolution section. This is the recommended path to address these security vulnerabilities. In the Resolution section (in case that you upgrade your system) you will also find potential side effects of the fix, covered in the subsections below.

  • Option B: This path is described in the Workaround section, and it is only recommended in case that you cannot immediately upgrade all your instances. This path consists on the installation of different plugins to workaround the vulnerability. Following this option your system will be protected in the agent/controller, and in the controller/operations center connection, but you will not be protected in the controller/controller connection. However, the exploit of this vulnerability can only be done by administrators of the controllers. If you are concerned about controller administrators being able to exploit the vulnerability, you can get full protection by upgrading the operations center, and use the documented workaround to install the remoting-security-workaround in the controllers can be done (without upgrading controllers). Please contact CloudBees support if you plan to upgrade, so we can prepare an assisted update.

With this option, the side effects that you can observe are the same as in Option A (you upgraded your system), as well as the workarounds except for the sections explicitly mentioned. We also provide installation paths for this option depending on your configuration and potential remoting-security-workaround plugin configuration details.

Resolution

The solution to these vulnerabilities is to upgrade your instance to the versions listed in the security release depending on which version you are currently using, as the security fixes are going to be delivered as:

Potential side effects

Remoting incomplete/improper role checks

Potential side effect: This security hardening would result in some plugins implementing role checks of file path filtering in a non secure way to fail. The security improvement detects when a plugin implementing an incomplete/improper role check attempts to send a Callable through the remoting channel from the agent to the controller and will throw a SecurityException causing a log message like the one shown below:

Security hardening prohibits the Callable implementation <failing_implementation_will_go_here>
from ignoring RoleChecker, see https://www.jenkins.io/redirect/required-role-check/

Workaround: This security hardening can be adjusted using the following system properties:

  • hudson.remoting.ChannelBuilder.specificCallablesCanIgnoreRoleChecker that can be set to contain a list of class names allowed to bypass the hardening. Our recommendation is that if your build fails, you should contact support@cloudbees.com to check whether allowlisting that class is advised or not.

  • hudson.remoting.ChannelBuilder.allCallablesCanIgnoreRoleChecker setting this property to true will completely ignore the security hardening, and essentially will dismiss the security fix. It is NOT recommended.

Agent to controller file path filter security improvements

Potential side effect: Some functionality may be newly subject to access control checks, but not allowed because paths outside the default allowlist are accessed. This can result in (partially) breaking builds. In particular, maven-plugin functionality (archiving Javadoc or sites) is likely to be affected.

If access to a file is rejected, an exception with a message similar to the one shown below will be visible in the agent logs, and also will show up in the build logs:

Agent may not access a file path. See the system log for more detais about the error ID
'<UUID here>' and https://www.jenkins.io/redirect/security-144 for more information.

As of 2021-11-04, no plugins are known to be affected by this security fix.

  • Workaround: As part of the SECURITY-144 fix, an admin-customizable allowlist of paths is provided. This list can (carefully) be extended to allow the prohibited-by-default actions if deemed safe. Documentation: Customizing Agent → Controller Security.

Agent to controller security fix related to build directory access

Potential side effect: If the Jenkins home directory is split using symbolic links (e.g. /var/jenkins_home/jobs -> /mnt/giant_disk/jenkins-jobs) or the builds directories are in a nonstandard location (jenkins.model.Jenkins.buildsDir system property), the default allowlist rules will not match, as they assume a default Jenkins home layout. Nonstandard builds directories have always had this problem (JENKINS-31554), but now more actions are subject to this protection.

  • Workaround: Restore a default Jenkins home directory layout, or customize the rules as above.

Potential side effect: Some job types, in particular Pipeline, may not support the new filtering of build directories that limits access to the agent on which the build is currently running.

  • Solution: Upgrade workflow-durable-task-step plugin to 2.38.1 or 2.40.

  • Workaround: Set the system property jenkins.security.s2m.RunningBuildFilePathFilter.FAIL to false to only log a warning, or set the system property jenkins.security.s2m.RunningBuildFilePathFilter.SKIP to true to skip this entirely. Either of these approaches remove the security hardening related to file path filtering, so these approaches are not recommended.

Operations center context security hardening

Potential side effect: As a potential side effect of the security hardening included in the newer versions for operations-center-context and operations-center-server plugins, you might see the following message in your logs:

RPCRequest blocked. Target controller does not have the necessary capabilities to check Callable security

This means that the receiving controller for an RPCRequest (used for used for Controller to Controller communication, e.g. for MCP) does not have the necessary code to properly check if the call is secure (ie. it does not have the required version of operations-center-context). This will not happen on every single call as there is an attempt to make a security check directly at operations center level directly without asking explicitly to the target controller. If this check fails, then the remote check is needed.

Workaround: In order to get rid of the error shown above, you might want to skip that check. This will bring your instance to a less secure state, but will help the operation to complete. This is very unsafe to do. You can set the system property: com.cloudbees.opscenter.server.messaging.Transport.allowInsecureMode to true using the Script Console in your operations center.

Workaround

In case that you cannot upgrade your instance immediately, but want to address these vulnerabilities, CloudBees provides a workaround consisting of two new plugins:

A new plugin, remoting-security-workaround, is provided to implement the security hardening even on older instances without the fixes in Jenkins core and affected plugins. With this hardening, the specific plugin security fixes aren’t needed (as they will not be exploitable). Additionally, it blocks all file path access.

Another new plugin, cloudbees-remoting-security-workaround, extends this to cover controller/operations center connections as well. However, this plugin does not cover the controller/controller connections, thus an administrator in a Controller could still be able to perform operations in different Controllers where they are not an administrator. In case you don’t trust all the administrator users defined in the Controllers, then CloudBees recommends upgrading the Operations Center to a version with the fix included in cases where your Controllers are running a supported version instead of installing the cloudbees-remoting-security-workaround. In this way, you will be fully protected.

Important note In a previous version of this KB article it was adviced to update operations-center-context and operations-center-server (if updates available in the Update center) , but this was meant to be used only in case the upgrade is offered through CAP or an incremental update. Currently, this is not an option as no incremental release has been published yet.

Additionally, you will also need to install cloudbees-remoting-security-workaround in the Operations Center. In case you already opted for this option, then you are fully protected against the vulnerability in the controller/controller, and operations center/controller connections. The main problem with this option is that CloudBees can’t guarantee that all the Operations Center functionalities will continue working correctly.

CloudBees recommends that all customers perform the steps in this article as soon as possible, to ensure you are protected from the vulnerability. Then, once you have completed these steps, you can plan for and test an upgrade to the versions containing the fix by contacting CloudBees for an Assisted Update. Once you have upgraded to a version containing the mitigation, you might remove the plugins above.

Workaround Summary

For Jenkins (to protect from agents):

For connected or standalone CloudBees CI controllers and CloudBees CI operations center (to protect from agents):

  • Install remoting-security-workaround plugin and restart

    • If the Operations Center is going to be upgraded to a version with the fix in place, it is not necessary to install this plugin.

For CloudBees CI operations center (to protect from admins of connected controllers):

  • To be fully protected CloudBees recommends upgrading the Operations Center to the suggested versions in the Resolution section. If it is not possible to upgrade the Operations Center, then you can proceed with the below steps, but you will not be protected from the controller/controller connections. This means that administrators in a Controller could potentially perform operations in another Controller where they are not administrators. Only use this option if you fully trust all the administrators in your Controllers.

Installation Scenarios

Installing the mitigation on the operations center

1.- Download the cloudbees-remoting-security-workaround plugin locally as well as remoting-security-workaround as the first depends on the second one.

2.- Go to Manage Jenkins -> Manage Plugins.

3.- Select the Advanced tab.

4.- Under Upload Plugin, select Choose File and then select the saved plugin file (remoting-security-workaround.hpi).

5.- Click Upload.

6.- Under Upload Plugin, select Choose File and then select the saved plugin file (cloudbees-remoting-security-workaround.hpi).

7.- Click Upload.

8.- Restart the instance. The plugins do not require extra configuration.

Installing the mitigation on a stand alone instance

1.- Download the remoting-security-workaround plugin locally.

2.- Go to Manage Jenkins -> Manage Plugins.

3.- Select the Advanced tab.

4.- Under Upload Plugin, select Choose File and then select the saved plugin file (remoting-security-workaround.hpi).

5.- Click Upload.

6.- Restart the instance. The plugin do not require extra configuration.

Installing on controllers in an operations center cluster

In any clustered environment with an Operations Center and Controllers connected to it, you will need to apply the workaround to the CloudBees Operations Center (see Installing on an Operations Center), as well as to the connected controllers.

To install the remoting-security-workaround plugin on multiple controllers managed by a CloudBees operations center, you can use a Cluster Operation.

Note: Controllers must be Running and Connected for the Cluster Operation to be successful and correctly apply the mitigation.

To create and configure a Cluster Operation:

1.- On your operations center, select New Item and then select Cluster Operations.

2.- Enter a name for the Cluster Operation (such as Apply SECURITY-2456 to all online controllers).

3.- (Optional) You can configure the Cluster Operation with Build Triggers -> Build periodically to run it periodically. It is safe to run the Cluster Operation multiple times on the same controllers (it is idempotent).

4.- In the Target Managed controllers section, add the controllers on which you want to block URLs (such as From Operations Center Root).

5.- Under Filters, select Add Filter and then select Client Controller / Managed Controller Is Online.

6.- Add the following steps:

  1. An Install plugin step with the plugin ID set to remoting-security-workaround and no version (leave the Version field blank). Using no version instructs the system to use the most recent plugin available for the given controller.

7.- Select Save, and then Run.

Installing the workaround using CasC

Installing the workaround in operations center

CloudBees customers running CloudBees CI 2.289.3.2 and later, could benefit from CasC in the operations center to mitigate this security vulnerability.

The remoting-security-workaround plugin and cloudbees-remoting-security-workaround are not included in the CloudBees Assurance Program (CAP), therefore, the corresponding plugin should be included in the plugin-catalog.yaml configuration file (depending on whether you are in an Operations Center or not). However, by default, the plugin-catalog.yaml can’t be used in the operations center unless the Java argument -Dcom.cloudbees.jenkins.cjp.installmanager.CJPPluginManager.enablePluginCatalogInOC=true is added to the operations center startup arguments, this startup argument is not required for controllers.

Installing the workaround in controllers

For controllers, plugin catalog is available by default and it is not necessary to add any Java argument as it happens with the operations center.

Snippets for CasC

  • plugin-catalog.yaml

type: plugin-catalog
version: '1'
name: security-2456-catalog
displayName: Simple SECURITY-2456 catalog
configurations:
- description: Exported plugins
  includePlugins:
    remoting-security-workaround:
      version: ‘xx’

  • plugins.yaml

plugins:
  - {id: “remoting-security-workaround”}

Configuration options

The remoting-security-workaround plugin does not have an UI. It can be configured through the Groovy Script Console (Manage Jenkins->Script Console), or through different system properties.

Configuring the plugin using the Script Console

  • Add a new callable class name to the block list that will always reject it when sent to a controller from a lower-privileged endpoint.

io.jenkins.plugins.remoting_security.ConfigurableCallableBlocker.SPECIFIC_CALLABLES_TO_ALWAYS_REJECT.add('org.acme.Whatever$MyCallable')

  • Remove a callable class name from the block list (may have been in there by default).

io.jenkins.plugins.remoting_security.ConfigurableCallableBlocker.SPECIFIC_CALLABLES_TO_ALWAYS_REJECT.remove('org.acme.Whatever$MyCallable')

  • Disable requirement for callables to perform a (any) role check

io.jenkins.plugins.remoting_security.RequiredRoleCheck.CALLABLES_CAN_IGNORE_ROLECHECKER = true

  • Allow a specific callable to not need to perform a role check

io.jenkins.plugins.remoting_security.RequiredRoleCheck.SPECIFIC_CALLABLES_CAN_IGNORE_ROLECHECKER.add('org.acme.Whatever$MyCallable')

  • Stop allowing a specific callable to not need to perform a role check (may have been in there by default)

io.jenkins.plugins.remoting_security.RequiredRoleCheck.SPECIFIC_CALLABLES_CAN_IGNORE_ROLECHECKER.remove('org.acme.Whatever$MyCallable')

Configuring the plugin using the Java system properties

The properties to control the plugin configuration are listed below:

  • io.jenkins.plugins.remoting_security.ConfigurableCallableBlocker.additionalCallablesToAlwaysReject is a comma-separated list of class names to add to the block list. Instances of these types will always be rejected.

  • io.jenkins.plugins.remoting_security.RequiredRoleCheck.specificCallablesCanIgnoreRoleChecker is a comma-separated list of class names to add to the allow list. Instances of these types will not be required to perform a role check.

  • io.jenkins.plugins.remoting_security.RequiredRoleCheck.allCallablesCanIgnoreRoleChecker will disable the required role check entirely if set to true, so that callables not performing a role check won’t be rejected for this. This is very unsafe to do unless Jenkins has been updated to 2.319 or later, or LTS 2.303.3 or later

There is no way to customize the default allow list for the required role check. It should only contain safe types, so it is not expected to need customization.

We strongly recommend you to contact support@cloudbees.com for additional recommendations on specific classes.

Validating the workaround

remoting-workaround-plugin

To test that the protection is effective for a given agent, you can run the following in the script console:

io.jenkins.plugins.remoting_security.Tester.testChannelToAgent("my-agent-name-here")

If there is no output, the agent/controller connection has the additional protections in this plugin applied.

Otherwise, one of the following applies:

  • You customized the options for the plugin (see below)

  • The protections have not been applied. The only regular reason for this to happen is when Jenkins hasn’t been restarted after installing this plugin, and the connection was created before the plugin was installed.

cloudbees-remoting-workaround-plugin

To test that the protection is effective for a given controller, you can run the following in the operations center script console:

com.cloudbees.jenkins.security.workaround.Tester.testChannelToController("full/path/to/controller")

If there is no output, controller connection has the additional protections in this plugin applied.

Otherwise, one of the following applies:

  • You customized the options for the plugin (see below)

  • The protections have not been applied. The only regular reason for this to happen is when Jenkins hasn’t been restarted after installing this plugin, and the connection was created before the plugin was installed.

Affected Functionality

With remoting-security-workaround plugin installed, some Jenkins plugin functionality may not work any longer:

This includes everything affected by the SECURITY-2458 and SECURITY-2455 changes.

In addition to that, the following functionality is known to break because of this plugin:

Plugin Feature Workaround Fix

Maven Integration

Archiving Maven sites and Javadoc fails.

Do not archive Maven sites or Javadoc.

Update Maven plugin to 3.15.1 or install one of the backported versions depending on your current version. Available backports are: 3.12.1, 3.10.1, 3.8.1, 3.7.1 and 3.6.1

Code Coverage API

Saving of source files (optional feature) in post-build step fails

Do not use the feature

n/a

Cobertura

"Publish Cobertura Coverage Report" post-build step fails

n/a

n/a

Log Parser

Log parsing fails

n/a

n/a

References