CloudBees CI on traditional platforms 2.414.3.7

Security advisories

CloudBees Security Advisory 2023-10-18

Security fixes

Upgraded multiple dependencies to address security concerns: Apache Ivy has been upgraded from 2.5.1 to 2.5.2 in the pipeline-groovy-lib plugin.

Guava has been upgraded from 31.0.1 to 32.1.2 and had j2objc-annotations:2.8 added in the google-storage-plugin.

Guava has been upgraded from 31.1-jre to 32.1.2-jre in the google-oauth-plugin.

Jetty IO has been upgraded from 10.0.13 to 10.0.16 where applicable.

Jetty XML has been upgraded from 10.0.13 to 10.0.16 where applicable.

commons-httpclient library vulnerability: The commons-httpclient library used in WebDAV and Azure stores (transitively through com.sun.jersey:jersey.client) is vulnerable to man in the middle (MITM) attacks. The fix removes the commons-httpclient library and uses the native Java HttpClient.

Upgrade Notes

Controllers fail to start due to plugins moved from the plugin catalog to the CloudBees Assurance Program (CAP)

With the 2.414.3.7 release, the following plugins were added to CAP:

GitLab API (gitlab-api)
GitLab Branch Source (gitlab-branch-source)
Pipeline Maven Integration (pipeline-maven)
Pipeline Maven API (pipeline-maven-api)

If your CasC bundle has any of these plugins in the plugin catalog, they must be removed. Otherwise, your bundle becomes invalid and will not start.

With the 2.414.3.7 release, the following plugin was removed from CAP:

Maven Integration (maven-plugin)

If this plugin is already installed, it will continue to be installed. If you wish to install this plugin through CasC from the October release onward, it must be included in the CasC bundle’s plugin catalog and in the plugins.yaml file.

Operations center CloudBees Assurance Program plugin changes since 2.414.2.2

The following plugins have been removed from the Operations center CloudBees Assurance Program since 2.414.2.2:

Maven Integration plugin (maven-plugin)

Controller CloudBees Assurance Program plugin changes since 2.414.2.2

The following plugins have been added to the Controller CloudBees Assurance Program since 2.414.2.2:

GitLab API Plugin (gitlab-api)
GitLab Branch Source Plugin (gitlab-branch-source)
Jersey 2 API (jersey2-api)
Pipeline Maven Integration Plugin (pipeline-maven)
Pipeline Maven Plugin API (pipeline-maven-api)

The following plugins have been removed from the Controller CloudBees Assurance Program since 2.414.2.2:

Maven Integration plugin (maven-plugin)

Jenkins 2.414 upgrade notes

New Features

Add GitLab support in the CloudBees Assurance Program (CAP): The GitLab Branch Source plugin and the GitLab API plugin were added to the CloudBees Assurance Program (CAP). These plugins enable users to have a supported way of creating multibranch and organization folder jobs backed by a GitLab server. These plugins will now receive necessary updates and security fixes under the CloudBees Assurance Program (CAP).

Bundle Update Timing is now available

The new CloudBees CI feature Bundle Update Timing is now available. With this new feature, instances using Configuration as Code can be configured to:

Automatically reload a new bundle version (If the bundle can be hot reloaded, the current restrictions remain.).
Automatically schedule a safe restart after the new bundle is detected.
Skip any new version detected. It is also possible to skip the versions when the instance is up and running and later apply the bundle during a restart.
Skip a concrete bundle version so it is never applied.
Reject or accept bundle versions with warnings, leaving the acceptance threshold to administrator users (administrator users can configure the instance).

Bundle Update Timing is enabled by default. However, if desired, you can disable Bundle Update Timing by setting the system property -Dcore.casc.bundle.update.timing.enabled=false in the start script.

For more information refer to Bundle update timing for the operations center or Bundle update timing for controllers.

Adding and removing Maven-related plugins from CAP: The Maven Integration (maven-plugin) plugin was removed from the CloudBees Assurance Program (CAP). As an alternative to this plugin, CloudBees recommends the use of the Pipeline Maven Integration plugin (pipeline-maven).

The Pipeline Maven plugin (pipeline-maven) and Pipeline Maven API (pipeline-maven-api) were added to CAP. See the upgrade notes for more details.

The CloudBees Maven Migration Assistant plugin (cloudbees-maven-migration-assistant), a plugin to help you migrate your Maven jobs to Pipeline jobs, is available from the CloudBees Update Center.

Bulk migration Groovy scripts will soon be available to ease the migration of Maven jobs.

Configuration as Code Bundle Report in the Operations center CasC Management Page: New ability to view a report of all of your controllers and their associated bundles. The report is now available on the Operations center CasC management page. The user can search for controllers as well as search and filter the bundles.

CasC Items support for GitLab organization folders and multibranch projects: CasC Items plugin now supports GitLab organization folders and multibranch projects.

Feature Enhancements

Display queue size in the High Availability (HA) summary: Now, the queue size on each cluster member is displayed in the High Availability (HA) summary.

Display the load per replica summary in the High Availability (HA) diagnostic page: The High Availability (HA) diagnostic page now displays a list of replicas and some load information, such as running builds and online agents.

Improve texts on the configuration page for controller items: Adapted the label and inline help texts for the Configuration as Code property on the configuration page of controller items.

Load-aware build adoption in High Availability (HA): Previously, when a High Availability (HA) controller terminated, its builds were adopted at random by other replicas. Now, the adoption is preferentially directed toward replicas that currently have the least load according to a simple metric.

Base64 support for secret masking: Base64 support for secret masking was added in this release. Now when users configuring jobs use any type of credentials, the Base64 version of their secret is added to the list of patterns that the application masks. This should reduce the number of leaks when the verbose mode of curl (or an equivalent mechanism) is used. However, there are limits to credentials masking. Refer to Limitations of Credentials Masking for more information.

Resolved Issues

YAML section in Pod Template not recognized

Starting with Kubernetes plugin (version 4029.v5712230ccb_f8) the YAML parsing library was updated to snakeyaml-engine:2.5 which implements YAML specification 1.2.2. In this specification, the YAML merge operator (<<) was removed. Therefore, any usage of the merge operator in pod definitions needs to be removed.

An example of a pod definition using YAML anchors and aliases instead of the merge operator:

apiVersion: v1
kind: Pod
spec:
  containers:
  - name: maven
    image: 3.9.4-eclipse-temurin-11
    resources:
      requests: &maven-container-resources
        cpu: 1500m
        memory: 3500Mi
      limits:
      *maven-container-resources
    command:
    - sleep
    args:
    - infinity▼

An example of the result of YAML specification 1.2.2 on a pod definition using anchors and aliases instead of the merge operator:

apiVersion: v1
kind: Pod
spec:
  containers:
    - name: maven
      image: 3.9.4-eclipse-temurin-11
      resources:
        requests:
          cpu: 1500m
          memory: 3500Mi
        limits:
          cpu: 1500m
          memory: 3500Mi
      command:
        - sleep
      args:
        - infinity▼

For more information, refer to YAML version 1.2.

When CloudBees Pipeline Explorer was enabled, some pages were very slow for complex Pipelines, leading to controller-wide performance issues in some cases: When CloudBees Pipeline Explorer was enabled, some pages, such as Blue Ocean and the Pipeline Steps view, were very slow for complex Pipelines. This can lead to high CPU usage on the controller, causing controller-wide performance issues. CloudBees Pipeline Explorer plugin now avoids performance-intensive computation in APIs that are used to access all of the Pipeline steps.

Older versions of controllers cannot start after receiving new bundle versions: All operations center supporting Branch and Tag for controller CasC bundles cannot send new bundle versions to the controllers older than 2.401.1.3. When those controllers receive a new bundle version, they cannot start.

Now, bundles sent to controller versions older than 2.401.1.3 are accepted.

Some invalid fields were not found during Configuration as Code bundle validation: Now a warning message is generated if a nonexistent or invalid field is found during validation.

Credentials cache not working when controller disconnected from operations center: When a controller disconnected from the operations center, yet the operations center was still running, attempts to look up credentials from the operations center would fail rather than falling back to a cached list as expected.

This issue is resolved.

Cluster Ops with backup controller on operations center is broken: Cluster Ops backup on operations center is failing because certain classes were not found during the backup cluster operation.

Cluster Ops with backup controller now functions correctly, but it requires the CloudBees Backup plugin (plugin ID: infradna-backup) to be installed on the controller.

Custom Update Center signature check uses an outdated certificate: The Update Center certificate was updated recently, and that modification needs to be taken care of for custom Update Center signature checks. The Custom Update Center signature now uses the updated Update Center certificate.

Updated expired embedded certificate of the Update Center: An expired certificate was previously embedded in the product, causing the download of some tool installers and plugins to fail. The certificate has been updated.

Operations center startup blocked by Update Center item loading: The loading of Update Center items is slowed down by the sha256 and sha512 calculation of the stored plugin and core binaries. This can delay the startup of the operations center for several minutes for Update Center items with a large plugins base.

Checksums are now saved during download and manual uploads, and during loading only if the checksums file does not exist.

External groups not propagated from the Reverse Proxy Auth plugin used with the operations center SSO: When the operations center SSO is used and the operations center is configured with an SSO-type security realm that only defines a user’s external groups within the context of a browser request, a connected controller is authenticated with the correct user ID, but none of the groups. This issue has been resolved.

Rename the CasC browser label gitLab to gitLabBrowser: The CasC browser label gitLab is used by both the git-plugin and gitlab-branch-source plugins. The label used by gitlab-branch-source has now been changed to gitLabBrowser.

Optimized handling of job next build number in HA controllers: Synchronizing the next build number of jobs can be a bottleneck in HA controllers on slow network filesystems. This process is now improved and the issue has been resolved.

More reliable cross-replica build loading for High Availability (HA) controllers: Previously, High Availability (HA) controller replicas could partially load build records even when the builds were in progress on other controllers, leading to errors. Now, a replica only loads a build record when the build is complete or when it is adopting a build in progress.

Fix a logging thread contention when the client controller calls the Operations center.: In some cases, logging call errors to the Operations center could cause thread contention. Rate limiting has now been applied to these use cases to prevent logging from causing thread contention.

UncheckedIOException related to bundle update can stop startup: File corruption in the bundle log directory can stop a controller startup, instead of a smooth recovery. This issue has been resolved.

Excessive HTTP client threads created: Certain operations, such as contacting the operations center for user lookup, could create excessive Java threads. This issue has now been resolved.

Retain queue ID when adopting queue items in High Availability (HA): If you schedule a build using a High Availability (HA) controller, but it does not start before the replica shuts down, the corresponding adopted queue item in another replica now uses the same ID. This allows external tools to track the build when it does start (or the item is cancelled).

Incompatible HA agent administrative monitor is displayed even if empty: In some cases, the administrative monitor that displays incompatible High Availability (HA) agents can be displayed, even when an incompatible agent is not found. This issue has been resolved and the administrative monitor is no longer displayed.

Log recorders synchronization does not work for creation and deletion: The log recorders are now correctly synchronized between replicas of a controller in High Availability (HA) mode.

CasC operations center RBAC "Granted at" offset is not honored when a group is created or updated: "Granted at" is now properly honored.

Known Issues

Failed parsing of data in the User Activity Monitoring plugin leads to incomplete data: Failed parsing of data from the User Activity Monitoring plugin will overwrite the user activity database. All user activity data that is logged up to that point in time is lost, in order to avoid this, refer to this knowledge base article Why is my user activity missing?.

HTTP Client used for Operations Center to Controllers connection leads to performance issues: Because of known issues in the Java HTTP Client, there could be performance issues in Operations Center to Controllers interactions in heavily loaded environments.

More details about this issue and workarounds are documented in Operations Center Client leaks HTTP Clients since version 2.401.1.3.

Error when renaming an existing EC2 cloud: When the name of an existing cloud node is updated, the user receives a 404 error after selecting save. This is because the cloud page uses the cloud name as part of its URL. When the user saves the name, Jenkins sends the user to the URL with the old cloud name. Please note that all node changes are successfully saved.

Clouds do not disappear after the Folder configuration update by a user without Overall/Administer permissions: Clouds deselect after a user without Overall/Administer permissions edits the Folder configuration.

Duplicate Pipeline Template Catalogs in the Configuration as Code for controllers jenkins.yaml file on each instance restart: If a Pipeline Template Catalog is configured in the Configuration as Code jenkins.yaml file and the id property is not defined, the catalog is duplicated on each instance restart and in the exported Configuration as Code configuration.

Duplicate Plugins in Operations center Plugin Manager UI: When searching for a specific plugin under 'Available' tab in operations center Plugin Manager, the search results shows duplicate entries of the searched plugin.