CloudBees CI release highlights

What’s new in CloudBees CI 2.414.3.7

Watch video

Security fixes

Upgraded multiple dependencies to address security concerns

Apache Ivy has been upgraded from 2.5.1 to 2.5.2 in the pipeline-groovy-lib plugin.

Guava has been upgraded from 31.0.1 to 32.1.2 and had j2objc-annotations:2.8 added in the google-storage-plugin.

Guava has been upgraded from 31.1-jre to 32.1.2-jre in the google-oauth-plugin.

Jetty IO has been upgraded from 10.0.13 to 10.0.16 where applicable.

Jetty XML has been upgraded from 10.0.13 to 10.0.16 where applicable.

commons-httpclient library vulnerability

The commons-httpclient library used in WebDAV and Azure stores (transitively through com.sun.jersey:jersey.client) is vulnerable to man in the middle (MITM) attacks. The fix removes the commons-httpclient library and uses the native Java HttpClient.

New Features

Add GitLab support in the CloudBees Assurance Program (CAP)

The GitLab Branch Source plugin and the GitLab API plugin were added to the CloudBees Assurance Program (CAP). These plugins enable users to have a supported way of creating multibranch and organization folder jobs backed by a GitLab server. These plugins will now receive necessary updates and security fixes under the CloudBees Assurance Program (CAP).

Bundle Update Timing is now available

The new CloudBees CI feature Bundle Update Timing is now available. With this new feature, instances using Configuration as Code can be configured to:

  • Automatically reload a new bundle version (If the bundle can be hot reloaded, the current restrictions remain.).

  • Automatically schedule a safe restart after the new bundle is detected.

  • Skip any new version detected. It is also possible to skip the versions when the instance is up and running and later apply the bundle during a restart.

  • Skip a concrete bundle version so it is never applied.

  • Reject or accept bundle versions with warnings, leaving the acceptance threshold to administrator users (administrator users can configure the instance).

Bundle Update Timing is enabled by default. However, if desired, you can disable Bundle Update Timing by setting the system property -Dcore.casc.bundle.update.timing.enabled=false in the start script.

Adding and removing Maven-related plugins from CAP

The Maven Integration (maven-plugin) plugin was removed from the CloudBees Assurance Program (CAP). As an alternative to this plugin, CloudBees recommends the use of the Pipeline Maven Integration plugin (pipeline-maven).

The Pipeline Maven plugin (pipeline-maven) and Pipeline Maven API (pipeline-maven-api) were added to CAP. See the upgrade notes for more details.

The CloudBees Maven Migration Assistant plugin (cloudbees-maven-migration-assistant), a plugin to help you migrate your Maven jobs to Pipeline jobs, is available from the CloudBees Update Center.

Bulk migration Groovy scripts will soon be available to ease the migration of Maven jobs.

Configuration as Code Operations center Bundle Retriever for SCM

The CasC Operations center Bundle Retriever is a service that ensures Operations center CasC bundles are always available and updated before the Operations center starts. The CasC Bundle Retriever is deployed alongside the Operations center container and allows you to manage it’s configuration via source control management (SCM) while ensuring that the latest bundle version is always locally available during both initialization and while running.

The Bundle Retriever uses a polling service to check the SCM by default. If GitHub is configured as the SCM then the Bundle Retriever can be configured to use webhooks for push notifications.

The Bundle Retriever can also be used in combination with Bundle Update Timing to optimize your Operations center CasC bundle management.

For more information refer to Retrieving bundles using SCM.

Configuration as Code Bundle Report in the Operations center CasC Management Page

New ability to view a report of all of your controllers and their associated bundles. The report is now available on the Operations center CasC management page. The user can search for controllers as well as search and filter the bundles.

CasC Items support for GitLab organization folders and multibranch projects

CasC Items plugin now supports GitLab organization folders and multibranch projects.

Feature Enhancements

Display queue size in the High Availability (HA) summary

Now, the queue size on each cluster member is displayed in the High Availability (HA) summary.

Display the load per replica summary in the High Availability (HA) diagnostic page

The High Availability (HA) diagnostic page now displays a list of replicas and some load information, such as running builds and online agents.

Include GitHub-related header values in the hibernation monitor logs

The pod logs now report the values of selected headers, without personal information, that can be correlated with webhook delivery logs in GitHub.

Windows agent container images for CloudBees CI

Agent container images for CloudBees CI based on Windows Server Core and Nano Server LTSC2019 and LTSC2022 are now available. These images must be used if you’re using the new High Availability mode for CloudBees CI. For more information, refer to High Availability mode.

When running a controller in High Availability (HA) mode, provision a pod disruption budget

When running a controller in replicated mode, set up a PodDisruptionBudget to ensure that the service is always running, even in maintenance events (node replacements). Use a pod disruption budget to ensure at least one replica is always up when scaling down node pools.

Add network policy support

Network policy support for the Hazelcast port has been added when a controller uses High Availability (HA) mode.

Improve texts on the configuration page for controller items

Adapted the label and inline help texts for the Configuration as Code property on the configuration page of controller items.

Omit on controllers in High Availability (HA) mode

When using HA mode, you can evict replicas when you scale down a node pool as it does not cause the loss of availability.

Load-aware build adoption in High Availability (HA)

Previously, when a High Availability (HA) controller terminated, its builds were adopted at random by other replicas. Now, the adoption is preferentially directed toward replicas that currently have the least load according to a simple metric.

Base64 support for secret masking

Base64 support for secret masking was added in this release. Now when users configuring jobs use any type of credentials, the Base64 version of their secret is added to the list of patterns that the application masks. This should reduce the number of leaks when the verbose mode of curl (or an equivalent mechanism) is used. However, there are limits to credentials masking. Refer to Limitations of Credentials Masking for more information.

Resolved Issues

YAML section in Pod Template not recognized

Starting with Kubernetes plugin (version 4029.v5712230ccb_f8) the YAML parsing library was updated to snakeyaml-engine:2.5 which implements YAML specification 1.2.2. In this specification, the YAML merge operator (<<) was removed. Therefore, any usage of the merge operator in pod definitions needs to be removed.

An example of a pod definition using YAML anchors and aliases instead of the merge operator:

apiVersion: v1 kind: Pod spec: containers: - name: maven image: 3.9.4-eclipse-temurin-11 resources: requests: &maven-container-resources cpu: 1500m memory: 3500Mi limits: *maven-container-resources command: - sleep args: - infinity

An example of the result of YAML specification 1.2.2 on a pod definition using anchors and aliases instead of the merge operator:

apiVersion: v1 kind: Pod spec: containers: - name: maven image: 3.9.4-eclipse-temurin-11 resources: requests: cpu: 1500m memory: 3500Mi limits: cpu: 1500m memory: 3500Mi command: - sleep args: - infinity

For more information, refer to YAML version 1.2.

When CloudBees Pipeline Explorer was enabled, some pages were very slow for complex Pipelines, leading to controller-wide performance issues in some cases

When CloudBees Pipeline Explorer was enabled, some pages, such as Blue Ocean and the Pipeline Steps view, were very slow for complex Pipelines. This can lead to high CPU usage on the controller, causing controller-wide performance issues. CloudBees Pipeline Explorer plugin now avoids performance-intensive computation in APIs that are used to access all of the Pipeline steps.

Older versions of controllers cannot start after receiving new bundle versions

All operations center supporting Branch and Tag for controller CasC bundles cannot send new bundle versions to the controllers older than 2.401.1.3. When those controllers receive a new bundle version, they cannot start.

Now, bundles sent to controller versions older than 2.401.1.3 are accepted.

Some invalid fields were not found during Configuration as Code bundle validation

Now a warning message is generated if a nonexistent or invalid field is found during validation.

Credentials cache not working when controller disconnected from operations center

When a controller disconnected from the operations center, yet the operations center was still running, attempts to look up credentials from the operations center would fail rather than falling back to a cached list as expected.

This issue is resolved.

Cluster Ops with backup controller on operations center is broken

Cluster Ops backup on operations center is failing because certain classes were not found during the backup cluster operation.

Cluster Ops with backup controller now functions correctly, but it requires the CloudBees Backup plugin (plugin ID: infradna-backup) to be installed on the controller.

Send a warning if the managed-master-hibernation plugin is installed on a High Availability (HA) controller

Added an administrative monitor warning to administrators who run an High Availability (HA) controller that the managed-master-hibernation plugin should be uninstalled as it is incompatible with it.

Custom Update Center signature check uses an outdated certificate

The Update Center certificate was updated recently, and that modification needs to be taken care of for custom Update Center signature checks. The Custom Update Center signature now uses the updated Update Center certificate.

New health check endpoint

Previously, the health check endpoint for the operations center pods used the /login URL, that creates a new Jetty session each time. Now, a new endpoint has been selected that uses an anonymous REST API that avoids session creation. (The analogous change was made for managed controllers in a previous release.)

Updated expired embedded certificate of the Update Center

An expired certificate was previously embedded in the product, causing the download of some tool installers and plugins to fail. The certificate has been updated.

Operations center startup blocked by Update Center item loading

The loading of Update Center items is slowed down by the sha256 and sha512 calculation of the stored plugin and core binaries. This can delay the startup of the operations center for several minutes for Update Center items with a large plugins base.

Checksums are now saved during download and manual uploads, and during loading only if the checksums file does not exist.

Managed controller YAML validation section remains empty

Since prototype.js usage was removed, the YAML validation section remains empty when creating managed controllers.

The YAML validation section now properly contains the necessary data.

External groups not propagated from the Reverse Proxy Auth plugin used with the operations center SSO

When the operations center SSO is used and the operations center is configured with an SSO-type security realm that only defines a user’s external groups within the context of a browser request, a connected controller is authenticated with the correct user ID, but none of the groups. This issue has been resolved.

Rename the CasC browser label gitLab to gitLabBrowser

The CasC browser label gitLab is used by both the git-plugin and gitlab-branch-source plugins. The label used by gitlab-branch-source has now been changed to gitLabBrowser.

Optimized handling of job next build number in HA controllers

Synchronizing the next build number of jobs can be a bottleneck in HA controllers on slow network filesystems. This process is now improved and the issue has been resolved.

More reliable cross-replica build loading for High Availability (HA) controllers

Previously, High Availability (HA) controller replicas could partially load build records even when the builds were in progress on other controllers, leading to errors. Now, a replica only loads a build record when the build is complete or when it is adopting a build in progress.

Fix a logging thread contention when the client controller calls the Operations center.

In some cases, logging call errors to the Operations center could cause thread contention. Rate limiting has now been applied to these use cases to prevent logging from causing thread contention.

UncheckedIOException related to bundle update can stop startup

File corruption in the bundle log directory can stop a controller startup, instead of a smooth recovery. This issue has been resolved.

Excessive HTTP client threads created

Certain operations, such as contacting the operations center for user lookup, could create excessive Java threads. This issue has now been resolved.

Retain queue ID when adopting queue items in High Availability (HA)

If you schedule a build using a High Availability (HA) controller, but it does not start before the replica shuts down, the corresponding adopted queue item in another replica now uses the same ID. This allows external tools to track the build when it does start (or the item is cancelled).

Incompatible HA agent administrative monitor is displayed even if empty

In some cases, the administrative monitor that displays incompatible High Availability (HA) agents can be displayed, even when an incompatible agent is not found. This issue has been resolved and the administrative monitor is no longer displayed.

Log recorders synchronization does not work for creation and deletion

The log recorders are now correctly synchronized between replicas of a controller in High Availability (HA) mode.

CasC operations center RBAC "Granted at" offset is not honored when a group is created or updated

"Granted at" is now properly honored.

Known Issues

YAML file is not validated in the Default YAML field of the Configure Controller Provisioning page

On the Configure Controller Provisioning page, the YAML file is not validated when it is configured in the Default YAML field.

Auto-update bundle version may incorrectly update the wrong bundle version if multiple bundle.yaml files exist in the repository

When using "Auto-update bundle version", the wrong bundle may be updated if multiple bundle.yaml files exist in the repository.

Wrapped token file can not be deleted by the operations center

The .wrappedToken file is inadvertently being included in the bundle by the operations center bundle retriever and giving a structural warning, [STRUCTURE] - The following files have been detected in the bundle folder but they are not yaml files: .wrappedToken.

Add CloudBees CI communications to the CasC Bundle Retriever

When a bundle is retrieved from the SCM there are benign warnings in the logs about communication with the operations center.

Error when renaming an existing EC2 cloud

When the name of an existing cloud node is updated, the user receives a 404 error after selecting save. This is because the cloud page uses the cloud name as part of its URL. When the user saves the name, Jenkins sends the user to the URL with the old cloud name. Please note that all node changes are saved successfully.

Duplicate Pipeline Template Catalogs in the Configuration as Code (CasC) for Controllers jenkins.yaml file on each instance restart

If a Pipeline Template Catalog is configured in the CasC jenkins.yaml file and the id property is not defined, the catalog is duplicated on each instance restart and in the exported CasC configuration.

Upgrade Notes

Controllers fail to start due to plugins moved from the plugin catalog to the CloudBees Assurance Program (CAP)

With the 2.414.3.7 release, the following plugins were added to CAP:

  • GitLab API (gitlab-api)

  • GitLab Branch Source (gitlab-branch-source)

  • Pipeline Maven Integration (pipeline-maven)

  • Pipeline Maven API (pipeline-maven-api)

If your CasC bundle has any of these plugins in the plugin catalog, they must be removed. Otherwise, your bundle becomes invalid and will not start.

With the 2.414.3.7 release, the following plugin was removed from CAP:

  • Maven Integration (maven-plugin)

If this plugin is already installed, it will continue to be installed. If you wish to install this plugin through CasC from the October release onward, it must be included in the CasC bundle’s plugin catalog and in the plugins.yaml file.

Operations center CloudBees Assurance Program plugin changes since 2.414.2.2

The following plugins have been removed from the Operations center CloudBees Assurance Program since 2.414.2.2:

  • Maven Integration plugin (maven-plugin)

Controller CloudBees Assurance Program plugin changes since 2.414.2.2

The following plugins have been added to the Controller CloudBees Assurance Program since 2.414.2.2:

  • GitLab API Plugin (gitlab-api)

  • GitLab Branch Source Plugin (gitlab-branch-source)

  • Jersey 2 API (jersey2-api)

  • Pipeline Maven Integration Plugin (pipeline-maven)

  • Pipeline Maven Plugin API (pipeline-maven-api)

The following plugins have been removed from the Controller CloudBees Assurance Program since 2.414.2.2:

  • Maven Integration plugin (maven-plugin)