CloudBees CI release highlights

What’s new in CloudBees CI 2.426.1.2

Watch video

New Features

Add compatibility between the Remote Directory API and the High Availability (HA) controllers

The transport used by the Remote Directory API was changed from Remoting to HTTP to provide compatibility with the High Availability (HA) controllers. This can be turned off using -Dcom.cloudbees.opscenter.client.plugin.RemoteDirectoryServiceImpl.useHttp=false in case of a compatibility issue.


Add explicit load balancing for High Availability (HA) controllers

A newly scheduled build is now, in most cases, directed toward the replica with the least load (according to a simple metric that counts builds already running, queue items already scheduled, and online agents). This applies to interactive triggers (Build Now), scheduled builds, branch indexing, webhooks (multibranch events), REST API triggers, and event triggered builds. The downstream builds scheduled by the build or triggerRemoteJob steps continue to run on the same replica as the upstream build.


Add CloudBees CI communications to the CasC Bundle Retriever

The CasC Bundle Retriever can now communicate with CloudBees CI. This feature will:

  • Remove the benign warning that was generated in the bundle retriever logs

  • Allow the bundle retriever to inform CloudBees CI immediately of the existence of a new version of a bundle

  • Allow the bundle retriever to pre-validate bundles on PRs

Feature Enhancements

Add a configurable queue throttler for the High Availability (HA) controllers

The High Availability (HA) configuration screen now has an option to set a maximum aggregate load for the controller (number of builds, agents, and queue items across all replicas). If the maximum aggregate load is exceeded, the new builds are rejected to avoid overloading and destabilizing the system.


Correct asynchronous messaging for High Availability (HA) controllers

Asynchronous messaging relies on the remoting channel that exists between the operations center and the controllers. When you run a High Availability (HA) controller, only one replica has a live remoting channel, so asynchronous messaging did not work between other replicas and the operations center or other controllers.

Now, the messaging implementation works between any replica and the operations center or any other controller.


More efficient metadata storage for the completed builds

After a Pipeline build that uses the regular durability mode is completed, its metadata storage is now compacted for faster access from tools, such as the CloudBees Pipeline Explorer. This also reduces the disk space and inode usage.


When controllers are configured with High Availability (HA), add the ability to Move/Copy/Promote Pipelines

The Move/Copy/Promote operations were not working in the High Availability (HA) controller. This issue was resolved and you can now Move/Copy/Promote the Pipelines to and from the HA controller.


Compatibility of publishEvent with High Availability (HA) controllers

This fix provides compatibility with the publishEvent step when executed from within a High Availability (HA) controller.


High Availability (HA) controllers now supports triggerRemoteJob

High Availability (HA) controllers now have the triggerRemoteJob step. This allows you to use the Pipeline build step to trigger a build on another replica.


Add an aggregated list of running builds in the job API in the High Availability (HA) controller

Certain API endpoints that return a list of builds, such as /job/name/api/json?tree=builds[number], give incomplete information in a High Availability (HA) controller. Information about builds that run on a replica other than the one processing the request are now automatically inserted into the response.


Usages of prototype.js were removed from the CloudBees CAP plugins

With the removal of prototype.js from Jenkins core, usages of this library were removed from the following CloudBees Assurance Program (CAP) plugins:

  • CloudBees Folders Plus plugin (cloudbees-folders-plus)

  • CloudBees Pipeline Stage View Extensions plugin (cloudbees-workflow-ui)

  • CloudBees Pipeline: Templates Plugin (cloudbees-workflow-template)

  • Credentials plugin (credentials)

  • Declarative Pipeline Migration Assistant (declarative-pipeline-migration-assistant-plugin)

  • Prism API (prism-api-plugin)

  • SSE Server Sent Events (SSE) Gateway (sse-gateway)


Controllers CasC bundles sources in the operations center to allow the same repository name from different organizations

We can now define different sources for bundles with the same repository name from different organizations. In addition, the whole URL is used to perform the checkout, allowing the same repository and organization in a different host.


Bundle version included on the Operations center Load Configuration as Code bundles screen

The bundle update log now displays the bundle version on the Load Configuration as Code bundles screen for the Operations center.


Configuration as Code UI timestamp updates

Timestamps now display on the bundle status page and bundle update log of the CloudBees Configuration as Code bundles page.


CloudBees Pipeline Explorer Feature Enhancements

The following feature enhancements were added to this release:

  • A test insights panel is available in the CloudBees Pipeline Explorer that is based on the JUnit test results. In this panel, users can display a summary of their test results and a list of test failures. Each failure includes the stage where it was found and a link to that test in the JUnit plugin page. Users can also search for test names and sort tests by name or duration.

  • A build artifacts panel is available that allows users to access the build artifacts directly from the CloudBees Pipeline Explorer.

  • A preference is now available that displays agents and nodes in the tree view. The menu for the node steps includes a View agent info button that takes you to the main page of that computer.

  • The cloudbees-slack-plugin links now redirect to the preferred user interface that can be configured for the CloudBees Pipeline Explorer.

  • Contextual information has been added when users select multiple lines and open the context menu. Context information includes the duration between the first and last line selected and the number of lines selected.

  • The CloudBees Pipeline Explorer now supports ANSI escape sequences that specify 8-bit and 24-bit colors.

  • The states of the icons are improved when they hover and are active.


Maven deprecation administrative monitor changes

The Maven plugin has been removed from CAP. The existing administrative monitor that informs users about this deprecation has been modified and now informs users about the removal and provides a link to the documentation that describes how to migrate jobs based on the Maven plugin.

Resolved Issues

CloudBees Pipeline Explorer Resolved Issues

The following issues have been resolved:

  • The log lines from the Pipeline steps that fail or emit a warning now display a colored indication on the left side of the log view next to the line as intended. This was inadvertently broken in CloudBees Pipeline Explorer 1.4.

  • Loading the CloudBees Pipeline Explorer can take a long time for failed builds whose flow graph has not been accessed recently. This was primarily an issue for complex Pipelines when the controller uses a network file system. The CloudBees Pipeline Explorer now no longer performs complex operations as part of the initial load, so any slow operations occur after the CloudBees Pipeline Explorer loads the logs.

  • Direct search links with filter and text were not working and the Search all logs button did not display. Behavior changes were also implemented to the search functionality. This issue has been resolved.

  • Scrolling occurred twice when accessing the CloudBees Pipeline Explorer using a line within the page boundaries. This issue has been resolved.

  • Users experienced missing elements on a page until they clicked on an element with actions. The issue has been fixed and all elements now load correctly.

  • When you use the toggle icon in the search input, it starts a new search. This issue is resolved.

  • The JS error occurs when accessing a search directly in Pipeline Explorer. This issue is resolved.


Checkout of controller CasC bundles in the operations center might cause a timeout

When several checkout requests arrived before the checkout request in progress finished, the queue was blocked and timed out.

Changing the queue to an asynchronous checkout resolved the issue.


Auto-update bundle version may incorrectly update the wrong bundle version if multiple bundle.yaml files exist in the repository

If the configured SCM repository containsed multiple CasC bundles it was possible that the wrong bundle version would be updated if Auto-update bundle version was enabled


Bundle Update Timing updates are not passed on to High Availability (HA)

Now, whenever an update is made to Bundle Update Timing configuration, change is synchronized across replicas.


Wrapped token file can not be deleted by the operations center

Now that the operations center no longer needs to delete the token, it won’t appear in the bundle location and the false warning will not appear. Token deletion is delegated to the bundle retriever.


Fix the SSH connections in the CasC Bundle Retriever

The SSH connections to SCM were broken in the last CasC Bundle Retriever update. This issue is resolved.


YAML file is not validated in the Default YAML field of the Configure Controller Provisioning page

If the YAML file content in the Default YAML field is very long, an HTTP error 431 Request Header Fields Too Large would occur when trying to validate it. This issue has been fixed.


The triggerRemoteJob step can hang if the controller is restarted

If a Pipeline build that runs the triggerRemoteJob step waits for a downstream build event (normally completion) when the upstream controller restarts, and that event occurred when the upstream controller was restarting, the step can hang indefinitely (or fails with a timeout error) when it should have continued.


IllegalArgumentException: Illegal character in path when accessing the job REST API

Accessing the job REST API, such as https://jenkins/job/my%20project/api/json?tree=builds, can return an error when the job name contains spaces or other special characters.


Make adjustments to the behavior of load balancing

When many queue items were scheduled in a High Availability (HA) controller, the load balancing can result in momentary anomalous statuses from REST APIs. Quiet periods were also not consistently honored.


Warnings about ApiTokenStats

Every use of an API token to make a REST request produced a warning in the system log of a High Availability (HA) controller. This warning is now suppressed.


A High Availability (HA) controller that will proxy the REST API requests access to queue items can result in a 502 error

A High Availability (HA) controller automatically will proxy the REST API requests to look up (or cancel) queue items for the replica that owns the item. If the registered replica is unresponsive, for example, because it just terminated, the originating replica now automatically retries the registration lookup and will proxy every second up to ten times.


Inbound agent adoption could fail with “Unknown client name” error

Under certain race conditions, when a build that runs on a High Availability (HA) controller was adopted because the owning replica terminated, an inbound agent in use by that build can fail to reconnect (giving an “Unknown client name” error) ultimately hitting a 5m timeout. Some background processes are now suspended after a replica begins shutdown that may have been responsible for the cluster losing track of the proper owner of the agent.


SnakeYAML API Upgrade

SnakeYAML API has been upgraded from 1.33 to 2.2.


Upgrade org.json:json from 20230227 to 20231013

Upgraded org.json:json from 20230227 to 20231013.


Remove call to hetero-list YUI button

Removed the call to the hetero-list YUI button that was removed from Jenkins core.


Optimized handling of job next build number in HA controllers

Synchronizing the next build number of jobs can be a bottleneck in HA controllers on slow network filesystems. This process is now improved and the issue has been resolved.


More reliable cross-replica build loading for High Availability (HA) controllers

Previously, High Availability (HA) controller replicas could partially load build records even when the builds were in progress on other controllers, leading to errors. Now, a replica only loads a build record when the build is complete or when it is adopting a build in progress.


Retain queue ID when adopting queue items in High Availability (HA)

If you schedule a build using a High Availability (HA) controller, but it does not start before the replica shuts down, the corresponding adopted queue item in another replica now uses the same ID. This allows external tools to track the build when it does start (or the item is cancelled).

Known Issues

HTTP Client used for Operations Center to Controllers connection leads to performance issues

Because of known issues in the Java HTTP Client, there could be performance issues in Operations Center to Controllers interactions in heavily loaded environments.

More details about this issue and workarounds are documented in Operations Center Client leaks HTTP Clients since version 2.401.1.3.


Duplicate Pipeline Template Catalogs in the Configuration as Code (CasC) for Controllers jenkins.yaml file on each instance restart

If a Pipeline Template Catalog is configured in the CasC jenkins.yaml file and the id property is not defined, the catalog is duplicated on each instance restart and in the exported CasC configuration.


Error when renaming an existing EC2 cloud

When the name of an existing cloud node is updated, the user receives a 404 error after selecting save. This is because the cloud page uses the cloud name as part of its URL. When the user saves the name, Jenkins sends the user to the URL with the old cloud name. Please note that all node changes are saved successfully.

Upgrade Notes

Operations center CloudBees Assurance Program plugin changes since 2.414.3.7

The following plugins have been added to the Operations center CloudBees Assurance Program since 2.414.3.7:

  • Master Provisioning Kubernetes (master-provisioning-kubernetes)

  • Prism API Plugin (prism-api)

The following plugins have been removed from the Operations center CloudBees Assurance Program since 2.414.3.7:

  • Master Provisioning Core (master-provisioning-core)

  • Master Provisioning Kubernetes (master-provisioning-kubernetes)