CloudBees CI on modern cloud platforms 2.504.1.6

Upgrade Notes

Agents.SeparateNamespace.Enabled is now a required argument in the CloudBees CI Helm chart

Agents.SeparateNamespace.Enabled=true|false is now a required argument in the CloudBees CI Helm chart. You must pass this argument to your Helm chart or include it in your custom values file prior to installing or upgrading CloudBees CI on modern cloud platforms, or the installation or upgrade will fail with the following error:

Error: UPGRADE FAILED: values don't meet the specifications of the schema(s) in the following chart(s):
cloudbees-core:
- Agents.SeparateNamespace.Enabled: Invalid type. Expected: boolean, given: null▼

CloudBees strongly recommends that you use a separate namespace for agents to prevent pods from accessing sensitive information on the operations center. If separate namespaces are not used for agents, an administrative monitor is now displayed in CloudBees CI. To use a separate namespace for agents:

Add the following to your custom values file to enable the namespace:
```
Agents:
  SeparateNamespace:
    Enabled: true▼
```
Create the namespace. You can either create the namespace manually and label it with cloudbees.com/role: agents or allow Helm to create it automatically by adding Create: true to your custom values file:
```
Agents:
  SeparateNamespace:
    Enabled: true
    Create: true▼
```
Perform the installation or upgrade. For more information, refer to Installing CloudBees CI on modern cloud platforms on Kubernetes.

The rbac.masterServiceAccountName Helm chart property is now deprecated for managed controllers

The rbac.masterServiceAccountName Helm chart property is deprecated. Each controller now uses its own Kubernetes service account. The name of the service account matches the sanitised controller name. Special characters in the controller name are replaced with hyphens (-); the resulting name is also used to generate the controller URL.

The rbac.masterServiceAccountName setting is retained for backwards compatibility. This allows controllers to continue running after upgrading to this version and before completing a stop/provision cycle. After reprovisioning the controllers, rbac.masterServiceAccountName is no longer used.

This change won’t break any core functionality. However, if your setup relies on a single service account shared across all controllers for external processes (outside of CloudBees CI), those processes may be affected.

YahooUI library has been removed: As previously noted in the 2.492.2.2 release, starting with this release, the YahooUI library has been removed. This may cause user interface rendering issues with custom or unmaintained community (Tier 3) plugins.

If you have not done so already, you should make plans to remove usage of the YahooUI library in your custom plugins.

Removed support for non-CloudBees Assurance Program SCM plugins from SDA Analytics: SDA Analytics kept support for some SCM plugins that were not in CloudBees Assurance Program, including Subversion (subversion), Mercurial (mercurial), Gitea (gitea), and multi-SCM (multiple-scms). They are now removed.

Jenkins 2.504 upgrade notes

New Features

CasC Controller Bundle Service: A new service has been added to CloudBees CI. The Configuration as Code controller Bundle Service reduces operations center load and avoids a single point of failure. By using this new service, users can now offload Configuration as Code bundle distribution from the operations center to a dedicated service for better performance and reliability. It is built-in with the Helm chart, making it easily enabled during installation, with seamless integration into the user’s CloudBees CI environment.

The service supports SCM connectors with several authentication methods, such as: username/password, token, SSH keys and GitHub app keys to display validation results in GitHub pull requests. For more information, refer to Setting up a managed controller using the CasC Controller Bundle Service.

Feature Enhancements

Improved Node Display on Label Index Screen in High Availability (HA) controller: The /label/$name/ page on a High Availability (HA) controller displayed inconsistent results. This depended on which replica hosted the sticky sessions and which replicas agents were connected to. Now, this page displays all permanent or cloud agents defined in the controller, with their online/offline status icons, regardless of which replica handles the request.

New healthcheck endpoint for managed controllers: Managed controllers now offer a new health check endpoint. This allows for more reliable failure detection, particularly for High Availability (HA) controllers. Enabling this new option is recommended for any controller running the same version as operations center.

An admin monitor displays if the Quiet Down mode is enabled.: Quiet Down mode in a controller prevents the acceptance of new builds. This mode is useful when restarting the controller. However, in a High Availability (HA) controller, one replica is always available (restarts are managed by rolling restart). As such, Quiet Down mode isn’t required, so the synchronization of the Quiet Down status among replicas isn’t performed. Quiet Down mode is maintained for backward compatibility with non-High Availability (HA) controllers. Now, an administrator monitor appears to the user when they put a replica in Quiet Down mode.

New administrative monitor for Pod Security Standards enforcement in Kubernetes clusters: CloudBees CI on modern cloud platforms now attempts to detect that Pod Security Standards (PSS) are enforced in the Kubernetes cluster. If PSS is not enforced in the Kubernetes cluster, an administrative monitor is displayed in CloudBees CI. For more information, refer to Pod Security Admission for CloudBees CI on modern cloud platforms.

The CloudBees Pipeline Explorer plugin (cloudbees-pipeline-explorer-plugin) no longer has a dependency on the Operations Center Context plugin (operations-center-context): Previously, the CloudBees Pipeline Explorer plugin (cloudbees-pipeline-explorer-plugin) had a hard dependency on the Operations Center Context plugin (operations-center-context). This is now optional. If features that are supported by the Operations Center Context plugin are not in use (for example, the triggerRemoteJob Pipeline step), the Operations Center Context plugin may be disabled or uninstalled.

Deleting outdated usage analytics about PodSecurityPolicy: CloudBees CI no longer supports the older versions of Kubernetes which supports PodSecurityPolicy. Therefore, usage analytics about PodSecurityPolicy are no longer useful.

Improved appearance of table elements to align with the Jenkins UI: The appearance of table elements has been updated to align with recent improvements to the Jenkins UI.

Resolved Issues

High Availability (HA) controllers failed to synchronize a metadata file for completed builds.: When a Pipeline build completed High Availability (HA) controller, the workflow-completed/flowNodeStore.xml file was written by the replica managing the build. Other replicas were notified that they could now load the completed build. However, they weren’t instructed to wait until the expected timestamp of this file was observed. This could lead to problems loading the build from other replicas, depending on the file system (such as NFS).

Resolved a performance issue when using permanent outbound agents on High Availability (HA) controllers with a large number of nodes.: When using many permanent outbound agents on High Availability (HA) controllers , the recommended CloudBees High Availability retention strategy experienced a performance issue at high scale. This issue caused excessive CPU usage and could even cause queue processing to hang.

Removed excessive calls to retention strategy check to prevent performance issues: When adding, updating, or removing nodes in a controller, the retention strategies of all other nodes were checked.

In systems with a large number of nodes and high activity among cloud nodes (which were frequently added and removed), this could cause high CPU usage and associated performance issues.

Next and previous buttons in CloudBees Pipeline Explorer inaccurate in High Availability (HA) controllers: On a High Availability (HA) controller with multiple builds of a given job running across multiple replicas, the Next and Previous buttons in CloudBees Pipeline Explorer would only navigate among completed builds or builds running on the replica hosting the sticky session. Now, these links reflect the newest build older than the displayed build, or the oldest build newer than the displayed build, respectively, even on High Availability (HA) controllers.

Resolved issue with fallback security realm activation during High Availability (HA) rolling upgrades: During a rolling upgrade of a High Availability (HA) controller connected to operations center, a race condition could occur when the replica connected to operations center shuts down. The remaining replicas, detecting the connection as offline, would simultaneously attempt to switch to the fallback security realm via SecurityEnforcer#useFallbackIfOffline.

Because this process involved concurrent writes to config.xml from multiple replicas, it could cause issues on certain file systems, such as CIFS. These issues included temporary write failures or, in rare cases, deletion of the config.xml file due to locking conflicts.

The fallback mechanism has been improved to prevent simultaneous write attempts, ensuring safe and consistent configuration management during rolling upgrades.

Managed controller’s URL was activated in operations center before the controller was ready to serve traffic: The managed controller’s URL is displayed only after the underlying pod is ready and the load balancer registers routing. Previously, users were shown the URL immediately after provisioning. Navigating to the URL at that point could result in a 503 Service Unavailable error until the initial pod readiness process was completed. Now, users remain on the provisioning page where the live pod status is visible. This change prevents premature redirection and improves clarity during startup.

Retention strategy checks skipped when a High Availability (HA) replica is shutting down.: The High Availability (HA) retention strategy no longed does unnecessary checks when shutting down a replica.

Avoid unnecessary logging when the High Availability (HA) controller is terminating: If the High Availability (HA) controller replica were already terminating, operations relying on Hazelcast availability would fail and generate unnecessary exceptions.

These operations are now skipped in such cases, eliminating the logging noise.

Extra executor agents in an High Availability (HA) controller offered launch instructions: When configuring an inbound permanent agent on a High Availability (HA) controller with multiple HA executors, the main (Status) page of the agent, when offline, displayed incorrect instructions for connecting it. These instructions were inappropriate because you should connect only directly to the main agent.

Users could directly launch clones of High Availability (HA) multi-executor agents: Users should connect only to the original agent when configuring a multi-executor inbound agent on a High Availability (HA) controller. This agent automatically launches additional processes to handle the extra executors. Previously, this constraint wasn’t enforced, which could result in duplicated processes or other unexpected behavior. Now, direct attempts to launch any executor other than the original agent are blocked.

High Availability (HA) controllers no longer switch to the offline backup security realm when applying a rolling restart.: When applying a rolling restart to a High Availability (HA) controller, some replicas could briefly switch to the offline backup security realm.

This behavior no longer occurs. As long as operations center is available, the High Availability (HA) controller will continue to use single sign-on (SSO).

Managed controller provisioning now displays a correct status for the Statefulset: The Backend Status section of a non-HA managed controller’s manage page incorrectly displayed Statefulset 1/1 replicas even when the controller pod was not ready. This issue is now resolved. The section now displays Statefulset 0/1 replicas until the pod is ready and changes to Statefulset 1/1 replicas when the pod is ready to serve traffic, updating with a live page refresh.

CloudBees Pipeline Explorer was unusable if the log or its metadata was malformed: When writing Pipeline build logs, CloudBees Pipeline Explorer also writes a log-metadata file, which contains information used to support various CloudBees Pipeline Explorer features. Errors when writing logs or restarting CloudBees CI may result in this metadata becoming out of sync with the main log file.

Previously when this occurred, CloudBees Pipeline Explorer was unable to load the build log and returned an error. Now, CloudBees Pipeline Explorer is typically able to recover from this type of issue, affected log lines are marked as [malformed metadata], and CloudBees Pipeline Explorer displays a warning message explaining that the log contains malformed metadata. If this occurs, download the log to view the raw log data.

Avoid thread spikes with controller lifecycle notifications: Controller Lifecycle Notifications can cause thread spikes and memory issues when managing a large number of controllers. To avoid such issues, the maximum number of parallel http requests is now limited by default.

Set the System property com.cloudbees.opscenter.server.webhooks.WebhooksSender.BOUNDED_WEBHOOK_DELIVERY to false to maintain the previous unbounded behavior.

Hashicorp Vault self client token not revoked due to trailing slash: When the global configuration of the CloudBees Hashicorp Vault plugin contains a trailing slash (/) in the Vault URL, the request to revoke the self client token fails. The token is not revoked and the Jenkins logs are spammed with Revoke self client token error errors.

The revoke token requests now handle the trailing slash accordingly.

CloudBees Update Center incorrectly displayed plugin updates if using a plugin catalog: If a plugin catalog was used to install non-CloudBees Assurance Program plugins to a controller, the CloudBees Update Center may have incorrectly indicated that a new version of the plugin was available for installation, when the catalog’s version of the plugin was already installed to the controller.

Fixed Red Hat OpenShift resource naming inconsistency: Fixed a resource naming inconsistency where the Red Hat OpenShift service exposure resources were named incorrectly. This also resolves an issue that prevented the resources from displaying properly in the UI.

Known Issues

Duplicate plugins in the Operations Center Plugin Manager UI: When you search for a specific plugin under the Available tab in the operations center Plugin Manager, the search results show duplicate entries for the plugin.

If Agents.SeparateNamespace.Enabled=true all agents are installed to the same namespace: If a managed controller is installed in its own namespace and Agents.SeparateNamespace.Enabled=true is passed to the Helm chart or included in the custom values file, all agents will run in the cbci-builds namespace by default, where cbci is the namespace of the operations center, and the agents are not installed in a namespace corresponding to the managed controller. This also prevents agents with service accounts from being installed in a namespace corresponding to the managed controller, and could potentially lead to security vulnerabilities since a managed controller admin could tamper with agents for another managed controller.

Wrong error message being displayed: Under certain conditions, after applying a new Configuration as Code Bundle without restarting the controller (using the Reload Configuration option), users might see the following message in the Manage Jenkins section of the affected controller:

A new version of the Configuration Bundle () is available, but it cannot be applied because it has validation errors.

This message does not affect the system stability.

To remove the false message, go to Manage Jenkins CloudBees Configuration as Code export and update Bundle Update.

The controller and operations center fail to start when upgrading CloudBees CI: When upgrading or restarting CloudBees CI, the controller or operations center fails to start and returns a Messaging.afterExtensionsAugmented error. The operations center can also fail to start with an OperationsCenter.afterExtensionsAugmented error. Refer to CloudBees CI startup failure due to IndexOutOfBoundsException related to corrupt messaging transport files for a workaround for this issue.