CloudBees CD/RO v2024.03.0

Released: March 21, 2024

This page provides important notices about releases that may affect your upgrade efforts to the newest CloudBees CD/RO release.

Upgrade notes

The following upgrade notes affect this release:

CloudBees CD/RO UBI-minimal image version

The UBI-minimal image associated with v2024.03.0 is 9.3-1552.

Plugins that use SSL/TLS certificate validation must be updated

When upgrading to v2024.03.0, plugins that use SSL/TLS certificate validation must be updated to their newest released version, due to changes in Perl libraries. If you do not upgrade these plugins, when running plugin procedures with Ignore SSL issues parameter enabled, SSL issues are not ignored and may cause issues in your pipelines and procedures.

Support for nginx-ingress will be deprecated from CloudBees CD/RO Helm charts in upcoming release

In Kubernetes v1.22, nginx-ingress was deprecated and replaced with ingress-nginx, and since then, most major cloud providers have removed support for Kubernetes v1.21 or earlier. Because of this, nginx-ingress will be removed from CloudBees CD/RO Helm charts in an upcoming release (target v2024.06.0), and no longer be supported by CloudBees CD/RO for Kubernetes.

If your project still uses nginx-ingress in your CloudBees CD/RO Kubernetes Helm charts, you must update them to use ingress-nginx prior to upgrading to future releases of CloudBees CD/RO. For help updating from nginx-ingress to ingress-nginx, refer to Before upgrading from Kubernetes v1.21 or earlier to v1.22 or later.

Installation notes

The following information is a running list of installation notes intended to help understand changes across versions:

When upgrading to v2023.12.0 on Kubernetes, new Analytics server certificates are required

When upgrading to v2023.12.0 on Kubernetes, you must regenerate your Analytics server certificates. If you attempt to upgrade to v2023.12.0 with certificates generated using image tags for v2023.10 or earlier, the Analytics server (flow-devopsinsight) fails to start. For more information on generating Analytics server certificates, refer to Configure CloudBees Analytics server certificates.

Remove support for Perl v5.8 from CloudBees CD/RO

In v2023.12.0, CloudBees CD/RO has been updated to use Perl v5.32, and no longer supports Perl v5.8. This update was made to improve the overall security posture of CloudBees CD/RO. Starting with v2023.12.0, both ec-perl and cb-perl utilize v5.32, and automation scripts that currently use ec-perl will be redirected to use v5.32. Additionally, instances of ec-perl within the CloudBees CD/RO documentation have been updated to cb-perl, but from a functional standpoint, ec-perl and cb-perl are identical from v2023.12.0 and later. The CloudBees CD/RO server remains backward compatible with older agents, and this change will only affect your platform if your CloudBees CD/RO agents are upgraded to v2023.12.0.

There are functional changes between Perl v5.8.9 and v5.32.1 that may impact your current automation. Before upgrading your agents to v2023.12.0, you can test your custom-coded steps that use ec-perl with cb-perl, using one of the following options:

  • In your custom-coded steps, replace all shell references to ec-perl with cb-perl, which already uses Perl v5.32, and run your custom-coded steps.

  • In ec-perl (<INSTALLATION_DIRECTORY>/bin), remove PERLDIR="$ECHOME/perl-legacy", which will then default to use Perl v5.32, and run your custom-coded steps.

    • In default installations, ec-perl is located in:

      • Windows: C:\Program Files\CloudBees\Software Delivery Automation\

      • Linux: /opt/cloudbees/sda/

      • Mac & ARM (legacy installer): /opt/electriccloud/electriccommander/

For help troubleshooting or migrating existing automations, refer to KBEC-00515 - Potential issues when upgrading from ec-perl v5.8 to cb-perl v5.32.

Kubernetes boundAgent images removed from Helm charts

In v2023.10.0, the cloudbees-flow Helm charts were updated to use global images (.values.global). As part of this change, boundAgent.imageRepository was removed from the bound agent Helm chart default values. Bound agents are now treated as child-chart deployments and created from cloudbees-flow-agent.values.images chart values.

If your projects use bound agent custom images, this change will cause upgrade issues since the CloudBees CD/RO Helm chart will not attempt to load your bound agent custom image. For information on using bound agent custom images, refer to Kubernetes configuration options.

Unmatched DSL entries are by default treated as properties

When passing values in DSL, unmatched entry types are treated as properties. This is expected behavior. However, in some instances, this may result in unexpected errors. For instance, providing an array as a value when only a single value is expected results in the error message:

Could not parse XML: multiple values provided for argument 'value' where only one value is expected.
▼

To bypass this behavior, you can either:

  • Disable this behavior with the server setting: /server/settings/useAPIContextForDslProperties=false.

  • Use the Groovy keyword def with the variable definition in the DSL code.

Kerberos keytabs management will be discontinued

In v2023.10.0, CloudBees deprecated the Kerberos keytab APIs and removed the Kerberos keytabs management feature from menu:Configurations [SSO Configurations]. If your projects currently use Kerberos keytabs, they are unaffected. For information on configuring new keytabs, refer to Configuring single sign-on with Kerberos.

Behavior to update lists of Global Personas has changed

In previous CloudBees CD/RO versions, when adding new personas using the modifyGroup and modifyUser APIs with the --personas argument, new personas would completely overwrite the list of existing personas, leaving only the newly added ones (default: replace).

In v2023.10.0, this behavior changed to only append the new personas passed using --personas to the existing personas list (default: merge). However, if you want to overwrite the list of existing personas with a list of new personas passed using --personas, you can do so using the --clearPersonas true argument.

If you are currently using automation to replace your complete list of personas when adding a new one, you must now include --clearPersonas true argument or any new personas will only be appended to the existing personas list.

ec-jruby and ec-jython wrappers deprecated

CloudBees deprecated the CloudBees CD/RO ec-jruby and ec-jython wrapper programs with v10.11. The wrapper programs are no longer installed as part of CloudBees CD/RO tools.

Potential backward incompatibility

CloudBees CD/RO Docker images are now based on UBI minimal instead of UBI standard, as in previous releases. Some packages are not installed on our Docker images by default. This may cause backward incompatibility if your environment depends on these tools and utilities. CloudBees CD/RO Docker images now use microdnf as a package manager and yum and dnf package managers are no longer available. To retain backward compatibility, CloudBees CD/RO provides a symlink for microdnf as dnf. These package managers are not 100% compatible, which may cause unexpected errors and require modification of the scripts.

CGI module and support removed

CloudBees removed the CGI module and support with v.2023.02.0. After you upgrade your CloudBees CD/RO server to v2023.02.0, you must also upgrade your CloudBees web server.

UI file upload limits are now controlled by PHP in /opt/cloudbees/sda/apache/conf/php.ini.

For default parameter values and more information, refer to Configuration settings preserved after an upgrade.

Duplicate applications warning

CloudBees CD/RO v10.9 included a fix for an issue that caused duplicate applications for applications created in the UI. The upgrade process fails when there are duplicate applications. CloudBees resolved this issue for MariaDB, MySQL, and Postgres databases. Ensure that your current installation does not contain applications with duplicate names. You can rename or delete the applications. You must remove duplicate applications by ID using the deleteObjects API. You cannot delete duplicate applications by name. For more details and scripts to assist with this process, refer to KBEC-00513 - How to resolve applications with duplicate names before upgrading from a version prior to v10.9.

Apache ZooKeeper required update

The ZooKeeper version bundled with CloudBees CD/RO v10.5 was updated from v3.4.6 to v3.8.0. CloudBees CD/RO v10.5+ requires ZooKeeper v3.8.0. For installation and upgrade instructions, refer to Install ZooKeeper and Upgrade a clustered environment.

Legacy services applications and container entities

In CloudBees CD/RO v10.3, the legacy Services applications and Traditional applications with containers were deprecated and removed. Before you upgrade to CloudBees CD/RO v10.3 or later, you must migrate your applications to the current microservices application model.

Legacy webhook triggers

As of v10.8, webhook triggers configured and scheduled before v10.1 have been deleted. Polling triggers configured and scheduled prior to v10.1 continue to work, but they are not available from the UI to review or run.

CloudBees CD/RO server installation binaries are signed for traditional installations so that you can verify their origin and authenticity. Verifying binaries is an optional step in the installation process that can help prevent a man-in-the-middle attack. For more information, refer to Verify installation binaries.

Upgrading gateway agents

All gateway agents that meet the following criteria must be updated to CloudBees CD/RO v10.2+:

  • Your enterprise implements a multi-zone environment.

  • Agent versions are a combination of pre-v10.2 and v10.2+.

  • The access route to a v10.2+ agent is configured through a pre-v10.2 gateway agent.

Configuring autostart services for Linux installations

Linux installations that you perform as a non-root user or without sudo permissions cannot automatically start the CloudBees CD/RO server, web server, repository server, or agents. Instead, you must set up the service autostart after installation is complete. Refer to Configure autostart for non-root/non-sudo Linux installations to learn more.

Upgrading your CloudBees CD/RO environment
Before starting an upgrade, make sure to back up your existing CloudBees CD/RO data.
Upgradable versions

Upgrades to CloudBees CD/RO 10.x are supported only from ElectricCommander 5.0. For upgrade instructions, refer to the Upgrade on traditional platforms.

Updating the MySQL configuration before upgrading

Since release 8.0.1, CloudBees has instructed customers using a MySQL database to add the following two lines to their MySQL configuration:

init_connect='SET collation_connection = utf8_unicode_ci, NAMES utf8'
skip-character-set-client-handshake

Before upgrading CloudBees CD/RO, you must remove these lines or comment them out. Otherwise, jobs will not start.

Ensuring the correct default MySQL default collation

Make sure that the default collation for the MySQL database schema is set to utf8_unicode_ci or utf8_general_ci and that no table in the schema overrides this setting. The CloudBees CD/RO server checks this configuration on startup and logs errors in the server log if it is not set correctly.

If the collation is not configured correctly, entering non-ASCII text into CloudBees CD/RO can cause errors. For example, setting a release name to a non-ASCII value, and attempting a search, causes an exception.

If your MySQL database schema, or any tables within, are set to a non-UTF-8 collation order, refer to the Knowledge Base article KBEC-00385 - Converting a MySQL Database From Latin-1 to UTF-8 for detailed instructions about safely converting your schema to UTF-8. [NMB-26521, NMB-27459]

Upgrading agents that run the ec-groovy job step in multizone deployments

In multizone CloudBees CD/RO deployments, CloudBees CD/RO agents that are in a different zone than the CloudBees CD/RO server must be upgraded to version 9.0 or later for the ec-groovy job step to run successfully on those agents. You must also upgrade the gateway agents that lead back to the server’s zone, including those in any zones in between the agent’s zone and the server’s zone. [NMB-27490]

For details about multiple zones and gateway agents, refer to Zones and gateways.

Removing the SSL 2.0 Client Hello or SSLv2Hello protocol from your security configurations

CloudBees recommends removing the SSL 2.0 Client Hello or SSLv2Hello protocol from your security configurations for all components. [NMB-27934, NMB-29326]

  1. Upgrade agents to the latest operating system version for security reasons.

  2. If this warning appears on the Automation Platform UI:

    Note: We recommend removing `SSL 2.0 Client Hello` format from server configuration and upgrade older agents as indicated on the Cloud/Resources Page to avoid security risk.
    ▼

    then enter the following command on the CloudBees CD/RO server:

    $ ecconfigure --serverTLSEnabledProtocol=TLSv1.2
    ▼
Upgrading the CloudBees Analytics server

This section provides information about upgrading the CloudBees Analytics server.

Potential breaking change: Elasticsearch update

The Elasticsearch version shipped with CloudBees Analytics v10.2 has been updated from v6.6.2 to v7.10.2. As such, this update may create breaking changes in your custom reports. All changes related to the new version are described in Elasticsearch documentation.

Your customs reports may be affected due to changes for missing document values handling. The doc['field'].value now throws an exception if the document is missing a value for the field field. To check if a document is missing a value, you can use doc['field'].size() == 0.

  • It is not possible to upgrade CloudBees Analytics v9.0.1 and below to CloudBees Analytics v10.2.0 and above. The installer exits with an error and an appropriate message when such an update is attempted. If you need to upgrade CloudBees Analytics v9.0.1 and below, you must first upgrade to a version between 9.1.0 and 10.1.0, or 9.0.2 and above. After that, you can upgrade CloudBees Analytics to v10.3.0 or higher. [NMB-31030]

  • For previous CloudBees Analytics upgrades from v9.0.1 and below: CloudBees Analytics data may contain obsolete indexes that are incompatible with CloudBees Analytics v10.2.0 and above. To work correctly, it is necessary to re-index these indexes before an upgrade. The installer prompts you to do this before upgrading.

    • In console mode and UI mode, the installer displays the following prompt if outdated indexes are detected:

      One or more Elasticsearch indexes were created in an obsolete version of Elasticsearch.
These indexes must be re-indexed for the upgrade to be successful.
Do you want to start the reindexation? [n/Y]
      ▼

      After an affirmative answer, the installer automatically reindexes and continues the upgrade.

    • In silent mode, the installer reindexes automatically.

  • Backing up and restoring custom settings

    The CloudBees Analytics installer overwrites the elasticsearch.yml configuration file with a new file. This file includes a Custom Settings section, which lets you add Elasticsearch settings not managed by the CloudBees Analytics server without being lost during an upgrade. The installer preserves the settings in the Custom Settings section. [NMB-25850]

  • Upgrading CloudBees Analytics clusters

    The principle of forming a cluster in CloudBees Analytics has changed in v10.2 due to the update of Elasticsearch v7.10.2. In this regard, an additional action is required to upgrade to CloudBees Analytics v10.2 or later:

    When updating the first master node, you must explicitly specify that it is the first node to be updated. If this action is not performed, any cluster that is being updated is placed out of service.

    All installers have been instrumented to accommodate this change. Refer to Upgrade the CloudBees Analytics server for more details. [BEE-2717]

  • CloudBees Analytics server configuration notes

    For a production environment, CloudBees recommends that you install the CloudBees Analytics server on a system separate from systems running other CloudBees CD/RO components (such as the CloudBees CD/RO server, web server, repository server, or agent). If you must install it on the same system (such as for testing or other non-production or trial basis situations), refer to CloudBees Analytics server with other components for details.

    If your CloudBees Analytics server is configured with multiple nodes in a Kubernetes environment, you must pre-generate your certificates. For more information, refer to Install CloudBees CD/RO within Kubernetes.
CloudBees CI operations center configurations

After upgrading to CloudBees CD/RO v10.7 from v10.0.x, you may need to rework your CloudBees CI operations center configurations.

  • In v10.0.x, CloudBees CI operations center URLs specified in configurations are silently appended at runtime with the /cjoc path component.

  • In v10.1, URLs are used as defined in configurations. The /cjoc component is not appended. To maintain pre-v10.1 runtime compatibility, the v10.1 upgrade process modifies CloudBees CI operations center URLs in existing configurations by hardcoding the /cjoc path component. You need to rework existing URLs in configurations where appending the /cjoc path component is inappropriate.

Configuration notes

The following information highlights important configuration notices from previous releases:

Performing a full import

During a full import, the import operation might hang in the following scenarios. To import successfully into CloudBees CD/RO 8.0 and newer versions, perform the appropriate workarounds [CEV-15447, CEV-11873]:

  • A manual process step in a process has formal parameters. The workaround is to remove the entry related to the property sheet for the job step that is associated with the manual process step.

  • In the exported XML file from an earlier release, two pipelines are in different projects, and both pipelines have no gate tasks. The flow associated with the pipeline is duplicated under both projects. The workaround is to remove the flow element under the projects.

Limitations

When an application is cloned from one project (the original project) to another (the destination project), the tier maps for the application point to the environments with the same names in the destination project. To deploy the application to the environments in the original project, you must first create tier maps connecting the application to those environments.