CloudBees CD/RO v2023.10.0

CloudBees is pleased to announce the CloudBees CD/RO v2023.10.0 long-term support (LTS) release. You can find specific information about this release and upgrading to it in the following sections:

Security fixes

This release includes the following security updates to address potential vulnerabilities:
Java updated to version 17.0.7.

Java has been updated to the latest version - 17.0.7.

Jetty version updated

Updated Jetty from v11.0.15 to v11.0.16.

BouncyCastle version updated

Updated BouncyCastle to version 1.76 to improve security.

Enhanced license digest comparison

Enhanced security by using MessageDigest.isEqual(..) for license digest comparison.

Updated Java SSL context from TLS to TLS 1.2

Changed SSLContext.getInstance("TLS") to SSLContext.getInstance("TLSv1.2") in the code.

Third-party libraries versions updated

New features

This release includes the following new features:
Preview Feature: Managing pipelines and other objects as code with dslsync

Based on insights from the EC-DslDeploy plugin, the new dslsync CLI-based utility extends the functionality of the generateDsl and evalDsl APIs to help manage and promote DSL across multiple files and directories in a GitOps and IDE friendly manner. However, dslsync also adds intelligent DSL evaluation based on Git logs, including actions for delete, modify and apply only changes. Additionally, and unlike EC-DslDeploy, you can run dslsync commands directly from your CLI without requiring a CloudBees CD/RO agent.

To learn more, you can run dslsync --help to review the current CLI help page.

The new dslsync utility is a preview feature.

Linux service management add to Linux installers

Support for systemd service management system has been added to Linux platform installers. To ensure backward compatibility, systemd is only used in environments where the System V service management system is not supported. Additionally, you can now explicitly specify a service management system using the command line parameter --useServiceManagement when running Linux platform installers.

Save releases as templates in the catalog

You can now create catalog items based on existing releases. Once created, the item appears as an option in the Releases  New  Create from catalog item…​ dialog. For more information, refer to Save release as catalog item.

Task Hook

Task hook functionality is now available in CloudBees CD/RO v2023.10. This feature allows the addition of a procedure or command as the last step of task. Adding a task hook extends functionality of plugins and third-party deployers by enabling log parsing, extraction of deployment data and updating of the environment inventory.

Preview Feature: New agent and tools supporting the ARM Linux platform

ARM Linux is now supported as a first class platform in CloudBees CD/RO along with the existing support for x86 64-bit Linux, Windows and Mac. This includes Mac M1 through Rosetta II emulation.

The new agent supporting ARM Linux is a preview feature.

New system-generated admin-readonly user

A new admin-readonly system user has been added. This user is enabled once the admin sets a password for it and will have all the same read privileges as the admin user.

As this is a system user, it cannot be deleted or renamed.
CloudBees CD/RO now supports Kubernetes v1.26

CloudBees CD/RO Helm charts now supports Kubernetes v1.26.

CloudBees CD/RO now supports Kubernetes v1.27

CloudBees CD/RO Helm charts now support Kubernetes v1.27.

Feature enhancements

This release includes the following feature enhancements:
Third-party tool updates for CloudBees CD/RO agents

The following components have been updated:

Enhanced task usability

Tasks are enhanced with unifying behavior; selecting a task name opens the task definition. A link will be present when the task references a procedure, pipeline, or release, allowing the user to navigate to the object’s definition.

Select a plugin without specifying a category

You can now search for a plugin and select one without having to select a category.

EC-ServiceNow plugin dependency removed from MTTR and CFR dashboards

To improve usability, you no longer have to install and configure the EC-ServiceNow plugin to use the mean time to recover (MTTR) and change failure rate (CFR) dashboards.

CloudBees on Kubernetes values file now supports global images

CloudBees CD/RO on Kubernetes Helm chart now supports global values for configuring image settings used by the chart in a single place in the cloudbees-flow values file. When set, global values override the corresponding values set at the component level. For more information, refer to How to set global values in CloudBees CD/RO Helm charts.

Added support for suppressing empty Export DSL catalog items

Added support for suppressEmpty into the Export DSL catalog item.

Integrate custom CloudBees Analytics init container images

Support to use custom CloudBees Analytics init container images was added to the 2023.10.0 Helm charts and values file. For more information, refer to How to install custom CloudBees Analytics init container images.

Customizable wget image

You can now specify an internal image that contains wget. This image is not required for installing and running CloudBees CD/RO on Kubernetes. It is only used by tests included in the cloudbees-flow and cloudbees-flow-agent charts. Previously, the tests relied on a public repository, which may not have been accessible in certain environments.

Elasticsearch upgrade

Elasticsearch has been updated to v7.17.12.

Dropdown menus have been enhanced to allow multiple parameter selections

Where applicable, you can now select multiple options in dropdown menus for formal parameter types:

  • Application

  • Environment

  • Pipeline

  • Release

  • Procedure

Support added for special characters in task artifact version field

Now allowing substitution special characters $[] to be used.

Convert a catalog item into a release template

In CloudBees CD/RO v2023.10, the ability to convert a service catalog item into a release template was added. Once converted, this template can be used to create new releases from the catalog item. Refer to: Convert service catalog item into a release template

Improved support for custom certificates and chained CAs

CloudBees CD/RO now supports using custom certificates issued by custom chained certification authorities, as well as managing CRLs as files or distribution points.

EC-DslDeploy now supports importing Service Account entities

Within EC-DslDeploy procedures, you can now import serviceAccount objects.

New retention policy option to keep a number of the most recent items

Added new numberToRetain argument for the createDataRetentionPolicy and modifyDataRetentionPolicy APIs to allow the retention of a given number of the most recent items as specified by the numberToRetain argument.

Plugin enhancements

CloudBees CD/RO plugins catalog

The CloudBees CD/RO plugins catalog is available on the CloudBees CD/RO documentation site.

For more information about plugin support and versioning, refer to Plugin Concepts.

Legacy platform plugin manager deprecation information

The plugin manager built into the legacy CloudBees CD/RO Automation Platform UI was deprecated with v2023.02.0. CloudBees strongly recommends that you migrate to the new plugin manager. For plugins that support the new plugin format, you can migrate legacy plugin configurations using a built-in Migrate configurations Service catalog item.

Plugin migration information

Migrating your custom plugins

To use the new plugin manager, your plugins must align with the CloudBees plugin development kit (PDK) configuration model and contain the plugin configuration as a first class object. To migrate an existing plugin to the new plugin manager/configuration, CloudBees recommends rebuilding your plugins using the PDK. For more information, refer to Migrate legacy plugins to PDK. Once a plugin is built to support the PDK configuration, legacy plugin configurations for the plugin may be migrated using a built-in Migrate configurations service catalog item.

Plugin configurations based on GWT may not be available to view in the new plugin manager. To view legacy plugin configurations:

  1. Navigate to DevOps Essentials  Platform Home page.

  2. Select the Administration link. The Event Logs page displays.

  3. Select the Plugins page link. The Plugins Manager page displays.

  4. Select the name of the plugin name in the Plugin Label column. The Project Details page displays.

  5. Select the Properties tab.

  6. In the property <PluginName>_cfgs, you can find all configurations for the plugin.

  7. Note any specific configuration names that need to be migrated. You can also migrate all configurations.

For more information, refer to Migrating your plugin configurations.

Current plugin disposition

Plugins released to the community

The following plugins are not supported by the new plugin manager and have been migrated to the community but still remain in the plugin catalog:

  • ECSCM-Perforce

  • ECSCM-Property

  • ECSCM-TFS

  • ECSCM-Accurev

  • ECSCM-Bazaar

  • ECSCM-ClearCase

  • ECSCM-CVS

  • ECSCM-Mercurial

  • ECSCM-Repo

  • ECSCM-StarTeam

  • ECSCM-SVN

  • ECSCM-Synergy

  • ECSCM-Vault

  • EC-Twitter

To generate a list of legacy plugin configurations on your system, you can use this DSL script.

Contact CloudBees Support if you require assistance updating legacy plugins to support the new configuration.

Plugin updates
Plugin and version Release notes link

Bitbucket-DataCenter 1.0.0

EC-DslDeploy 4.2.2 / 4.3.0

EC-JIRA 2.3.0

EC-Jenkins 2.4.1

EC-Kubectl 1.4.1

EC-Nexus 2.3.0

EC-SendEmail 1.1.3

EC-Splunk 1.4.0 / 1.5.0

GitLab 1.1.0

When upgrading plugins that have been migrated to PDK, you must migrate your existing plugin configurations. You can use the Migrate configurations procedure from the Service catalog to migrate your existing plugin configurations.

Plugin Development Kit enhancements

New platform support

This section lists new platform support:

  • RHEL v9 is now supported for traditional installations. The package libxcrypt-compat is required. Use the dnf list libxcrypt-compat command to verify the package is installed. If the libxcrypt-compat package is not installed, install it using sudo dnf -y install libxcrypt-compat.

  • Kubernetes v1.26 and v1.27 are now supported.

On September 1, 2023, CloudBees CD/RO v10.8.x reached EOL. For more information, refer to CloudBees maintenance lifecycle policies. CloudBees recommends upgrading to the newest CloudBees CD/RO release to maintain support.

On October 6, 2023, CloudBees CD/RO v10.9 reached EOL. For more information, refer to CloudBees maintenance lifecycle policies. CloudBees recommends upgrading to the newest CloudBees CD/RO release to maintain support.

Refer to the following topics for a list of officially supported platforms for CloudBees CD/RO:

Resolved issues

Fixed conflicts caused by parent context objects property names

Fixed DSL property name issue that created conflicts between parent context objects and their nested operation arguments.

Calling getPipelineRuntimes API without arguments now supported

Calls to the getPipelineRuntimes API in REST and in ec-groovy can now be made without arguments.

Transaction rollback errors returned by the RDBMS are now being properly interpreted.

Fixed the misinterpretation of transaction rollback errors returned by the RDBMS when committing transactions.

Manual approver user contexts in triggered sub-pipelines affected user context of main pipeline

In situations where sub-pipelines with manual approver tasks were triggered by a main pipeline, the user context of the manual approver was incorrectly applied to the LaunchedByUser context of the main pipeline. This issue has been fixed, and now the LaunchedByUser contexts of main pipelines are unaffected by manual approver contexts of sub-pipelines.

Resolved issue enabling deactivate on a grouped task

Resolved an issue where a task could not be disabled after being added to a group.

Resolved restore inheritance issue

With this release, breaking inheritance at one level does not break inheritance beyond that level.

Default DSL values passed from catalog items were not evaluated

Solved the problem of catalog item process failing, because of missing default value by adding default value of False to the replaceObjects parameter of validateFormalParameter API.

DSL Editor now supports syntax with slash in object name

Resolved an issue where the DSL editor did not work correctly when the object name included a slash.

Changing from step type subprocedure to command removes non-applicable parameters

When changing a step type from subprocedure to command, non-applicable actual parameters are now automatically removed.

Resolved DSL object property import

Fixed simple form properties import without wrapping DSL object, so it creates a property in the project now:

  • ectool evalDsl "propName='propValue'" --projectName projName

Possible to pass both jobStepId and jobId causing failed runs

It was possible to pass both jobStepId and jobId arguments, which caused an exception. This has been updated in the argument validation, and is no longer possible.

Application version not expanded in stage summary

Fixed issue where applications version defined as a property reference were not expanded in the stage summary.

Resolved issue with grouped task copy function

Resolved an issue where the task copy functionality did not work correctly for grouped tasks.

Behavior for assigneesForSkipStage field during DSL Import

Fixed an issue where the assigneesForSkipStage field was not resetting when importing DSL in overwrite mode.

Fixed DependsOn evaluation inconsistencies

Fixed DependsOn evaluation inconsistencies by adding validation when reordering or moving formal parameters, which prevents issues with unresolved dependencies.

The stageSummaryParameter entries when importing to DSL are ordered consistently

Resolved Issue of the inconsistent ordering of stageSummaryParameter entries when importing to DSL by updating the index when processSteps are reinserted into the list within a process.

Sorting results of getUsers API as filtering is now supported

When calling the getUsers API, its sorting functions were applied to results after they were returned, instead of functioning as filters to generate the initial results. This lead to situations where getUsers returned unexpected results that were then sorted. getUsers sorting functions now act similar to filtering functions and are applied to generate the initial results.

Step summaries were not displayed if step failed

When the /myJobStep/summary property was set in a job step, the value was only displayed if the step exits with 0. If the job step fails, the value was not displayed. This issue has now been fixed, and the summary is now displayed without depending on the 0 exit code.

Serially grouped tasks with an initial manual task could not be retried

Fixed issue where serially grouped tasks that contained a manual first task that failed could not be retried after a pipeline restart.

Processes with many linked parallel steps would cause job creation to hang

Fixed issue where, when creating jobs with processes that had many parallel steps linked to each other, job creation would hang.

The convert-credentials-jar-with-dependencies no longer fails with OutOfMemoryError

Fixed issue of convert-credentials-jar-with-dependencies failing with OutOfMemoryError by reimplemented the utility to successfully handled large files.

Actual parameter values are recognized by runReport API

Fixed issue of the runReport API ignoring actual parameters by giving priority to formalParameters passed to runReport. Now the actual parameter value overrides the default parameter from report or widget.

REST requests in the ec-groovy to the CloudBees CD/RO server can contain capital letters

Fixed issue with REST requests in ec-groovy to the CloudBees CD/RO server failing when the hostname contains capital letters.

Resolved widget Drill-down target to Widget in Analytics

The widget drill-down target to widget now filters properly based on the source widget selected option. The destination widget will correctly apply the filter based on the selected source option.

GitHub plugin issue fixed for procedure type changes

Resolved EC-DSLDeploy:installDslFromDirectory import issue with GitHub plugins when procedure type is changed from 'subprocedure' to 'command' or 'command' to 'subprocedure'.

Passing unescaped JSON strings caused CI jobs to fail

Fixed issue where CI job failed when CI Job parameter is passed as JSON string value that was not escaped using Java string rules.

outOfOrderRunAssignee option for tasks was not cleaned up

Fixed issue where the outOfOrderRunAssignee option for tasks was not cleaned up when tasks were imported using evalDSL overwrite equals true.

Retry error handling was not cleaned up when importing tasks

Fixed issue where the following retry error handling options were not cleaned up (string = null, integer = null, and boolean=0) when a task was imported using evalDSL overwrite equals true:

  • afterLastRetry

  • retryCount

  • retryInterval

  • retryType

  • notificationEnabled

  • notificationTemplate

  • approver

WaitDependency DSL non longer remains improperly active during DSL import

Removed extra wait dependencies on DSL import in overwrite mode to fix issue of the WaitDependency DSL improperly remaining active when using evalDSL.

DSL credential order updated

Fixed issue of evaluate DSL not working for project file containing an impersonation credential reference by changing the credentials order in the DSL for a project entity. The impersonation credential for project was moved to after the persistent credentials.

Known issues

MeanLeadTime report does not work correctly without release runs

The MeanLeadTime report does not work correctly when Elasticsearch only has pipeline runs but no release runs.

Artifacts can’t be transferred across zones using UI

The CloudBees CD/RO UI does not allow you to transfer artifacts across zones.

Data from a custom data retention policy schedule is not purged for single runs

When a custom data retention policy schedule is set to run once, the data is not purged after archiving. To purge data after archiving, use a repeat schedule or the global data retention setting.

Using PostgreSQL change tracking may generate errors

When using PostgreSQL with change tracking enabled, EcAuditStrategy errors may appear in the server log. This is a known issue, but is not expected to have any effect on the performance of the system.

Events generated from CloudBees CI create URLs that cause 401 errors

Events that originate from the default CloudBees CI create default configurations. URLs for these new controllers are not Jenkins configured URLs and cause 401 errors.

Kerberos SSO sign-in issues

You may experience SSO sign-in issues when using Kerberos due to a Microsoft known issue.

Process steps modified during runs to be manual will hang

When a process step that is not manual is modified to be manual after the process runs, but before the associated job step evaluated, the step hangs and adds a java.lang.IllegalStateException: Unknown step type: manual exception to the log.

flowRuntime reports existing CloudBees CI job when switching platforms

The flowRuntime response contains hasCIJobs=1 if a release was started from CloudBees CD/RO and the previous release run was triggered within CloudBees CI.

Catalog item objects cannot end in spaces on Windows agents

On Windows agents, "Export DSL" catalog item fails to export objects that end in spaces.

CI build logs are not accessible using getCIBuildLog without controller restart

When running getCIBuildLog for a CloudBees CI build, the build log cannot be accessed without restarting the build CloudBees CI controller. As a workaround, restart your CloudBees CI controller, and set up a number of executors, and getCIBuildLog can then be used to access the CloudBees CI build logs.

v10.2 and earlier legacy services may cause failed upgrades and break database consistency

Before upgrading from CloudBees CD/RO v10.2 and earlier, if legacy services exist in your system, upgrades may fail and database consistency break. Additionally, even if the upgrade returns successfully, it may still be impossible to run the validateDatabase API.

As a workaround, before upgrading from v10.2 and earlier, delete all legacy services and containers, and then perform the upgrade.

Undefined parameters returned in CloudBees CI job response

In CloudBees CI job responses, actual parameters are returned that are not defined within the job. Additionally, saving and reloading the tasks doesn’t clear undefined actual parameters.

Multi-select menu options don’t define specific projects of project objects

Currently, if a formal parameter depends on a dropdown menu to get project parameter dependencies for object-like parameters, such as projectName, you can select multiple options in dropdown menus. However, there is only an object name (or list of names in case of multi-select) in the parameter value with no connection to a project and without the ability to identify which object exists in which projects.

CloudBees does not recommend using multi-select options for parameters used as project parameter dependency for object-like parameters when configuring formal parameters. This applies for the following formal parameter types:

  • Application

  • Procedure

  • Pipeline

  • Release

  • Environment

SyncArtifactVersions procedure completes with success when it should fail

SyncArtifactVersions procedure completes with success, rather than showing a warning, when manifest is missing and overwrite = false.

Automation Platform UI requires artifacts to use English characters in their file names

When you use the Automation Platform UI to upload and publish artifact files with non-English characters in their file names, the operation fails with the following error: Upload file: Exit code 1: ERROR: Publish failure: Unexpected retrieval exception for repository error.

Must restart server to apply LDAP changes

Modifications of LDAP user data (such as email addresses) on an Active Directory server after registration in CloudBees CD/RO do not appear properly in user details (in the Automation Platform UI, the Deploy UI, or ectool) until the CloudBees CD/RO server is restarted.

Not all Elasticsearch operations can be performed in a red state

(Microsoft Windows platforms only) If the Elasticsearch cluster used by CloudBees Analytics is in the red state (meaning that it only partly functions and some data is unavailable), then upgrade, reconfigure, and uninstall operations will not work. Since the Elasticsearch service cannot be stopped when a cluster is in a red state, you must stop the Elasticsearch service process from the task manager before running the installer for these actions.

Microsoft Edge® doesn’t support SAML 2.0

The Microsoft Edge® browser does not work with SAML 2.0 and is missing a self-signed certificate during redirection from the identity provider to the service provider. Microsoft Edge® is not recommended for sign-in via SAML 2.0.

LANG environment variable must be set to en.US.UTF-8

The LANG environment variable must be set to en.US.UTF-8; otherwise, the upgrade fails. Refer to KBEC-00452 - Error installing CloudBees CD/RO 10.0.x when Lang environment variable is different than en.US.UTF-8 for details.

Schedules missing configuration do display runtime error prompts

Error prompts for runtimes started by a schedule are not visible if the schedule was created with a missing configuration.

Changing name in Release Dashboard changes stage status color

The stage inclusion status in the Release Dashboard changes color after a stage is renamed.

Steps that cannot access their child steps are not retried

If an application process step cannot expand to its child steps (because of an invalid run condition or an invalid formal parameter), then the step is not retried even if it uses retry on error error handling. The job eventually completes with an error.

Retry count missing from pipeline runtime page

The retry count for group tasks or rules using automated retry on error is missing from the Pipeline runtime page.

Email notifications are not supported for complex environment mapping

Multiple mapped environments with the same name from different projects are not supported in email notifications.

Path-to-production view missing from imported project

A project import might not include the path-to-production view.

All subreleases must be present to link to a release

All subreleases of a release must appear before the release in the DSL for the release-to-subrelease links to be created.

CloudBees Analytics report editor doesn’t include search by assignee

The ability to search by assignee in a Deployment Report is not available in the CloudBees Analytics report editor.

Additional Release Command Center configurations for Jira

If Release Command Center was set up for Jira for user stories and defects, and the JIRA project name was mapped to the release project name using the field mapping projectName:releaseProjectName, then before upgrading to 10.0, the field mapping must be updated to mention the actual release project name using the following field mapping format: "release-project-name-in-CloudBees CD/RO":releaseProjectName.

Approval by email on manual tasks

Approval by email on manual tasks should not expect parameters.

ectool export and ectool import should only be used between same server versions

If you use the ectool export to export your system configuration from a previous release, and then use ectool import to import the same configuration to a CloudBees CD/RO 10.0 server, some out-of-the-box content introduced in the releases since the version from which the full export was done, such as new or updated plugins, new catalog items, and persona-based menu items, may be missing in the CloudBees CD/RO server UI. It is recommended to use ectool export and ectool import only between servers at the same version.

SSO requires additional PHP configuration

SSO does not work unless PHP configuration is changed due to a security-related request. As a workaround, change session.cookie_samesite to "Strict" in /opt/electriccloud/electriccommander/apache/conf/php.ini and restart the web server.

No UI to run or review pre-v10.1 triggers

CloudBees CD/RO v10.1 introduced new triggers and an updated UI for them. Pre-v10.1 triggers will continue to work but there is no UI to review or run them.

Legacy definitions and references cause unexpected behavior for full data exports

Before using the export command to perform a full data export from the CloudBees CD/RO database, delete any legacy definitions and references to service objects from applications and releases.

Reverting changes is not possible for all objects

You can only revert changes for high-level design objects such as applications procedures, procedure steps, workflow definitions, and state definitions.

Restarting the CloudBees CD/RO server while new records are created for all tracked objects might take at least as long as an export or import of all projects (10 to 40 minutes for a large project).
Recursively traversing nested group hierarchies may cause performance issues

Enabling Recursively Traverse Group Hierarchy might impact system performance when the LDAP group hierarchy is traversed. The amount of impact varies with the configurations of the CloudBees CD/RO and LDAP servers, the depth of group hierarchy in the LDAP server, and the network latency between the servers. Ensure that your directory provider can handle the additional load for supporting nested group hierarchy traversal.

Disabling and re-enabling change tracking may cause performance issues

System performance might decrease if you disable change tracking at the server level and then re-enable it. Change tracking is enabled by default. For details about using change tracking, refer to change tracking.

Behavior changes

Kubernetes boundAgent images removed from Helm charts

In v2023.10.0, the cloudbees-flow Helm charts were updated to use global images (.values.global). As part of this change, boundAgent.imageRepository was removed from the bound agent Helm chart default values. Bound agents are now treated as child-chart deployments and created from cloudbees-flow-agent.values.images chart values.

If your projects use bound agent custom images, this change will cause upgrade issues since the CloudBees CD/RO Helm chart will not attempt to load your bound agent custom image. For information on using bound agent custom images, refer to Kubernetes configuration options.

Behavior to update lists of Global Personas has changed

In previous CloudBees CD/RO versions, when adding new personas using the modifyGroup and modifyUser APIs with the --personas argument, new personas would completely overwrite the list of existing personas, leaving only the newly added ones (default: replace).

This behavior has been changed now to only appends the new personas passed using --personas to the existing personas list (default: merge). However, if you want to overwrite the list of existing personas with a list of new personas passed using --personas, you can do so using the --clearPersonas true argument.

If you are currently using automation to replace your complete list of personas when adding a new one, you must now include --clearPersonas true argument or any new personas will only be appended to the existing personas list.

Kerberos keytabs management will be discontinued

CloudBees has deprecated the Kerberos keytab APIs and will remove the Kerberos keytabs management feature from Configurations  SSO Configurations. If your projects currently use Kerberos keytabs, they are unaffected. For information on configuring new keytabs, refer to Configuring single sign-on with Kerberos.

The ec-perl utility will be upgraded to use an updated Perl version in a future release

In an upcoming release (planned for v2023.12.0), ec-perl, which currently uses Perl v5.8, will be upgraded to use Perl v5.32. This update is being made to improve the overall security posture of CloudBees CD/RO, and to align the Perl versions used by ec-perl and cb-perl, which already uses Perl v5.32. If your projects use custom-coded steps implemented with ec-perl, they may be impacted by this change.

+ There are steps you can take to prepare for this change. Before upgrading your production environments to v2023.12.0, CloudBees recommends testing all custom-coded steps using cb-perl in a development environment to help to identify possible issues. To test your custom-coded steps with cb-perl, use one of the following options:

  • In your custom-coded steps, replace all shell references to ec-perl with cb-perl, which already uses Perl v5.32, and run your custom-coded steps.

  • In ec-perl (<INSTALLATION_DIRECTORY>/bin), remove PERLDIR="$ECHOME/perl-legacy", which will then default to use Perl v5.32, and run your custom-coded steps.

    • In default installations, ec-perl is located in:

      • Windows: C:\Program Files\CloudBees\Software Delivery Automation\

      • Linux: /opt/cloudbees/sda/

      • Mac & ARM (legacy installer): /opt/electriccloud/electriccommander/

Clarification was added for extended support software versions

For CloudBees CD/RO on Kubernetes supported platforms, the definition of unsupported versions was updated to clarify CloudBees does not support third-party extended support versions. For more information, refer to Supported platforms for CloudBees CD/RO on Kubernetes.

CloudBees CD/RO UBI-minimal image version

The UBI-minimal image associated with v2023.10.0 is 9.2-717.

Unmatched DSL entries are by default treated as properties

When passing values in DSL, unmatched entry types are treated as properties. This is expected behavior. However, in some instances, this may result in unexpected errors. For instance, providing an array as a value when only a single value is expected results in the error message:

Could not parse XML: multiple values provided for argument 'value' where only one value is expected.

To bypass this behavior, you can either:

  • Disable this behavior with the server setting: /server/settings/useAPIContextForDslProperties=false.

  • Use the Groovy keyword def with the variable definition in the DSL code.

Kubernetes bound agent will no longer be specially designated

Starting in v2023.12.0, CloudBees CD/RO on Kubernetes will no longer use a specifically designed bound agent created via the cloudbees-flow values file. Instead, CloudBees CD/RO will use an agent from the agent resource pool.

Installation notes

For complete installation and upgrade instructions, refer to CloudBees CD/RO on Kubernetes and Install CloudBees CD/RO on traditional platforms.

CloudBees CD/RO on Kubernetes

CloudBees CD/RO server and agent Helm chart values are publicly available and provide the CloudBees default installation values. The CloudBees CD/RO images.tag value associated with v2023.10.0 is:

2023.10.0.169425_3.2.54_20231002

CloudBees CD/RO Docker images and Helm charts are signed so that you can verify their origin and authenticity. Verifying Docker tags and Helm charts is an optional step in the installation process that can prevent a man-in-the-middle attack. For more information, refer to Verify Docker images and Verify Helm charts.

Updated Helm charts

Updated Helm charts are available for CloudBees CD/RO v2023.10.0.

Name Chart version App version Description

cloudbees/cloudbees-flow

2.27.0

2023.10.0.169425

A Helm chart for CloudBees Flow

cloudbees/cloudbees-flow-agent

2.27.0

2023.10.0.169425

A Helm chart for CloudBees Flow Agent

Deprecating ec-jruby and ec-jython wrapper programs

CloudBees deprecated the CloudBees CD/RO ec-jruby and ec-jython wrapper programs with v10.11. The wrapper programs are no longer installed as part of CloudBees CD/RO tools.

Potential backward incompatibility

CloudBees CD/RO Docker images are now based on UBI minimal instead of UBI standard, as in previous releases. Some packages are not installed on our Docker images by default. This may cause backward incompatibility if your environment depends on these tools and utilities. CloudBees CD/RO Docker images now use microdnf as a package manager and yum and dnf package managers are no longer available. To retain backward compatibility, CloudBees CD/RO provides a symlink for microdnf as dnf. These package managers are not 100% compatible, which may cause unexpected errors and require modification of the scripts.

CGI module and support removed

CloudBees removed the CGI module and support with v.2023.02.0. UI file upload limits are now controlled by PHP in /opt/cloudbees/sda/apache/conf/php.ini.

For default parameter values and more information, refer to Configuration settings preserved after an upgrade.

Duplicate applications warning

CloudBees CD/RO v10.9 included a fix for an issue that caused duplicate applications for applications created in the UI. The upgrade process fails when there are duplicate applications. CloudBees resolved this issue for MariaDB, MySQL, and Postgres databases. Ensure that your current installation does not contain applications with duplicate names. You can rename or delete the applications. You must remove duplicate applications by ID using the deleteObjects API. You cannot delete duplicate applications by name. For more details and scripts to assist with this process, refer to KBEC-00513 - How to resolve applications with duplicate names before upgrading from a version prior to v10.9.

Apache ZooKeeper required update

The ZooKeeper version bundled with CloudBees CD/RO v10.5 was updated from v3.4.6 to v3.8.0. CloudBees CD/RO v10.5+ requires ZooKeeper v3.8.0. For installation and upgrade instructions, refer to Install ZooKeeper and Upgrade a clustered environment.

Legacy services applications and container entities

In CloudBees CD/RO v10.3, the legacy Services applications and Traditional applications with containers were deprecated and removed. Before you upgrade to CloudBees CD/RO v10.3 or later, you must migrate your applications to the current microservices application model.

Also, before upgrading from CloudBees CD/RO v10.2 or earlier you must delete all legacy services and containers. This will prevent upgrade failure, a database consistency break or inability to run the validateDatabase API.

Legacy webhook triggers

As of v10.8, webhook triggers configured and scheduled before v10.1 have been deleted. Polling triggers configured and scheduled prior to v10.1 continue to work, but they are not available from the UI to review or run.

CloudBees CD/RO server installation binaries are signed for traditional installations so that you can verify their origin and authenticity. Verifying binaries is an optional step in the installation process that can help prevent a man-in-the-middle attack. For more information, refer to Verify installation binaries.

Upgrading gateway agents

All gateway agents that meet the following criteria must be updated to CloudBees CD/RO v10.2+:

  • Your enterprise implements a multi-zone environment.

  • Agent versions are a combination of pre-v10.2 and v10.2+.

  • The access route to a v10.2+ agent is configured through a pre-v10.2 gateway agent.

Configuring autostart services for Linux installations

Linux installations that you perform as a non-root user or without sudo permissions cannot automatically start the CloudBees CD/RO server, web server, repository server, or agents. Instead, you must set up the service autostart after installation is complete. Refer to Configure autostart for non-root/non-sudo Linux installations to learn more.

Upgrading your CloudBees CD/RO environment
Before starting an upgrade, make sure to back up your existing CloudBees CD/RO data.
Upgradable versions

Upgrades to CloudBees CD/RO 10.x are supported only from ElectricCommander 5.0. For upgrade instructions, refer to the Upgrade on traditional platforms.

Updating the MySQL configuration before upgrading

Since release 8.0.1, CloudBees has instructed customers using a MySQL database to add the following two lines to their MySQL configuration:

init_connect='SET collation_connection = utf8_unicode_ci, NAMES utf8'
skip-character-set-client-handshake

Before upgrading CloudBees CD/RO, you must remove these lines or comment them out. Otherwise, jobs will not start.

Ensuring the correct default MySQL default collation

Make sure that the default collation for the MySQL database schema is set to utf8_unicode_ci or utf8_general_ci and that no table in the schema overrides this setting. The CloudBees CD/RO server checks this configuration on startup and logs errors in the server log if it is not set correctly.

If the collation is not configured correctly, entering non-ASCII text into CloudBees CD/RO can cause errors. For example, setting a release name to a non-ASCII value, and attempting a search, causes an exception.

If your MySQL database schema, or any tables within, are set to a non-UTF-8 collation order, refer to the Knowledge Base article KBEC-00385 - Converting a MySQL Database From Latin-1 to UTF-8 for detailed instructions about safely converting your schema to UTF-8. [NMB-26521, NMB-27459]

Upgrading agents that run the ec-groovy job step in multizone deployments

In multizone CloudBees CD/RO deployments, CloudBees CD/RO agents that are in a different zone than the CloudBees CD/RO server must be upgraded to version 9.0 or later for the ec-groovy job step to run successfully on those agents. You must also upgrade the gateway agents that lead back to the server’s zone, including those in any zones in between the agent’s zone and the server’s zone. [NMB-27490]

For details about multiple zones and gateway agents, refer to Zones and gateways.

Removing the SSL 2.0 Client Hello or SSLv2Hello protocol from your security configurations

CloudBees recommends removing the SSL 2.0 Client Hello or SSLv2Hello protocol from your security configurations for all components. [NMB-27934, NMB-29326]

  1. Upgrade agents to the latest operating system version for security reasons.

  2. If this warning appears on the Automation Platform UI:

    Note: We recommend removing `SSL 2.0 Client Hello` format from server configuration and upgrade older agents as indicated on the Cloud/Resources Page to avoid security risk.

    then enter the following command on the CloudBees CD/RO server:

    $ ecconfigure --serverTLSEnabledProtocol=TLSv1.2
Upgrading the CloudBees Analytics server

This section provides information about upgrading the CloudBees Analytics server.

Potential breaking change: Elasticsearch update

The Elasticsearch version shipped with CloudBees Analytics v10.2 has been updated from v6.6.2 to v7.10.2. As such, this update may create breaking changes in your custom reports. All changes related to the new version are described in Elasticsearch documentation.

Your customs reports may be affected due to changes for missing document values handling. The doc['field'].value now throws an exception if the document is missing a value for the field field. To check if a document is missing a value, you can use doc['field'].size() == 0.

  • It is not possible to upgrade CloudBees Analytics v9.0.1 and below to CloudBees Analytics v10.2.0 and above. The installer exits with an error and an appropriate message when such an update is attempted. If you need to upgrade CloudBees Analytics v9.0.1 and below, you must first upgrade to a version between 9.1.0 and 10.1.0, or 9.0.2 and above. After that, you can upgrade CloudBees Analytics to v10.3.0 or higher. [NMB-31030]

  • For previous CloudBees Analytics upgrades from v9.0.1 and below: CloudBees Analytics data may contain obsolete indexes that are incompatible with CloudBees Analytics v10.2.0 and above. To work correctly, it is necessary to re-index these indexes before an upgrade. The installer prompts you to do this before upgrading.

    • In console mode and UI mode, the installer displays the following prompt if outdated indexes are detected:

      One or more Elasticsearch indexes were created in an obsolete version of Elasticsearch. These indexes must be re-indexed for the upgrade to be successful. Do you want to start the reindexation? [n/Y]

      After an affirmative answer, the installer automatically reindexes and continues the upgrade.

    • In silent mode, the installer reindexes automatically.

  • Backing up and restoring custom settings

    The CloudBees Analytics installer overwrites the elasticsearch.yml configuration file with a new file. This file includes a Custom Settings section, which lets you add Elasticsearch settings not managed by the CloudBees Analytics server without being lost during an upgrade. The installer preserves the settings in the Custom Settings section. [NMB-25850]

  • Upgrading CloudBees Analytics clusters

    The principle of forming a cluster in CloudBees Analytics has changed in v10.2 due to the update of Elasticsearch v7.10.2. In this regard, an additional action is required to upgrade to CloudBees Analytics v10.2 or later:

    When updating the first master node, you must explicitly specify that it is the first node to be updated. If this action is not performed, any cluster that is being updated is placed out of service.

    All installers have been instrumented to accommodate this change. Refer to Upgrade the CloudBees Analytics server for more details. [BEE-2717]

  • CloudBees Analytics server configuration notes

    For a production environment, CloudBees recommends that you install the CloudBees Analytics server on a system separate from systems running other CloudBees CD/RO components (such as the CloudBees CD/RO server, web server, repository server, or agent). If you must install it on the same system (such as for testing or other non-production or trial basis situations), refer to CloudBees Analytics server with other components for details.

    If your CloudBees Analytics server is configured with multiple nodes in a Kubernetes environment, you must pre-generate your certificates. For more information, refer to Install CloudBees CD/RO within Kubernetes.
CloudBees CI operations center configurations

After upgrading to CloudBees CD/RO v10.7 from v10.0.x, you may need to rework your CloudBees CI operations center configurations.

  • In v10.0.x, CloudBees CI operations center URLs specified in configurations are silently appended at runtime with the /cjoc path component.

  • In v10.1, URLs are used as defined in configurations. The /cjoc component is not appended. To maintain pre-v10.1 runtime compatibility, the v10.1 upgrade process modifies CloudBees CI operations center URLs in existing configurations by hardcoding the /cjoc path component. You need to rework existing URLs in configurations where appending the /cjoc path component is inappropriate.

Configuration notes

Performing a full import

During a full import, the import operation might hang in the following scenarios. To import successfully into CloudBees CD/RO 8.0 and newer versions, perform the appropriate workarounds [CEV-15447, CEV-11873]:

  • A manual process step in a process has formal parameters. The workaround is to remove the entry related to the property sheet for the job step that is associated with the manual process step.

  • In the exported XML file from an earlier release, two pipelines are in different projects, and both pipelines have no gate tasks. The flow associated with the pipeline is duplicated under both projects. The workaround is to remove the flow element under the projects.

Limitations

When an application is cloned from one project (the original project) to another (the destination project), the tier maps for the application point to the environments with the same names in the destination project. To deploy the application to the environments in the original project, you must first create tier maps connecting the application to those environments.