We are pleased to announce the CloudBees CD/RO 10.3 LTS release of the Software Delivery Automation platform. In this release, we continue the effort to bring CloudBees CI and CloudBees CD/RO into a single platform through improvements to:
Introduction of CloudBees CI license usage reporting
Improvements to both CloudBees CI and CloudBees CD/RO audit reports
In addition, platform enhancements such as improved database performance, simplified multi-zone deployment in Kubernetes, and plugin configuration and management are part of this long term supported release.
- This release includes the following security updates
PHP is upgraded from 7.4.16 to 7.4.22. For details, see https://www.php.net/releases/7_4_22.php. [NMB-30951]
OpenSSL remains at version 1.1.1k. For details, see https://www.openssl.org/news/openssl-1.1.1-notes.html. [BEE-3557]
Apache web server is upgraded from 2.4.46 to 2.4.48. For details, see https://httpd.apache.org/download.cgi#apache24. [NMB-30951]
Elasticsearch is upgraded from v7.10.2 to v7.13.4.
Logstash is upgraded from v7.12.1 to v7.13.4.
- Custom Docker images
CloudBees CD/RO component binaries and associated Dockerfiles are available for download to use in creating custom Docker images. [BEE-6485]
Documentation: Creating custom Docker images.
- Single sign-on with OpenID Connect
Support for OpenID Connect identity provider is added. [BEE-5484]
Documentation: Single sign-on with OpenID Connect
- Updated plugin manager
The plugin manager user interface, now called plugin management, has been updated and is now located from CloudBees CD/RO.
Documentation: Plugin management
- Plugin configurations as code
Plugin configurations are now available as first class objects that can be created and managed from a CloudBees CD/RO project via REST APIs, DSL, and the
ectoolcommand line interface. Access to each of the new plugin configurations can be managed and controlled using access control lists.
Documentation: Creating plugin configurations
|If you are migrating to CloudBees CD/RO v10.3 or later from a pre-v10.3 version, you must perform a one-time migration for each plugin configuration at your site. Refer to latest@cloudbees-cd:plugin-manager:plugin-config.adoc#plugin-config-migration for details.|
- Custom plugin catalog
Plugin management now includes an in-product version of the plugin catalog that can be customized for your CloudBees CD/RO environment. [BEE-7958]
Documentation: Managing the plugin catalog
- SSO improvements to remove directory provider requirements
SAML v2 support in CloudBees CD/RO is enhanced to provide an option to eliminate the need to set up a directory provider like Active Directory for authorization. Users can optionally choose to not configure Directory providers. When this option is selected, users are automatically registered upon authentication by the SSO identity provider.
Documentation: Single sign on with SAML
- Helm chart improvements for installation and configuration of gateway agents in a multi-zone deployment
The CloudBees CD/RO Helm chart has been enhanced to install and configure a pair of gateway agents as part of the product installation for a multi-zone deployment where some CD agents might be running outside the Kubernetes cluster and other agents might be inside the Kubernetes cluster. The CD agent Helm chart has also been enhanced to make the installation and configuration of gateway agents easier.
Documentation: Configuration with internal and external agents
- Other features
Projects view on the CloudBees CD/RO UI now includes procedures. [CEV-28333]
The repository server is enhanced to use GCP storage buckets as backing stores in addition to AWS S3 buckets and NFS mounted filesystems.
The version of Perl used by the
ec-perlcommand line tool has been upgraded from v5.8.9 to v5.32.1. [BEE-7521]
- Database support
- Browser support
- MySQL scalability and performance optimizations
MySQL database scalability and performance on CloudBees CD/RO has been improved significantly when compared with previous versions. [NMB-30755]
Performance improvements for queries doing full table scans
Performance for pipeline executions improved an average of 66%.
CPU utilization now stays below 10%.
These plugins have been updated to support plugin configurations as first class objects. The details of this feature are described here: Creating plugin configurations.
Support removed for the legacy microservices model. This plugin along with EC-Helm can be used to support microservices applications and deployments.
Added support for CloudBees Analytics release command center dashboard integration with CloudBees Feature Management.
Fixed CAM plugin code to handle
Added correct error during attempt to upgrade proxy agent with CAM plugin. [NMB-30950]
Updated with new procedures for promote and demote plugins. [FLOWPLUGIN-9340]
Improved results of some procedures to make them easier to use. [FLOWPLUGIN-9247]
Resolved an incompatibiliy issue in the
Fixed an issue with icroservice deployment failing when application and environment are in different projects. [FLOWPLUGIN-8996]
Fixed up incorrect references to file and image names. [FLOWPLUGIN-9360]
- Discontinued plugins
The following is a list of discontinued plugins:
These plugins have been removed as they are no longer relevant. Use EC-Helm in their place
These plugins have been removed in favor of EC-Git, which can be used for most use cases.
This plugin is no longer supported.
- Plugin Development Kit enhancements
PDK 126.96.36.199 includes:
Updated to these components
Toolkit version: 3.3.0
Groovy core library version: 1.2.0
Perl core library version: 1.4.0
Layout version: 1.3.4
Support for the new plugin configuration as first-class objects.
Fixed expandable values in legacy plugin configurations.
This section lists new platform support. For the current list of supported platforms, consult Supported platforms for CloudBees CD/RO
Support for Ubuntu 20.04 has been added. [NMB-31041]
- Agent support
Support for Ubuntu 20.04 has been added. [NMB-31041]
The 64-bit CloudBees CD/RO agent installer no longer includes 32-bit DLLs. [NMB-30954]
CloudBees CD/RO requires Rosetta2 to be installed on mac M1-based computers. As such, it will automatically be installed on demand by the installer. [NMB-30158]
- CloudBees CD/RO on Kubernetes
Support for Kubernetes versions 1.19x, 1.20.x, and 1.21.x has been added for GKE, Amazon EKS, and AKS. [BEE-8118]
CI URLs used in Native CI integration were not encoded according to HTTP 1.1 spec.
CloudBees CD K8s base images migrated from Ubuntu to Red Hat UBI v8.
Project column on CI builds list was sometimes blank for in-progress CI builds.
ActiveMQ journal buffer size was not large enough to handle conditions of high load. This issue has been addressed by increasing the ActiveMQ journal buffer size to 1 MB. It is now configurable via the environment variable
Pipeline stage with
EC-DSLDeploy export was not generating command task
DSL schedule exports from the UI no longer produce null parameters.
Apache config changes lost after upgrading from CloudBees CD/RO 10.0.3 to any newer version.
Complex releases are failing with DSL export when suppress nulls were enabled in MySQL.
Multiple restarts of Zookeeper pods occured in Kubernetes due to Out Of Memory errors. The default Zookeeper memory limit has been increased to 1 Gi.
Exported DSL of plugin tasks contained in a task group failed to import properly.
Optimised the ping resources operation called from
MasterComponent imported via DSL followed by save threw a
Fixed the limitation of setting
MySQL DB failures with
Moving a property folder under
Standalone utilities jars did not log error messages properly. Affected jars include:
The CAM plugin code did’t handle null
CloudBees CD/RO occasionally failed to install on K8s with AWS EFS storage using CSI Driver.
The job folders in workspace had wide permissions.
Creating a webhook with DSL for an invalid schedule caused all the webhooks to stop processing.
CloudBees CD/RO installation problem on AKS.
now indicates which server settings require restart and which do not.
Removed unused build artifacts from Apache’s distribution including
Installing and uninstalling EC-DSLIDE plugin rendered system unusable.
Date fields were changing the value with up/down/left/right keys, even when the Date field was out of focus.
CloudBees CD/RO Installation in cluster mode on OpenShift fails because Zookeeper could not be installed.
Property picker in release: command task in pipeline was different between planning and running release.
EC-DslDeploy duplicated command task contents when tasks were contained in a task group.
Could not remove all users from the group in the UI.
Tags were not created with
Different UI requests were used when querying the release runs via view previous runs from Pipeline Runs vs Running release from the context menu.
The filter in the
Breadcrumb became stale when viewing the Application process of an application that was recreated. Starting with v10.3, the user is warned that the old application model does not exist or that the application has been recreated, instead of
After application was deployed and snapshotted and if the application was recreated, the UI did not allow rerun of an old application run. Starting with v10.3, the user is warned that the old application model does not exist or application model has been recreated.
The CloudBees CD/RO UI did not show elapsed time taken for Stage and Gate tasks.
Couldn’t attach parameter credential to a component step and to a master component in the CloudBees CD/RO UI.
Custom validation check on credential parameter was not triggered when clicking outside of text box.
UI did not clearly show if system defined personas were editable or read-only.
Procedure step editor lost context. Needed to cancel without saving and reopen to edit the step again.
URL link of Service Catalog item required a space after URL if pointing to SVG file.
The Perl API did not consistently accept
For complete installation and upgrade information, see CloudBees CD/RO installation guide.
- CloudBees CD/RO on Kubernetes
Sample CloudBees CD/RO server and agent Helm chart values, found here, provide CloudBees’s default installation values. The CloudBees CD/RO
images.tagvalue associated with version 10.3 is
- Configuring autostart services for Linux installations
Linux installations that you perform as a non-root user or without
sudopermissions cannot automatically start the CloudBees CD/RO server, web server, repository server, or agents. This means that you must set up service autostart after installation is complete. Learn more here.
- Upgrading your CloudBees CD/RO environment
IMPORTANT: Before starting an upgrade, make sure to back up your existing CloudBees CD/RO data.
- Upgradable versions
Upgrades to CloudBees CD/RO 10.x are supported only from ElectricCommander 5.0. For upgrade instructions, see Upgrade roadmap.
- Updating elements containing applicationServiceMapping [CEV-16237 and CEV-16158]
If your XML export file from CloudBees CD/RO 8.0.1 or earlier versions has elements containing
applicationServiceMapping, you must change all instances of that string in the file to
serviceClusterMappingbefore importing the file into version 10.3. For example, change the following XML:
<applicationServiceMapping> <applicationServiceMappingId>9efcda31-a85f-11e7-8500-0800279f198d</applicationServiceMappingId> <applicationServiceMappingName>9efcda31-a85f-11e7-8500-0800279f198d</applicationServiceMappingName> … </applicationServiceMapping>
<serviceClusterMapping> <serviceClusterMappingId>9efcda31-a85f-11e7-8500-0800279f198d</serviceClusterMappingId> <serviceClusterMappingName>9efcda31-a85f-11e7-8500-0800279f198d</serviceClusterMappingName> … </serviceClusterMapping>
- Updating the MySQL configuration before upgrading
Since release 8.0.1, CloudBees has instructed customers using a MySQL database to use the following two lines in their MySQL configuration:
init_connect='SET collation_connection = utf8_unicode_ci, NAMES utf8' skip-character-set-client-handshake
Before upgrading CloudBees CD/RO, you must remove these lines or comment them out. Otherwise, jobs will not start.
- Ensuring the correct default MySQL default collation
Make sure that the default collation for the MySQL database schema is set to
utf8_general_ciand that no table in the schema overrides this. The CloudBees CD/RO server checks this configuration on startup and logs errors in the server log if it is not set correctly.
If the collation is not configured correctly, then entering non-ASCII text into CloudBees CD/RO might cause errors. For example, setting a release name to a non-ASCII value and attempting a search causes an exception.
If your MySQL database schema or any tables in it are set to a non-UTF-8 collation order, see Knowledge Base article KBEC-00385 - Converting a MySQL Database From Latin-1 to UTF-8 for detailed instructions about safely converting your schema to UTF-8. [NMB-26521, NMB_27459]
- Upgrading agents that run the ec-groovy job step in multizone deployments
In multizone CloudBees CD/RO deployments, CloudBees CD/RO agents that are in a different zone than the CloudBees CD/RO server must be upgraded to version 9.0 or later for the
ec-groovyjob step to run successfully on those agents. You must also upgrade the gateway agents that lead back to the server’s zone including those in any zones in between the agent’s zone and the server’s zone. [NMB-27490]
- Removing the SSL 2.0 Client Hello or SSLv2Hello protocol from your security configurations
CloudBees recommends removing the
SSL 2.0 Client Helloor
SSLv2Helloprotocol from your security configurations for all components. [NMB-27934, NMB-29326]
Upgrade agents older that fall into this category for security reasons:
Windows, Linux: 6.0.3 or older; 6.2 or older
Mac OS: 8.4 or older
If this warning appears on the Automation Platform UI:
Note: We recommend removing `SSL 2.0 Client Hello` format from server configuration and upgrade older agents as indicated on the Cloud/Resources Page to avoid security risk.
then enter the following command on the CloudBees CD/RO server:
$ ecconfigure --serverTLSEnabledProtocol=TLSv1.2
- Upgrading the CloudBees Analytics server
This section provides information about upgrading the CloudBees Analytics server.
It is not possible to upgrade CloudBees Analytics version 9.0.1 and below to CloudBees Analytics version 10.2.0 and above. Installer exits with an error and an appropriate message when such an update is attempted. If user needs to upgrade CloudBees Analytics version 9.0.1 and below, then user must first upgrade to a version between 9.1.0 and 10.1.0, or 9.0.2 and above. After that, the user can upgrade CloudBees Analytics to version 10.3.0 or higher. [NMB-31030]
For previous CloudBees Analytics upgrades from version 9.0.1 and below: CloudBees Analytics data may contain obsolete indices that are incompatible with CloudBees Analytics version 10.2.0 and above. To work correctly, it is necessary to re-index these indexes before an upgrade. The installer prompts the user to do this before upgrading.
In console mode and Ui mode, the installer displays the following prompt if outdated indexes are detected:
One or more Elasticsearch indices were created in an obsolete version of Elasticsearch. These indexes must be re-indexed for the upgrade to be successful. Do you want to start the reindexation? [n/Y]
After an affirmative answer, the installer automatically re-indexes and continues the upgrade.
In silent mode, the installer reindexes automatically.
Backing up and restoring custom settings
The CloudBees Analytics installer overwrites the
elasticsearch.ymlconfiguration file with a new file. This file includes a
Custom Settingssection, which lets you add Elasticsearch settings not managed by the CloudBees Analytics server without being lost during an upgrade. The installer preserves the settings in the
Custom Settingssection. [NMB-25850]
Upgrading CloudBees Analytics clusters
The principle of forming a cluster in CloudBees Analytics has changed in v10.2 due to the update of Elasticsearch v7.10.2. In this regard, an additional action is required to upgrade to CloudBees Analytics v10.2 or later:
When updating the first master node, the user must explicitly specify that it is the first node to be updated. If this action is not performed, a cluster being updated is placed out of service.
All installers have been instrumented to accommodate this change. See Upgrading the CloudBees Analytics server for more details. [BEE-2717]
CloudBees Analytics server configuration notes
For a production environment, CloudBees recommends that you install the CloudBees Analytics server on a system other than systems running other CloudBees CD/RO components (such as the CloudBees CD/RO server, web server, repository server, or agent). If you must install it on the same system (such as for testing or other non-production or trial-basis situations) see CloudBees Analytics server with other components for details.
- Performing a full import
During a full import, the import operation might hang in the following scenarios. To import successfully into CloudBees CD/RO 8.0 and newer versions, perform the appropriate workarounds [CEV-15447, CEV-11873]:
A manual process step in a process has formal parameters. The workaround is to remove the entry related to the property sheet for the job step that is associated with the manual process step.
In the exported XML file from the earlier release, two pipelines are in different projects, and both pipelines have no gate tasks. The flow associated with the pipeline is duplicated under both projects. The workaround is to remove the flow element under the projects.
When an application is cloned from one project (the original project) to another (the destination project), the tier maps for the application point to the environments with the same names in the destination project. To deploy the application to the environments in the original project, you must create tier maps connecting the application to those environments.
In the CloudBees CD/RO and CloudBees Analytics Continuous Integration Builds screen, if a build has a Running status, the Project column may temporarily not display project information. Once the build has completed, the Project column correctly displays the project information for each build.
With CloudBees CD/RO v 10.2.1 and earlier, the DSL Import service catalog fails for grouped tasks.
Browser redirects to port 2080 during first navigation to CD deployed from SDA and Flow Helmm charts.
When you use the Automation Platform UI to upload and publish artifact files with non-English characters in their file names the operation fails with the following error:
Modifications of LDAP user data (such as email addresses) on an Active Directory server after registration in CloudBees CD/RO do not appear properly in user details (in the Automation Platform UI, the Deploy UI, or
(Windows platforms only) If the Elasticsearch cluster, which is used by CloudBees Analytics, is in the red state (in Elasticsearch this means that it only partly functions and some data is unavailable) then upgrade reconfigure or uninstall operations will not work. Because the Elasticsearch service can not be stopped when a cluster is in red state kill the Elasticsearch service process by the task manager before running the installer for these actions.
The Microsoft Edge browser does not work with SAML 2.0 and a self-signed certificate during redirection from the identity provider to the service provider. Edge is not recommended for sign-in via SAML 2.0.
Can’t ignore server mismatch and override passkey from Database Configuration page.
The LANG environment variable must be set to
In some cases, job step diagnostic information is not available and server reports 507 error,
When an application with snapshots created in CloudBees CD/RO 6.1 or earlier is cloned and a project containing this application is imported to CloudBees CD/RO 6.3 or higher the import operation fails.
Error prompts for runtimes started by a schedule are not visible if the schedule was created with a missed configuration.
The stage inclusion status in the Release Dashboard changes color after a stage is renamed.
No error prompt appears for failed tasks and retry tasks during a pipeline runtime.
If an application process step cannot expand to its child steps (because of an invalid run condition or an invalid formal parameter) then the step is not retried even if it uses "retry on error" error handling. The job eventually completes with an error.
The retry count for group tasks or rules using "automated retry on error" is missing from the Pipeline runtime page.
Multiple mapped environments with the same name from different projects are not supported in email notifications.
A project import might not include the path-to-production view.
Jobs might not appear upon drill-down into the "Clusters With Most Deployments" widget in the CloudBees Analytics Microservices Dashboard if the service does not contain a deploy step in the process.
When you do a full import from version 8.0 to version 8.2 or newer and two or more releases have the same name (under different projects) and are associated to the same pipeline then after import the runs for all releases might become associated to the first imported release. This is because CloudBees CD/RO cannot differentiate runs between the releases since all runs are under the same pipeline project and have the same name. To work around this issue rename releases in the export file so that all their occurrences (in
All subreleases of a release must appear before the release in the DSL for the release-to-subrelease link to be created.
The ability to search by assignee in a Deployment Report is not available in the CloudBees Analytics report editor.
If Release Command Center was setup for JIRA for user-stories and defects and the JIRA project name was mapped to the release project name using the following field mapping: ` projectName:releaseProjectName` then before upgrading to 10.0 the field mapping must be updated to mention the actual release project name using the following field mapping format:
Long custom labels in email notifications do not render correctly.
Approval by email on manual tasks should not expect parameters.
Navigation to a sub-release editor takes user to the parent release editor. As a workaround, select the subrelease from the left-hand navigation in the parent’s release editor.
When you use the Deploy UI to edit a resource pool and add a tag while renaming it at the same time, the operation fails with the following error:
Running an application process with a parallel manual application process step or running an application process with a parallel manual application and component process steps fails to delete the project.
If you are signed in to the Deploy UI and upgrade to CloudBees CD/RO 10.0, the version 10.0 sign-in page for the Automation Platform UI goes into an infinite redirect. This is because the version 10.0 Automation Platform UI thinks that your sign-in session expired even though it is active. To work around this issue, do one of the following:
Attempt to delete a project containing a
Users will not be able to delete a project if there are Jenkins builds associated with this project that are references in releases not in the project.
Attempt to delete a build from a pipeline run via
If you use the
These service catalog items are disabled because underlying plugin has been removed.
Single Sign on does not work unless PHP configuration is changed due to a security related request. Workaround: change
CloudBees CD/RO v10.1 introduced new triggers and an updated UI for them. Pre-v10.1 triggers will continue to work but there is no UI to review or run them.
Before using the export command to perform a full data export from the CD/RO database, delete any legacy definitions and references to
You can revert changes only for high-level design objects such as applications procedures procedure steps workflow definitions and state definitions.
Enabling Recursively Traverse Group Hierarchy might impact system performance when the LDAP group hierarchy is traversed. The amount of impact varies with the configurations of the CloudBees CD/RO and LDAP servers the depth of group hierarchy in the LDAP server and the network latency between the servers. Make sure that your directory provider can handle the additional load for supporting nested group hierarchy traversal.
System performance might decrease if you disable change tracking at the server level and then re-enable it. (Change tracking is enabled by default.) For details about using change tracking see change tracking.