Software Delivery Automation CD/RO v10.3
CloudBees is pleased to announce the CloudBees CD/RO 10.3 LTS release of the Software Delivery Automation platform. In this release, CloudBees continues the effort to bring CloudBees CI and CloudBees CD/RO into a single platform, with:
-
Unified authentication
-
CloudBees Analytics integration with CloudBees Feature Management
-
Introduction of CloudBees CI license usage reporting
-
Improvements to both CloudBees CI and CloudBees CD/RO audit reports
In addition, platform enhancements such as improved database performance, simplified multi-zone deployment in Kubernetes, and plugin configuration and management are part of this long term supported release.
Security fixes
- This release includes the following security updates
-
-
PHP is upgraded from 7.4.16 to 7.4.22. For details, refer to https://www.php.net/releases/7_4_22.php. [NMB-30951]
-
OpenSSL remains at version 1.1.1k. [BEE-3557]
-
Apache web server is upgraded from 2.4.46 to 2.4.48. For details, refer to https://httpd.apache.org/download.cgi#apache24. [NMB-30951]
-
Elasticsearch is upgraded from v7.10.2 to v7.13.4.
-
Logstash is upgraded from v7.12.1 to v7.13.4.
-
New features
- Custom Docker images
-
CloudBees CD/RO component binaries and associated Dockerfiles are available for download to use in creating custom Docker images. [BEE-6485]
Documentation: Create custom Docker images.
- Single sign-on with OpenID Connect
-
Support for OpenID Connect identity provider is added. [BEE-5484]
Documentation: Configuring single sign-on with OpenID Connect
- CloudBees Analytics integration with CloudBees Feature Management
-
The Release Command Center dashboard (RCC) includes a new column that integrates with CloudBees Feature Management. The Feature Flags column shows metrics associated with feature flags configured for the current release. The CloudBees Software Delivery Automation EC-FeatureFlags plugin sources data from CloudBees Feature Management, which, in turn, supplies data to widgets in this new column. You configure the desired set of feature flags for your CloudBees Software Delivery Automation release from CloudBees Feature Management. You must have a CloudBees Feature Management license in order to source feature flag data.
Documentation: Release Command Center dashboard
- Plugin configurations as code
-
Plugin configurations are now available as first class objects that can be created and managed from a CloudBees CD/RO project via REST APIs, DSL, and the
ectool
command line interface. Access to each of the new plugin configurations can be managed and controlled using access control lists.Documentation: Create plugin configurations
If you are migrating to CloudBees CD/RO v10.3 or later from a pre-v10.3 version, you must perform a one-time migration for each plugin configuration at your site. Refer to latest@cloudbees-cd:plugin-manager:plugin-config.adoc#plugin-config-migration for details. |
Feature enhancements
- Updated plugin manager
-
The plugin manager user interface, now called plugin management, has been updated and is now located from CloudBees CD/RO
.Documentation: Manage plugins
- Custom plugin catalog
-
Plugin management now includes an in-product version of the plugin catalog that can be customized for your CloudBees CD/RO environment. [BEE-7958]
Documentation: Manage the plugin catalog
- SSO improvements to remove directory provider requirements
-
SAML v2 support in CloudBees CD/RO is enhanced to provide an option to eliminate the need to set up a directory provider like Active Directory for authorization. Users can optionally choose to not configure Directory providers. When this option is selected, users are automatically registered upon authentication by the SSO identity provider.
Documentation: Configuring single sign-on with SAML
- Helm chart improvements for installation and configuration of gateway agents in a multi-zone deployment
-
The CloudBees CD/RO Helm chart has been enhanced to install and configure a pair of gateway agents as part of the product installation for a multi-zone deployment where some CD agents might be running outside the Kubernetes cluster and other agents might be inside the Kubernetes cluster. The CD agent Helm chart has also been enhanced to make the installation and configuration of gateway agents easier.
Documentation: Configure Helm charts
- Other features
-
-
Projects view on the CloudBees CD/RO UI now includes procedures. [CEV-28333]
-
The repository server is enhanced to use GCP storage buckets as backing stores in addition to AWS S3 buckets and NFS mounted filesystems.
-
The version of Perl used by the
ec-perl
command line tool has been upgraded from v5.8.9 to v5.32.1. [BEE-7521]
-
- Database support
-
No changes.
- Browser support
-
No changes.
Performance improvements
- MySQL scalability and performance optimizations
-
MySQL database scalability and performance on CloudBees CD/RO has been improved significantly when compared with previous versions. [NMB-30755]
-
Performance improvements for queries doing full table scans
-
Performance for pipeline executions improved an average of 66%.
-
CPU utilization now stays below 10%.
-
Plugin enhancements
- CloudBees CD/RO plugin catalog
-
The CloudBees CD/RO plugin catalog is available on the main product documentation site.
- Updated plugins
-
The following is a list of key changes made to existing plugins. For a complete list, refer to Bundled plugin report
Plugin |
Version |
Changes |
EC-Git |
1.7.0 |
These plugins have been updated to support plugin configurations as first class objects. The details of this feature are described here: Create plugin configurations. |
EC-Jenkins |
1.21.1 |
|
EC-JIRA |
1.10.1 |
|
EC-Nexus |
1.2.1 |
|
EC-Rest |
2.1.1 |
|
EC-SonarQube |
1.5.0 |
|
EC-Github |
4.1.1 |
|
EC-Docker |
2.0.0 |
Support removed for the legacy microservices model. This plugin along with EC-Helm can be used to support microservices applications and deployments. |
EC-FeatureFlags |
1.2.0 |
Added support for CloudBees Analytics release command center dashboard integration with CloudBees Feature Management. |
EC-AgentManagement |
1.5.3 |
Fixed CAM plugin code to handle Added correct error during attempt to upgrade proxy agent with CAM plugin. [NMB-30950] |
EC-Core |
1.3.0 |
Updated with new procedures for promote and demote plugins. [FLOWPLUGIN-9340] |
EC-FileOps |
2.1.0 |
Improved results of some procedures to make them easier to use. [FLOWPLUGIN-9247] |
EC-GCP-ComputeEngine |
2.5.3 |
Resolved an incompatibiliy issue in the |
EC-Helm |
1.4.0 |
Added Fixed an issue with microservice deployment failing when application and environment are in different projects. [FLOWPLUGIN-8996] Fixed up incorrect references to file and image names. [FLOWPLUGIN-9360] |
- Discontinued plugins
-
The following is a list of discontinued plugins:
EC-Kubernetes |
These plugins have been removed as they are no longer relevant. Use EC-Helm in their place |
EC-OpenShift |
|
ECSCM |
These plugins have been removed in favor of EC-Git, which can be used for most use cases. |
ECSCM-File |
|
ECSCM-Git |
|
ECSCM-Property |
|
EC-CIManager |
This plugin is no longer supported. |
- Plugin Development Kit enhancements
-
PDK 3.3.0.6 includes:
-
Updated to these components
-
Toolkit version: 3.3.0
-
Groovy core library version: 1.2.0
-
Perl core library version: 1.4.0
-
Layout version: 1.3.4
-
-
Support for the new plugin configuration as first-class objects.
-
Fixed expandable values in legacy plugin configurations.
-
New platform support
This section lists new platform support.
-
Support for Ubuntu 20.04 has been added. [NMB-31041]
- Agent support
-
-
Support for Ubuntu 20.04 has been added. [NMB-31041]
-
The 64-bit CloudBees CD/RO agent installer no longer includes 32-bit DLLs. [NMB-30954]
-
CloudBees CD/RO requires Rosetta2 to be installed on mac M1-based computers. As such, it will automatically be installed on demand by the installer. [NMB-30158]
-
- CloudBees CD/RO on Kubernetes
-
Support for Kubernetes versions 1.19x, 1.20.x, and 1.21.x has been added for GKE, Amazon EKS, and AKS. [BEE-8118]
Resolved issues
BEE-6473 |
CI URLs used in Native CI integration were not encoded according to HTTP 1.1 spec. |
BEE-6483 |
CloudBees CD K8s base images migrated from Ubuntu to Red Hat UBI v8. |
BEE-4921 |
In the CloudBees CD/RO and CloudBees Analytics Continuous Integration Builds screen, if a build had a Running status, the Project column may not have displayed the project information, and the project information was only correctly displayed once the build completed. This issue has been resolved. If a build has a Running status, the Project column now correctly displays project information before the build has completed. |
NMB-31015 |
ActiveMQ journal buffer size was not large enough to handle conditions of high load. This issue has been addressed by increasing the ActiveMQ journal buffer size to 1 MB. It is now configurable via the environment variable |
NMB-30983 |
Pipeline stage with |
NMB-30981 |
EC-DSLDeploy export was not generating command task |
NMB-30979 |
DSL schedule exports from the UI no longer produce null parameters. |
NMB-30956 |
Apache config changes lost after upgrading from CloudBees CD/RO 10.0.3 to any newer version. |
NMB-30941 |
Complex releases are failing with DSL export when suppress nulls were enabled in MySQL. |
NMB-30901 |
Multiple restarts of Zookeeper pods occured in Kubernetes due to Out Of Memory errors. The default Zookeeper memory limit has been increased to 1 Gi. |
NMB-30890 |
Exported DSL of plugin tasks contained in a task group failed to import properly. |
NMB-30882 |
Running |
NMB-30873 |
The |
NMB-30865 |
Optimised the ping resources operation called from |
NMB-30859 |
MasterComponent imported via DSL followed by save threw a |
NMB-30821 |
Fixed the limitation of setting |
NMB-30817 |
The |
NMB-30816 |
MySQL DB failures with |
NMB-30799 |
Moving a property folder under |
NMB-30798 |
Standalone utilities jars did not log error messages properly. Affected jars include:
|
NMB-30784 |
The CAM plugin code did’t handle null |
NMB-30772 |
EF-Utilities |
NMB-30741 |
The |
NMB-30736 |
CloudBees CD/RO occasionally failed to install on K8s with AWS EFS storage using CSI Driver. |
NMB-30735 |
The |
NMB-30704 |
Invoking |
NMB-30541 |
The job folders in workspace had wide permissions. |
NMB-30121 |
Creating a webhook with DSL for an invalid schedule caused all the webhooks to stop processing. |
NMB-30008 |
CloudBees CD/RO installation problem on AKS. |
NMB-27240 |
now indicates which server settings require restart and which do not. |
NMB-26535 |
Removed unused build artifacts from Apache’s distribution including |
CEV-28647 |
Build duration was published to CloudBees Analytics as a text field instead of a number. Starting with v10.3, the build duration report works with both text field and number. |
CEV-28737 |
Installing and uninstalling EC-DSLIDE plugin rendered system unusable. |
CEV-28705 |
Date fields were changing the value with up/down/left/right keys, even when the Date field was out of focus. |
CEV-28318, CEV-28654 |
The CloudBees Analytics dashboard parameter could not be passed as a parameter to the widget using the |
CEV-27498 |
JIRA storyPoints value was not published to CloudBees Analytics Release Command Center since it is not a number. |
CEV-28616 |
CloudBees CD/RO Installation in cluster mode on OpenShift fails because Zookeeper could not be installed. |
CEV-28519 |
Property picker in release: command task in pipeline was different between planning and running release. |
CEV-28512 |
EC-DslDeploy duplicated command task contents when tasks were contained in a task group. |
CEV-28461 |
Could not remove all users from the group in the UI. |
CEV-28460 |
Tags were not created with |
CEV-28457 |
Different UI requests were used when querying the release runs via view previous runs from Pipeline Runs vs Running release from the context menu. |
CEV-28453 |
The filter in the |
CEV-28452 |
Breadcrumb became stale when viewing the Application process of an application that was recreated. Starting with v10.3, the user is warned that the old application model does not exist or that the application has been recreated, instead of |
CEV-28451 |
After application was deployed and snapshotted and if the application was recreated, the UI did not allow rerun of an old application run. Starting with v10.3, the user is warned that the old application model does not exist or application model has been recreated. |
CEV-28449, CEV-28427 |
The CloudBees CD/RO UI did not show elapsed time taken for Stage and Gate tasks. |
CEV-28411 |
The |
CEV-28295 |
Couldn’t attach parameter credential to a component step and to a master component in the CloudBees CD/RO UI. |
CEV-28161 |
Custom validation check on credential parameter was not triggered when clicking outside of text box. |
CEV-28158 |
Service Catalog |
CEV-27773 |
UI did not clearly show if system defined personas were editable or read-only. |
CEV-27613 |
Procedure step editor lost context. Needed to cancel without saving and reopen to edit the step again. |
CEV-25735 |
URL link of Service Catalog item required a space after URL if pointing to SVG file. |
CEV-16254 |
|
CEV-10580 |
The Perl API did not consistently accept |
Installation notes
For complete installation and upgrade information, refer to CloudBees CD/RO installation guide.
- CloudBees CD/RO on Kubernetes
-
Sample CloudBees CD/RO server and agent Helm chart values, found here, provide CloudBees’s default installation values. The CloudBees CD/RO
images.tag
value associated with version 10.3 is10.3.0.150860_3.0.32_20210917
. - Configuring autostart services for Linux installations
-
Linux installations that you perform as a non-root user or without
sudo
permissions cannot automatically start the CloudBees CD/RO server, web server, repository server, or agents. This means that you must set up service autostart after installation is complete. Learn more here.
- Upgrading your CloudBees CD/RO environment
-
IMPORTANT: Before starting an upgrade, make sure to back up your existing CloudBees CD/RO data.
- Upgradable versions
-
Upgrades to CloudBees CD/RO 10.x are supported only from ElectricCommander 5.0. For upgrade instructions, refer to Upgrade on traditional platforms.
- Updating elements containing applicationServiceMapping [CEV-16237 and CEV-16158]
-
If your XML export file from CloudBees CD/RO 8.0.1 or earlier versions has elements containing
applicationServiceMapping
, you must change all instances of that string in the file toserviceClusterMapping
before importing the file into version 10.3. For example, change the following XML:<applicationServiceMapping> <applicationServiceMappingId>9efcda31-a85f-11e7-8500-0800279f198d</applicationServiceMappingId> <applicationServiceMappingName>9efcda31-a85f-11e7-8500-0800279f198d</applicationServiceMappingName> … </applicationServiceMapping>
to
<serviceClusterMapping> <serviceClusterMappingId>9efcda31-a85f-11e7-8500-0800279f198d</serviceClusterMappingId> <serviceClusterMappingName>9efcda31-a85f-11e7-8500-0800279f198d</serviceClusterMappingName> … </serviceClusterMapping>
- Updating the MySQL configuration before upgrading
-
Since release 8.0.1, CloudBees has instructed customers using a MySQL database to use the following two lines in their MySQL configuration:
init_connect='SET collation_connection = utf8_unicode_ci, NAMES utf8' skip-character-set-client-handshake
Before upgrading CloudBees CD/RO, you must remove these lines or comment them out. Otherwise, jobs will not start.
- Ensuring the correct default MySQL default collation
-
Make sure that the default collation for the MySQL database schema is set to
utf8_unicode_ci
orutf8_general_ci
and that no table in the schema overrides this. The CloudBees CD/RO server checks this configuration on startup and logs errors in the server log if it is not set correctly.If the collation is not configured correctly, then entering non-ASCII text into CloudBees CD/RO might cause errors. For example, setting a release name to a non-ASCII value and attempting a search causes an exception.
If your MySQL database schema or any tables in it are set to a non-UTF-8 collation order, refer to Knowledge Base article KBEC-00385 - Converting a MySQL Database From Latin-1 to UTF-8 for detailed instructions about safely converting your schema to UTF-8. [NMB-26521, NMB_27459]
- Upgrading agents that run the ec-groovy job step in multizone deployments
-
In multizone CloudBees CD/RO deployments, CloudBees CD/RO agents that are in a different zone than the CloudBees CD/RO server must be upgraded to version 9.0 or later for the
ec-groovy
job step to run successfully on those agents. You must also upgrade the gateway agents that lead back to the server’s zone including those in any zones in between the agent’s zone and the server’s zone. [NMB-27490]For details about multiple zones and gateway agents, refer to Zones and gateways.
- Removing the SSL 2.0 Client Hello or SSLv2Hello protocol from your security configurations
-
CloudBees recommends removing the
SSL 2.0 Client Hello
orSSLv2Hello
protocol from your security configurations for all components. [NMB-27934, NMB-29326]-
Upgrade agents older that fall into this category for security reasons:
-
Windows, Linux: 6.0.3 or older; 6.2 or older
-
Mac OS: 8.4 or older
-
-
If this warning appears on the Automation Platform UI:
Note: We recommend removing `SSL 2.0 Client Hello` format from server configuration and upgrade older agents as indicated on the Cloud/Resources Page to avoid security risk.
then enter the following command on the CloudBees CD/RO server:
$ ecconfigure --serverTLSEnabledProtocol=TLSv1.2
-
- Upgrading the CloudBees Analytics server
-
This section provides information about upgrading the CloudBees Analytics server.
-
Upgrading CloudBees Analytics clusters
The principle of forming a cluster in CloudBees Analytics has changed in v10.2 due to the update of Elasticsearch v7.10.2. In this regard, an additional action is required to upgrade to CloudBees Analytics v10.2 or later:
When updating the first master node, the user must explicitly specify that it is the first node to be updated. If this action is not performed, a cluster being updated is placed out of service.
All installers have been instrumented to accommodate this change. For more details, refer to Upgrade the CloudBees Analytics server. [BEE-2717]
-
CloudBees Analytics server configuration notes
For a production environment, CloudBees recommends that you install the CloudBees Analytics server on a system other than systems running other CloudBees Software Delivery Automation components (such as the CloudBees Software Delivery Automation server, web server, repository server, or agent). If you must install it on the same system (such as for testing or other non-production or trial-basis situations), refer to CloudBees Analytics server with other components for details.
Configuration notes
- Performing a full import
-
During a full import, the import operation might hang in the following scenarios. To import successfully into CloudBees CD/RO 8.0 and newer versions, perform the appropriate workarounds [CEV-15447, CEV-11873]:
-
A manual process step in a process has formal parameters. The workaround is to remove the entry related to the property sheet for the job step that is associated with the manual process step.
-
In the exported XML file from the earlier release, two pipelines are in different projects, and both pipelines have no gate tasks. The flow associated with the pipeline is duplicated under both projects. The workaround is to remove the flow element under the projects.
-
- Limitations
-
When an application is cloned from one project (the original project) to another (the destination project), the tier maps for the application point to the environments with the same names in the destination project. To deploy the application to the environments in the original project, you must create tier maps connecting the application to those environments.
Known issues
BEE-4921 |
In the CloudBees CD/RO and CloudBees Analytics Continuous Integration Builds screen, if a build has a Running status, the Project column may temporarily not display project information. Once the build has completed, the Project column correctly displays the project information for each build. |
||
BEE-7512 |
With CloudBees CD/RO v 10.2.1 and earlier, the DSL Import service catalog fails for grouped tasks. |
||
BEE-18208 |
The Server properties page in new CloudBees CD/RO UI is not available when a property is created without a value from the Platform UI. The workaround is to use Server properties in the Platform UI. |
||
NMB-30095 |
Browser redirects to port 2080 during first navigation to CD deployed from SDA and Flow Helmm charts. |
||
NMB-24734 |
|
||
NMB-24949 |
When you use the Automation Platform UI to upload and publish artifact files with non-English characters in their file names the operation fails with the following error: |
||
NMB-26021 |
Modifications of LDAP user data (such as email addresses) on an Active Directory server after registration in CloudBees CD/RO do not appear properly in user details (in the Automation Platform UI, the Deploy UI, or |
||
NMB-26962 |
(Windows platforms only) If the Elasticsearch cluster, which is used by CloudBees Analytics, is in the red state (in Elasticsearch this means that it only partly functions and some data is unavailable) then upgrade reconfigure or uninstall operations will not work. Because the Elasticsearch service can not be stopped when a cluster is in red state kill the Elasticsearch service process by the task manager before running the installer for these actions. |
||
NMB-28135 |
The Microsoft Edge browser does not work with SAML 2.0 and a self-signed certificate during redirection from the identity provider to the service provider. Edge is not recommended for sign-in via SAML 2.0. |
||
NMB-28380 |
Can’t ignore server mismatch and override passkey from Database Configuration page. |
||
NMB-28381 |
The |
||
NMB-29391 |
The warning, |
||
NMB-29486 |
The LANG environment variable must be set to |
||
NMB-30263 |
In some cases, job step diagnostic information is not available and server reports 507 error, |
||
CEV-11106 |
When an application with snapshots created in CloudBees CD/RO 6.1 or earlier is cloned and a project containing this application is imported to CloudBees CD/RO 6.3 or higher the import operation fails. |
||
CEV-12363 |
Error prompts for runtimes started by a schedule are not visible if the schedule was created with a missed configuration. |
||
CEV-12429 |
The stage inclusion status in the Release Dashboard changes color after a stage is renamed. |
||
CEV-14689 |
No error prompt appears for failed tasks and retry tasks during a pipeline runtime. |
||
CEV-15122 |
If an application process step cannot expand to its child steps (because of an invalid run condition or an invalid formal parameter) then the step is not retried even if it uses "retry on error" error handling. The job eventually completes with an error. |
||
CEV-15829 |
The retry count for group tasks or rules using "automated retry on error" is missing from the Pipeline runtime page. |
||
CEV-16245 |
Multiple mapped environments with the same name from different projects are not supported in email notifications. |
||
CEV-16250 |
A project import might not include the path-to-production view. |
||
CEV-16930 |
Jobs might not appear upon drill-down into the "Clusters With Most Deployments" widget in the CloudBees Analytics Microservices Dashboard if the service does not contain a deploy step in the process. |
||
CEV-17164 |
When you do a full import from version 8.0 to version 8.2 or newer and two or more releases have the same name (under different projects) and are associated to the same pipeline then after import the runs for all releases might become associated to the first imported release. This is because CloudBees CD/RO cannot differentiate runs between the releases since all runs are under the same pipeline project and have the same name. To work around this issue rename releases in the export file so that all their occurrences (in |
||
CEV-18531 |
All subreleases of a release must appear before the release in the DSL for the release-to-subrelease link to be created. |
||
CEV-19239 CEV-19259 |
The ability to search by assignee in a Deployment Report is not available in the CloudBees Analytics report editor. |
||
CEV-21426 |
If Release Command Center was set up for JIRA for user-stories and defects and the JIRA project name was mapped to the release project name using the following field mapping: ` projectName:releaseProjectName` then before upgrading to 10.0 the field mapping must be updated to mention the actual release project name using the following field mapping format: |
||
CEV-22379 |
Long custom labels in email notifications do not render correctly. |
||
CEV-23624 |
Approval by email on manual tasks should not expect parameters. |
||
CEV-22644 |
Navigation to a sub-release editor takes user to the parent release editor. As a workaround, select the subrelease from the left-hand navigation in the parent’s release editor. |
||
CEV-23957 |
When you use the Deploy UI to edit a resource pool and add a tag while renaming it at the same time, the operation fails with the following error: |
||
CEV-23958 |
Running an application process with a parallel manual application process step or running an application process with a parallel manual application and component process steps fails to delete the project. |
||
CEV-23960 |
If you are signed in to the Deploy UI and upgrade to CloudBees CD/RO 10.0, the version 10.0 sign-in page for the Automation Platform UI goes into an infinite redirect. This is because the version 10.0 Automation Platform UI thinks that your sign-in session expired even though it is active. To work around this issue, do one of the following:
|
||
CEV-24712 |
Attempt to delete a project containing a |
||
CEV-24710 |
Users will not be able to delete a project if there are Jenkins builds associated with this project that are references in releases not in the project. |
||
CEV-24617 |
Attempt to delete a build from a pipeline run via |
||
CEV-25150 |
If you use the |
||
CEV-25400 |
These service catalog items are disabled because underlying plugin has been removed.
|
||
CEV-26700 |
Single Sign on does not work unless PHP configuration is changed due to a security related request. Workaround: change |
||
CEV-28704 |
CloudBees CD/RO v10.1 introduced new triggers and an updated UI for them. Pre-v10.1 triggers will continue to work but there is no UI to review or run them. |
||
CEV-28779 |
Before using the export command to perform a full data export from the CD/RO database, delete any legacy definitions and references to |
||
N/A |
You can revert changes only for high-level design objects such as applications procedures procedure steps workflow definitions and state definitions.
|
||
N/A |
Enabling Recursively Traverse Group Hierarchy might impact system performance when the LDAP group hierarchy is traversed. The amount of impact varies with the configurations of the CloudBees CD/RO and LDAP servers the depth of group hierarchy in the LDAP server and the network latency between the servers. Make sure that your directory provider can handle the additional load for supporting nested group hierarchy traversal. |
||
N/A |
System performance might decrease if you disable change tracking at the server level and then re-enable it. (Change tracking is enabled by default.) For details about using change tracking, refer to change tracking. |