CloudBees CD/RO v10.5.0

Released: April 28, 2022

CloudBees CD/RO v10.5

CloudBees is pleased to announce the CloudBees CD/RO 10.5 LTS release of the Software Delivery Automation platform. With this release, we added several new features and system improvements, including:

  • Self-service catalog enables deployment models with Ansible Tower

  • Component version added to the path-to-production view

  • A new global Environment Inventory view

  • Scheduled data retention rules

  • DevOps Insight backup and restore

  • Support for Kubernetes version 1.22x and OpenShift version 4.9.x has been added for GKE, Amazon EKS, and Azure AKS

  • Support for SSO for KeyCloak using OpenID Connect

Security fixes

This release includes the following security updates

  • The ZooKeeper version bundled with CloudBees CD/RO version 10.5 has been updated from v3.4.6 to v3.8.0. CloudBees CD/RO 10.5 requires ZooKeeper v3.8.0. For installation and upgrade instructions, refer to Install ZooKeeper and Upgrade a clustered environment.

  • Apache web server is upgraded from 2.4.52 to 2.4.53. For details, refer to https://httpd.apache.org/download.cgi#apache24. [BEE-14975]

  • PHP is upgraded from 7.4.27 to 7.4.28. For details, refer to https://www.php.net/releases/7_4_28.php. [BEE-14975]

  • OpenSSL is upgraded from 1.1.1m to 1.1.1n. [BEE-14975]

  • Jackson-databind is upgraded from 2.13.1 to 2.13.2.2. [BEE-17084]

  • Elasticsearch is upgraded to 7.17.2. [BEE-15174]

  • Logstash is upgraded to 7.17.2. [BEE-15174]

New features

Self-service catalog enables deployments with Ansible Tower

CloudBees CD/RO now features a self-service catalog item that allows you to create Ansible Tower deployment models. You can now use Ansible Tower playbooks for CloudBees CD/RO deployments, allowing you to track environment inventory, generate deployment analytics, and view staged artifacts in the path-to-production view.

For more information, refer to Ansible Tower plugin.

Global environment inventory view

A new global environment inventory view allows you to compare inventories across all of your environments. When you conduct troubleshooting or auditing, this feature lets you easily determine the drift, eliminating the need to create custom reports or individually compare environments.

For more information, refer to Global environment inventory.

Scheduled data retention rules

You can now create custom schedules for data retention rules. This allows you to spread out purge actions, minimizing system performance impact.

The default data retention service setting only purges archived objects after you re-run the service. If you set your custom data rentention policy to run once, the service does not purge your archived objects. As a workaround, run the Purge after archive operation on the global schedule, and disable the Purge after archive when run once option.

For more information, refer to Create object schedules.

Backup and restore of Elasticsearch snapshots of the CloudBees Analytics server on Kubernetes

The ability to create a backup of Elasticsearch indices using snapshots has been added to the CloudBees CD/RO Helm chart. This new capability enables you to protect your valuable analytics data.

The CloudBees Analytics server uses Elasticsearch as an analytics engine. The only reliable and supported way to back up Elasticsearch data is by taking a snapshot. You cannot back up Elasticsearch data by making copies of the data directories of its nodes. There are no supported methods to restore any data from a file system level.

For more information, refer to Maintain CloudBees Analytics server data on Kubernetes.

Feature enhancements

Component version added to the path-to-production view

The component version now displays in the path-to-production view grid.

For more information, refer to Path to production view.

Support for SSO for KeyCloak using OpenID Connect

Support has been added for SSO for KeyCloak using OpenID Connect.

For more information, refer to Configuring single sign-on with OpenID Connect.

Other features and enhancements

  • Added System Access Control to the main menu and made UI improvements. Select Administration  System Access Control. The page includes updated functionality, including search. For more information, refer to Access control.

  • Upgraded security to reduce potential vulnerabilities.

  • Improved email configurations management. For more information, refer to Notifications.

Performance improvements

None.

Plugin enhancements

CloudBees CD/RO plugin catalog

The CloudBees CD/RO plugin catalog is available on the CloudBees documentation site.

Plugin updates

Plugin and version

Changes

EC-Artifactory 1.6.0

Fixed potential vulnerabilities.

EC-ALM 1.2.5

Added user session validation.

EC-AWS-EC2 1.0.10

Fixed a potential vulnerability.

EC-Chef 1.2.6

Added user session validation.

EC-DslDeploy 4.1.1

Added an incremental import feature to increase efficiency.

EC-DBI 2.0.5

Added user session validation.

EC-FeatureFlags 1.2.3

Resolved an issue with dropdowns, upgraded Jackson-Databind to v2.13.2.2, upgraded third-party libraries, and fixed a potential vulnerability.

EC-FileOps 3.0.1

Fixed a potential vulnerability.

EC-GCloud 1.0.0

Updated documentation to clarify parameter names and the process to run commands.

EC-Gerrit 2.1.5

Added user session validation.

EC-Git 1.10.3

Updated documentation, resolved an issue with the JSON change list, fixed an issue with the changeLogDetails report, and upgraded Jackson-Databind to v2.13.2.2.

EC-Github 4.4.1

Updated documentation, fixed procedure issues, resolved an issue with requests that exceed the limit, added user session validation, and upgraded Jackson-Databind to v2.13.2.2.

EC-Glassfish 1.0.6

Added user session validation.

EC-Helm 1.5.1

Fixed an issue with the microservice.

EC-IIS 3.1.8

Added user session validation.

EC-JBoss 2.8.1

Added user session validation.

EC-Jenkins 2.0.1

Upgraded cd-perl, added support for the ignoreSSLErrors flag, resolved an authentication token issue, and added user session validation.

EC-Jira 2.0.4

Updated documentation and fixed an issue with updating subtasks with additional parameters.

EC-Maven 2.4.9

Added user session validation.

EC-MySQL 2.0.15

Added user session validation.

EC-Nexus 2.0.0

Fixed an issue causing component versions to fail.

EC-Nagios

Added user session validation.

EC-OpenStack 1.3.7

Added user session validation.

EC-Oracle 2.0.10

Added user session validation.

EC-PluginManager 1.9.2

Fixed a potential vulnerability.

EC-Rest 2.2.5

Fixed an issue with Jira responses, resolved a potential vulnerability, upgraded third-party libraries, and upgraded Jackson-Databind to v2.13.2.2.

EC-S3 1.1.3

Updated the Gradle project to replace deprecated Bintray references.

EC-Selenium 2.0.9

Added user session validation.

EC-SendEmail 1.0.7

Fixed a potential vulnerability and added user session validation.

EC-ServiceNow 3.1.0

Added user session validation.

EC-SQLServer 2.0.13

Added user session validation.

EC-Terraform 3.1.0

Fixed an issue with procedures.

EC-Tomcat 2.3.7

Fixed a potential vulnerability and added user session validation.

EF-Utilities 2.0.0

ACLs no longer reset to default settings when you upgrade CloudBees CD/RO.

EC-WebLogic 3.6.1

Updated documentation, fixed a potential vulnerability, resolved issues with procedures, and added user session validation.

EC-WebSphere 2.9.1

Added user session validation.
Plugin support changes

The following plugins are no longer supported with CloudBees CD/RO.

EC-DefectTracking

This plugin is no longer supported.

EC-ReportsEngine

This plugin is no longer supported.

ECSCM-Reports

This plugin is no longer supported.
Plugin Development Kit enhancements

PDK 3.7.0.9 includes:

  • Updated to these components

    • Toolkit version: 3.7.0

    • Groovy core library version: 1.2.3

    • Perl core library version: 1.4.1

    • Layout version: 1.3.5

  • Support for cb-perl.

  • Fixed issue with PATCH method payload in the REST plugin wizard.

  • Added autocomplete feature.

Resolved issues

BEE-4851

When you set an errorHandling value using the API that is not available in the CloudBees CD/RO UI (for example, ignore), the UI now displays the selected value.

BEE-9131

Fixed an issue that prevented approval emails from displaying correctly in Microsoft Outlook 2016.

BEE-10112

Updated the agent platform to be compatible with macOS Monterey 12.0.

BEE-10156

Added a new error message to inform you when a step cannot be deleted because it is part of an active job.

BEE-15007

Fixed DSL code that remained present after a procedure step command changed to a sub-procedure.

BEE-15055

If a user signs in to CloudBees CD/RO using Security Assertion Markup Language (SAML) single sign-on (SSO), and the user is also a member of local groups, the access control lists (ACLs) are now taken into account when checking the user’s permissions.

BEE-15277

A CloudBees CD/RO agent drive mapping error no longer occurs on Microsoft Windows when the agent user is part of a domain.

BEE-15359

Fixed an issue with service account credentials that affected CloudBees CI Pipeline jobs started with triggers.

BEE-15479

Added new input parameters to resolve an issue with timeouts when importing project DSL.

BEE-16063

Fixed an issue with REST API mapping to get parameters from jobs by ID.

BEE-16214

Fixed an issue with CGI scripts that prevented publishing the artifact version on Microsoft Windows from the UI.

BEE-16269

Upgrades to the CloudBees CD/RO server no longer fail with an InvalidSchema: Unable to validate the database schema: org.hibernate.exception.SQLGrammarException: could not execute statement error message.

BEE-16559

A NullPointerException no longer occurs when you edit or clone the Release Command Center dashboard.

BEE-16781

Fixed an issue in the CloudBees CD/RO UI that caused new artifacts that were created using Microsoft Windows to fail.

BEE-17049

Fixed an issued with DSL imports using --overwrite true that caused missing properties for objects not wrapped in project 'Default'.

BEE-17097

The process to clean a stalled job using the PostgreSQL database tool no longer fails.

BEE-17100

Fixed a configuration issue in the platform UI Plugin Manager for the following plugins: EC-ALM, EC-Chef, EC-DBI, EC-Gerrit, EC-Glassfish, EC-IIS, EC-JBoss, EC-Jenkins, EC-Maven, EC-MYSQL, EC-Nagios, EC-OpenStack, EC-Oracle, EC-Rally, EC-S3, EC-Selenium, EC-SendEmail, EC-ServiceNow, EC-SQLServer, EC-Tomcat, EC-WebLogic, and EC-WebSphere.

BEE-17255

Required parameter prompts have been removed for manual tasks in runtime when you select a failed action and provide values for parameters that are not required.

BEE-17538

Fixed an issue with a dropdown option that loaded options from a list containing "-".

Behavior changes

None.

Installation notes

For complete installation and upgrade information, refer to the CloudBees CD/RO installation guide.

Apache ZooKeeper required update

The ZooKeeper version bundled with CloudBees CD/RO version 10.5 has been updated from v3.4.6 to v3.8.0. CloudBees CD/RO 10.5 requires ZooKeeper v3.8.0. For installation and upgrade instructions, refer to Install ZooKeeper and Upgrade a clustered environment.

Legacy services applications and container entities

In CloudBees CD/RO v10.3, the legacy Services applications and Traditional applications with containers were deprecated and removed. Before you upgrade to CloudBees CD/RO v10.3 or later, you must migrate your applications to the current microservices application model.

Also, before upgrading from CloudBees CD/RO v10.2 or earlier you must delete all legacy services and containers. This will prevent upgrade failure, a database consistency break or inability to run the validateDatabase API.

CloudBees CD/RO on Kubernetes

Sample CloudBees CD/RO server and agent Helm chart values, found here, provide CloudBees’s default installation values. The CloudBees CD/RO images.tag value associated with version 10.5 is 10.5.0.154362__20220426.

Upgrading gateway agents

All gateway agents that meet these criteria must be updated to CloudBees CD/RO v10.2+:

  • Your enterprise implements a multi-zone environment.

  • Agent versions are a combination of pre-v10.2 and v10.2+.

  • The access route to a v10.2+ agent is configured through a pre-v10.2 gateway agent.

Configuring autostart services for Linux installations

Linux installations that you perform as a non-root user or without sudo permissions cannot automatically start the CloudBees CD/RO server, web server, repository server, or agents. This means that you must set up service autostart after installation is complete. Learn more here.

Upgrading your CloudBees CD/RO environment
Before starting an upgrade, make sure to back up your existing CloudBees CD/RO data.
Upgradable versions

Upgrades to CloudBees CD/RO 10.x are supported only from ElectricCommander 5.0. For upgrade instructions, refer to the Upgrade on traditional platforms.

Updating the MySQL configuration before upgrading

Since release 8.0.1, CloudBees has instructed customers using a MySQL database to use the following two lines in their MySQL configuration:

init_connect='SET collation_connection = utf8_unicode_ci, NAMES utf8'
skip-character-set-client-handshake

Before upgrading CloudBees CD/RO, you must remove these lines or comment them out. Otherwise, jobs will not start.

Ensuring the correct default MySQL default collation

Make sure that the default collation for the MySQL database schema is set to utf8_unicode_ci or utf8_general_ci and that no table in the schema overrides this. The CloudBees CD/RO server checks this configuration on startup and logs errors in the server log if it is not set correctly.

If the collation is not configured correctly, then entering non-ASCII text into CloudBees CD/RO might cause errors. For example, setting a release name to a non-ASCII value and attempting a search causes an exception.

If your MySQL database schema or any tables in it are set to a non-UTF-8 collation order, refer to Knowledge Base article KBEC-00385 - Converting a MySQL Database From Latin-1 to UTF-8 for detailed instructions about safely converting your schema to UTF-8. [NMB-26521, NMB-27459]

Upgrading agents that run the ec-groovy job step in multizone deployments

In multizone CloudBees CD/RO deployments, CloudBees CD/RO agents that are in a different zone than the CloudBees CD/RO server must be upgraded to version 9.0 or later for the ec-groovy job step to run successfully on those agents. You must also upgrade the gateway agents that lead back to the server’s zone including those in any zones in between the agent’s zone and the server’s zone. [NMB-27490]

For details about multiple zones and gateway agents, refer to Zones and gateways.

Removing the SSL 2.0 Client Hello or SSLv2Hello protocol from your security configurations

CloudBees recommends removing the SSL 2.0 Client Hello or SSLv2Hello protocol from your security configurations for all components. [NMB-27934, NMB-29326]

  1. Upgrade agents older that fall into this category for security reasons:

    • Microsoft Windows

    • Linux: 6.0.3 or older; 6.2 or older

    • macOS: 8.4 or older

  2. If this warning appears on the Automation Platform UI:

    Note: We recommend removing `SSL 2.0 Client Hello` format from server configuration and upgrade older agents as indicated on the Cloud/Resources Page to avoid security risk.
    ▼

    then enter the following command on the CloudBees CD/RO server:

    $ ecconfigure --serverTLSEnabledProtocol=TLSv1.2
    ▼
Upgrading the CloudBees Analytics server

This section provides information about upgrading the CloudBees Analytics server.

Potential breaking change: Elasticsearch update

The Elasticsearch version shipped with CloudBees Analytics version 10.2 has been updated from v6.6.2 to v7.10.2. As such, this update may create breaking changes in your custom reports. All changes related to the new version are described in Elasticsearch documentation. The following change may affect your reports the most. [BEE-2717]

  • Accessing missing document values throws an error. The doc['field'].value throws an exception if the document is missing a value for the field field.

    To check if a document is missing a value, you can use doc['field'].size() == 0.

  • It is not possible to upgrade CloudBees Analytics version 9.0.1 and below to CloudBees Analytics version 10.2.0 and above. The installer exits with an error and an appropriate message when such an update is attempted. If you need to upgrade CloudBees Analytics version 9.0.1 and below, you must first upgrade to a version between 9.1.0 and 10.1.0, or 9.0.2 and above. After that, you can upgrade CloudBees Analytics to version 10.3.0 or higher. [NMB-31030]

  • For previous CloudBees Analytics upgrades from version 9.0.1 and below: CloudBees Analytics data may contain obsolete indices that are incompatible with CloudBees Analytics version 10.2.0 and above. To work correctly, it is necessary to re-index these indexes before an upgrade. The installer prompts you to do this before upgrading.

    • In console mode and UI mode, the installer displays the following prompt if outdated indexes are detected:

      One or more Elasticsearch indices were created in an obsolete version of Elasticsearch.
These indexes must be re-indexed for the upgrade to be successful.

Do you want to start the reindexation? [n/Y]
      ▼

      After an affirmative answer, the installer automatically re-indexes and continues the upgrade.

    • In silent mode, the installer reindexes automatically.

  • Backing up and restoring custom settings

    The CloudBees Analytics installer overwrites the elasticsearch.yml configuration file with a new file. This file includes a Custom Settings section, which lets you add Elasticsearch settings not managed by the CloudBees Analytics server without being lost during an upgrade. The installer preserves the settings in the Custom Settings section. [NMB-25850]

  • Upgrading CloudBees Analytics clusters

    The principle of forming a cluster in CloudBees Analytics has changed in v10.2 due to the update of Elasticsearch v7.10.2. In this regard, an additional action is required to upgrade to CloudBees Analytics v10.2 or later:

    When updating the first master node, the user must explicitly specify that it is the first node to be updated. If this action is not performed, any cluster that is being updated is placed out of service.

    All installers have been instrumented to accommodate this change. Refer to Upgrade the CloudBees Analytics server for more details. [BEE-2717]

  • CloudBees Analytics server configuration notes

    For a production environment, CloudBees recommends that you install the CloudBees Analytics server on a system other than systems running other CloudBees CD/RO components (such as the CloudBees CD/RO server, web server, repository server, or agent). If you must install it on the same system (such as for testing or other non-production or trial-basis situations), refer to CloudBees Analytics server with other components for details.

    If your CloudBees Analytics server is configured with multiple nodes in a Kubernetes environment, you must pre-generate your certificates. For more information, refer to Install CloudBees CD/RO within Kubernetes.
CloudBees CI operations center configurations

After upgrading to CloudBees CD/RO v10.5 from v10.0.x, your CloudBees CI operations center configurations may need to be reworked.

  • In v10.0.x, CloudBees CI operations center URLs specified in configurations are silently appended at runtime with the /cjoc path component.

  • In v10.1, URLs are used as defined in configurations. The /cjoc component is not appended.

To maintain pre-v10.1 runtime compatibility, the v10.1 upgrade process modifies CloudBees CI operations center URLs in existing configurations by hardcoding the /cjoc path component. You need to rework existing URLs in configurations where appending the /cjoc path component is inappropriate.

Configuration notes

Performing a full import

During a full import, the import operation might hang in the following scenarios. To import successfully into CloudBees CD/RO 8.0 and newer versions, perform the appropriate workarounds [CEV-15447, CEV-11873]:

  • A manual process step in a process has formal parameters. The workaround is to remove the entry related to the property sheet for the job step that is associated with the manual process step.

  • In the exported XML file from the earlier release, two pipelines are in different projects, and both pipelines have no gate tasks. The flow associated with the pipeline is duplicated under both projects. The workaround is to remove the flow element under the projects.

Limitations

When an application is cloned from one project (the original project) to another (the destination project), the tier maps for the application point to the environments with the same names in the destination project. To deploy the application to the environments in the original project, you must create tier maps connecting the application to those environments.

Known issues

BEE-7512

With CloudBees CD/RO v 10.2.1 and earlier, the DSL Import service catalog fails for grouped tasks.

BEE-10612

CloudBees CI build details are not present after project import.

BEE-14581

The MeanLeadTime report does not work correctly when Elasticsearch only has pipeline runs but no release runs.

BEE-16705

CloudBees CI jobs time out when triggered, resulting in pipeline failures. Increase the COMMANDER_HTTP_TIMEOUT property.

BEE-17259

When a custom data rentention policy schedule is set to run once, the data is not purged after archiving. To purge data after archiving, use a repeat schedule or the global data retention setting.

BEE-17801

The getReleaseInventory API call does not return the application type and service name for environmentInventoryDetail or releaseinventoryDetail. The path-to-production view does not properly display an application’s microservice type because of this issue.

BEE-17812

The Preview tab does not display a correct list of objects to be archived for data retention rules.

BEE-18208

The Server properties page in the new CloudBees CD/RO UI is not available when a property is created without a value from the Platform UI. The workaround is to use Server properties in the Platform UI.

NMB-30095

Browser redirects to port 2080 during first navigation to CloudBees CD/RO deployed from CloudBees Software Delivery Automation and Flow Helm charts.

NMB-24734

SyncArtifactVersions procedure completes with success, rather than showing a warning, when manifest is missing and overwrite = false.

NMB-24949

When you use the Automation Platform UI to upload and publish artifact files with non-English characters in their file names, the operation fails with the following error: Upload file: Exit code 1: ERROR: Publish failure: Unexpected retrieval exception for repository error.

NMB-26021

Modifications of LDAP user data (such as email addresses) on an Active Directory server after registration in CloudBees CD/RO do not appear properly in user details (in the Automation Platform UI, the Deploy UI, or ectool) until the CloudBees CD/RO server is restarted.

NMB-26962

(Microsoft Windows platforms only) If the Elasticsearch cluster, which is used by CloudBees Analytics, is in the red state (in Elasticsearch this means that it only partly functions and some data is unavailable) then upgrade, reconfigure, and uninstall operations will not work. Since the Elasticsearch service cannot be stopped when a cluster is in a red state, you must stop the Elasticsearch service process from the task manager before running the installer for these actions.

NMB-28135

The Microsoft Edge browser does not work with SAML 2.0 and a self-signed certificate during redirection from the identity provider to the service provider. Edge is not recommended for sign-in via SAML 2.0.

NMB-29486

The LANG environment variable must be set to en.US.UTF-8; otherwise, the upgrade fails. Refer to KBEC-00452 - Error installing CloudBees CD/RO 10.0.x when Lang environment variable is different than en.US.UTF-8 for details.

CEV-11106

When an application with snapshots created in CloudBees CD/RO 6.1 or earlier is cloned and a project containing this application is imported to CloudBees CD/RO 6.3 or higher, the import operation fails.

CEV-12363

Error prompts for runtimes started by a schedule are not visible if the schedule was created with a missing configuration.

CEV-12429

The stage inclusion status in the Release Dashboard changes color after a stage is renamed.

CEV-14689

No error prompt appears for failed tasks and retry tasks during a pipeline runtime.

CEV-15122

If an application process step cannot expand to its child steps (because of an invalid run condition or an invalid formal parameter) then the step is not retried even if it uses "retry on error" error handling. The job eventually completes with an error.

CEV-15829

The retry count for group tasks or rules using "automated retry on error" is missing from the Pipeline runtime page.

CEV-16245

Multiple mapped environments with the same name from different projects are not supported in email notifications.

CEV-16250

A project import might not include the path-to-production view.

CEV-16930

Jobs might not appear upon drill-down into the "Clusters With Most Deployments" widget in the CloudBees Analytics Microservices Dashboard if the service does not contain a deploy step in the process.

CEV-18531

All subreleases of a release must appear before the release in the DSL for the release-to-subrelease link to be created.

CEV-19239 CEV-19259

The ability to search by assignee in a Deployment Report is not available in the CloudBees Analytics report editor.

CEV-21426

If Release Command Center was set up for JIRA for user stories and defects, and the JIRA project name was mapped to the release project name using the field mapping ` projectName:releaseProjectName`, then before upgrading to 10.0 the field mapping must be updated to mention the actual release project name using the following field mapping format: "release-project-name-in-CloudBees CD/RO":releaseProjectName

CEV-23624

Approval by email on manual tasks should not expect parameters.

CEV-25150

If you use the ectool export to export your system configuration from a previous release and then use ectool import to import the same configuration to a CloudBees CD/RO 10.0 server, some out-of-the-box content introduced in the releases since the version from which the full export was done, such as new or updated plugins, new catalog items, and persona based menu items, may be missing in the CloudBees CD/RO server UI. It is recommended to use ectool export and ectool import only between servers at the same version.

CEV-26700

SSO does not work unless PHP configuration is changed due to a security-related request. Workaround: Change session.cookie_samesite to "Strict" in /opt/electriccloud/electriccommander/apache/conf/php.ini and restart the web server.

CEV-28704

CloudBees CD/RO v10.1 introduced new triggers and an updated UI for them. Pre-v10.1 triggers will continue to work but there is no UI to review or run them.

CEV-28779

Before using the export command to perform a full data export from the CloudBees CD/RO database, delete any legacy definitions and references to service objects from applications and releases.

N/A

You can revert changes only for high-level design objects such as applications procedures, procedure steps, workflow definitions, and state definitions.

Restarting the CloudBees CD/RO server while new records are created for all tracked objects might take at least as long as an export or import of all projects (10 to 40 minutes for a large project).

N/A

Enabling Recursively Traverse Group Hierarchy might impact system performance when the LDAP group hierarchy is traversed. The amount of impact varies with the configurations of the CloudBees CD/RO and LDAP servers, the depth of group hierarchy in the LDAP server and the network latency between the servers. Make sure that your directory provider can handle the additional load for supporting nested group hierarchy traversal.

N/A

System performance might decrease if you disable change tracking at the server level and then re-enable it. Change tracking is enabled by default. For details about using change tracking, refer to change tracking.