CloudBees CD/RO v10.1.1

CloudBees CD/RO v10.1.1 is a maintenance release (MR) applied to the v10.1.0 long-term release. As such, consult the following in combination with these notes for everything that this release provides.

CloudBees CD/RO v10.1.0 release notes, released 2021-02-26

Breaking change: Configuration parameter changes related to proxy settings

New configuration parameters to support HTTPS proxy setting have been added for the CloudBees CD/RO server, agent, and repository server as of v10.1.

Existing configuration parameters related to HTTP proxy settings have been renamed to make it clear that they pertain to HTTP. The pre-v10.1 names are deprecated and no longer recognized by CloudBees CD/RO. See renamed configuration parameters to learn how these parameters may affect your migration to v10.1.1. [NMB-29472, NMB-30155]

Security fixes

This release includes the following security updates

The Apache web server used in CloudBees CD/RO: no updates. Support version remains at v2.4.46.
PHP is upgraded from 7.4.13 to 7.4.16. For details, see https://www.php.net/releases/7_4_16.php. [BEE-3525]
OpenSSL is upgraded from 1.1.1i to 1.1.1k. [BEE-3525]

New features

None.

Database support: No changes.

Browser support: No changes.

Plugin enhancements

New plugins

EC-AWS-EC2: Replaces EC-EC2

Deprecated plugins

EC-EC2: Replaced by EC-AWS-EC2

Updated plugins

EC-Nexus 1.1.2: Plugin now correctly retrieves the version from .tar.gz files.

Plugin support changes

None.

New platform support

This section lists new platform support. For the current list of supported platforms, consult Supported platforms for CloudBees CD/RO

Server support: No changes.

Agent support: No changes.

CloudBees CD/RO on Kubernetes: No changes.

Database support: No changes.

Browser support: No changes.

Resolved issues

NMB-30590

CloudBees CD/RO UI My Work auto-refresh for a user in a large number of groups generates a huge DB query for that slows all other DB queries, which causes long CD Server restart.

NMB-30579

Problem resolved with CloudBees CD application link not working correctly from Okta dashboard.

NMB-30566

Background deleter was removing session still in use.

NMB-30564

Updated a KB link that gets displayed under a MySQL database issue

NMB-30543

Use of transaction block to runProcedure in a Service Catalog context does not work and fails with NPTR in CD 10.1.

NMB-30296

No jobStep logs are viewable from UI after NullPointerException in agent. ServerRequestCompletionCallback.onError.

NMB-30208

SAML credential wasn’t converted at time of upgrade to cd v10.0.

NMB-30131

Error in generated DSL if a formalParameter contains a single quote char.

NMB-29304

PHP Deprecated: Array and string offset access syntax with curly braces is deprecated notices in apache error.log after upgrade to PHP v7.4.6.

CEV-28029

Main menu links not working as expected.

CEV-28024

Fix the SearchGuard permissions for scroll. Multiple hours delay from exponental backoff when processing sendReportingData requests that fail with HTTP 400 errors in updating pipeline tags.

CEV-27948

Running pipelines is offering the already run tasks detail as link but not redirecting to any url.

CEV-25526

Problem resolved with preconditions not working with FlowRuntime properties.

Behavior changes

New default installation path

As of CloudBees CD/RO v10.1, the default installation paths for new installations have changed. You can change these defaults at installation time. Default paths for update installations are not affected. [CBSDA-258]

See Default installation directories to find out how this affects your enterprise and what action you may need to take.

Installation notes

For complete installation and upgrade information, see CloudBees CD/RO installation guide.

Configuration parameter changes related to proxy settings

Configuration parameters related to HTTP proxy settings have been renamed to make it clear that they pertain to HTTP. The pre-v10.1 names are deprecated and no longer recognized by CloudBees CD/RO. This table lists old and new names. Full list of configuration parameters can be found here. [NMB-29472, NMB-30155]

Pre-v10.1 name	v10.1 name	Description
--agentEnableProxySettings	--agentEnableHttpProxySettings=<1\|0>	Enable (1) or disable (0) the HTTP proxy server configuration. If enabling for the first time, --agentHttpProxyHost and --agentHttpProxyPort must be specified.
--agentNoProxyHosts	--agentNonProxyHosts=hosts	Comma delimited list of hosts that should be reached directly, bypassing the HTTP/HTTPS proxy servers.
--agentProxyHost	--agentHttpProxyHost=host	The IP address of the HTTP proxy server.
--agentProxyPort	--agentHttpProxyPort=port	The port of the HTTP proxy server.
--serverEnableProxySettings	--serverEnableHttpProxySettings=<1\|0>	Enable (1) or disable (0) the HTTP proxy server configuration. If enabling for the first time, --serverHttpProxyHost and --serverHttpProxyPort must be specified.
--serverProxyHost	--serverHttpProxyHost=host	The IP address of the HTTP proxy server.
--serverProxyPort	--serverHttpProxyPort=port	The port of the HTTP proxy server.
--serverNoProxyHosts	--serverNonProxyHosts=hosts	Comma delimited list of hosts that should be reached directly, bypassing the HTTP/HTTPS proxy servers.
--repositoryNoProxyHosts	--repositoryNonProxyHosts=hosts	Comma delimited list of hosts that should be reached directly, bypassing the proxy server.

Pre-v10.1 name

v10.1 name

Description

--agentEnableProxySettings

--agentEnableHttpProxySettings=<1|0>

Enable (1) or disable (0) the HTTP proxy server configuration. If enabling for the first time, --agentHttpProxyHost and --agentHttpProxyPort must be specified.

--agentNoProxyHosts

--agentNonProxyHosts=hosts

Comma delimited list of hosts that should be reached directly, bypassing the HTTP/HTTPS proxy servers.

--agentProxyHost

--agentHttpProxyHost=host

The IP address of the HTTP proxy server.

--agentProxyPort

--agentHttpProxyPort=port

The port of the HTTP proxy server.

--serverEnableProxySettings

--serverEnableHttpProxySettings=<1|0>

Enable (1) or disable (0) the HTTP proxy server configuration. If enabling for the first time, --serverHttpProxyHost and --serverHttpProxyPort must be specified.

--serverProxyHost

--serverHttpProxyHost=host

The IP address of the HTTP proxy server.

--serverProxyPort

--serverHttpProxyPort=port

The port of the HTTP proxy server.

--serverNoProxyHosts

--serverNonProxyHosts=hosts

Comma delimited list of hosts that should be reached directly, bypassing the HTTP/HTTPS proxy servers.

--repositoryNoProxyHosts

--repositoryNonProxyHosts=hosts

Comma delimited list of hosts that should be reached directly, bypassing the proxy server.

Legacy services applications and container entities

The legacy Services applications and Traditional applications with containers have been deprecated in this version of CloudBees CD/RO. We recommend upgrading your applications to the current microservices application model.

CloudBees CD/RO on Kubernetes

Sample CloudBees CD/RO server and agent Helm chart values, found here, provide CloudBees’s default installation values. The CloudBees CD/RO images.tag value associated with version 10.1.1 is 10.1.1.148196_2.1.22_20210513.

Gateway agents

For gateway agents v10.1 and higher, you must configure the CloudBees CD/RO server IP address.

Navigate to Administration Server settings.
Select the System Settings from the left-hand menu.
Scroll to the Server IP address entry and add the fully qualified domain name of your CloudBees CD/RO server’s load balancer.

This setting controls how agents contact the CloudBees CD/RO server when they send results from job steps and similar prompts.

For more information about gateways, see Zones and gateways

Configuring autostart services for Linux installations: Linux installations that you perform as a non-root user or without sudo permissions cannot automatically start the CloudBees CD/RO server, web server, repository server, or agents. This means that you must set up service autostart after installation is complete. Learn more here.

Upgrading your CloudBees CD/RO environment

IMPORTANT: Before starting an upgrade, make sure to back up your existing CloudBees CD/RO data.

Upgradable versions

Upgrades to CloudBees CD/RO 10.x are supported only from ElectricCommander 5.0 or any version before 9.0. Upgrades to version 10.x from version 4.2 or earlier are not supported. For upgrade instructions, see Upgrade on traditional platforms.

Upgrading an older CloudBees Analytics version

Upgrading from a CloudBees CD version 9.0.x , or earlier, requires upgrading the CloudBees Analytics server to 10.0. If upgrading from CloudBees CD version 9.1, then upgrading version 9.1 the CloudBees Analytics server is not required.

Updating elements containing applicationServiceMapping [CEV-16237 and CEV-16158]

If your XML export file from CloudBees CD 8.0.1 or earlier versions has elements containing applicationServiceMapping, you must change all instances of that string in the file to serviceClusterMapping before importing the file into version 10.0. For example, change the following XML:

<applicationServiceMapping>
  <applicationServiceMappingId>9efcda31-a85f-11e7-8500-0800279f198d</applicationServiceMappingId>
  <applicationServiceMappingName>9efcda31-a85f-11e7-8500-0800279f198d</applicationServiceMappingName>
  …
</applicationServiceMapping>▼

<serviceClusterMapping>
  <serviceClusterMappingId>9efcda31-a85f-11e7-8500-0800279f198d</serviceClusterMappingId>
  <serviceClusterMappingName>9efcda31-a85f-11e7-8500-0800279f198d</serviceClusterMappingName>
  …
</serviceClusterMapping>▼

Backing up and restoring custom settings

The CloudBees Analytics installer overwrites the elasticsearch.yml configuration file with a new file. As of CloudBees Analytics version 8.3, the file includes a Custom Settings section, which lets you add Elasticsearch settings not managed by the CloudBees Analytics server without being lost during an upgrade. If you added settings to this file in version 8.2 or earlier that you want to preserve, you must back up the file to a separate location before upgrading to version 9.2 or newer versions and then add the settings to the Custom Settings section after the upgrade. During future upgrades, the installer will preserve the settings in the Custom Settings section. [NMB-25850]

Updating the MySQL configuration before upgrading

Since release 8.0.1, CloudBees has instructed customers using a MySQL database to use the following two lines in their MySQL configuration:

init_connect='SET collation_connection = utf8_unicode_ci, NAMES utf8'
skip-character-set-client-handshake

Before upgrading CloudBees CD/RO, you must remove these lines or comment them out. Otherwise, jobs will not start.

Ensuring the correct default MySQL default collation

Since release 8.0.1, CloudBees has instructed customers using a MySQL database to ensure that the default collation for their database schema is set to utf8_unicode_ci or utf8_general_ci and that no table in their schema overrides this. As of release 9.0, the CloudBees CD/RO server checks this configuration on startup and logs errors in the server log if it is not set correctly.

If the collation is not configured correctly, then entering non-ASCII text into CloudBees CD/RO might cause errors. For example, setting a release name to a non-ASCII value and attempting a search causes an exception.

If your MySQL database schema or any tables in it are set to a non-UTF-8 collation order, see Knowledge Base article KBEC-00385 - Converting a MySQL Database From Latin-1 to UTF-8 for detailed instructions about safely converting your schema to UTF-8. [NMB-26521, NMB_27459]

Upgrading agents that run the ec-groovy job step in multizone deployments

In multizone CloudBees CD/RO deployments, CloudBees CD/RO agents that are in a different zone than the CloudBees CD/RO server must be upgraded to version 9.0 or later for the ec-groovy job step to run successfully on those agents. You must also upgrade the gateway agents that lead back to the server’s zone including those in any zones in between the agent’s zone and the server’s zone. [NMB-27490]

For details about multiple zones and gateway agents, refer to Zones and gateways.

Removing the SSL 2.0 Client Hello or SSLv2Hello protocol from your security configurations

CloudBees recommends removing the SSL 2.0 Client Hello or SSLv2Hello protocol from your security configurations for all components. [NMB-27934, NMB-29326]

Upgrade agents older that fall into this category for security reasons:
- Windows, Linux: 6.0.3 or older; 6.2 or older
- Mac OS: 8.4 or older

If this warning appears on the Automation Platform UI:

Note: We recommend removing `SSL 2.0 Client Hello` format from server configuration and upgrade older agents as indicated on the Cloud/Resources Page to avoid security risk.▼

then enter the following command on the CloudBees CD/RO server:

$ ecconfigure --serverTLSEnabledProtocol=TLSv1.2▼

Upgrading the CloudBees Analytics server

This section provides information about upgrading the CloudBees Analytics server from Version 7.3 to Version 10.1.

Re-Specifying configuration settings

The installers (GUI, interactive console, and silent mode) for the CloudBees Analytics server do not preserve the configuration setting for the CloudBees Analytics server host name ( --hostName ) or the setting for the Elasticsearch number of shards ( --elasticsearchNumberOfShards ) during the upgrade from 7.3 to 9.2. If you specified non-default values during the 7.3 Reporting server installation, you must re-specify these settings during the upgrade. (All other settings are preserved.)

CloudBees Analytics server configuration notes

For a production environment, CloudBees recommends that you install the CloudBees Analytics server on a system other than systems running other CloudBees CD/RO components (such as the CloudBees CD/RO server, web server, repository server, or agent). If you must install it on the same system (such as for testing or other non-production or trial-basis situations) see CloudBees Analytics server with other components for details.

Configuration notes

Performing a full import

During a full import, the import operation might hang in the following scenarios. To import successfully into CloudBees CD/RO 8.0 and newer versions, perform the appropriate workarounds [CEV-15447, CEV-11873]:

A manual process step in a process has formal parameters. The workaround is to remove the entry related to the property sheet for the job step that is associated with the manual process step.
In the exported XML file from the earlier release, two pipelines are in different projects, and both pipelines have no gate tasks. The flow associated with the pipeline is duplicated under both projects. The workaround is to remove the flow element under the projects.

Limitations: When an application is cloned from one project (the original project) to another (the destination project), the tier maps for the application will point to the environments with the same names in the destination project. To deploy the application to the environments in the original project, you must create tier maps connecting the application to those environments.

Known issues

NMB-24734

SyncArtifactVersions procedure completes with success, rather than showing a warning, when manifest is missing and overwrite = false.

NMB-24949

When you use the Automation Platform UI to upload and publish artifact files with non-English characters in their file names the operation fails with the following error: Upload file: Exit code 1: ERROR: Publish failure: Unexpected retrieval exception for repository error .

NMB-26021

Modifications of LDAP user data (such as email addresses) on an Active Directory server after registration in CloudBees CD/RO do not appear properly in user details (in the Automation Platform UI, the Deploy UI, or ectool ) until the CloudBees CD/RO server is restarted.

NMB-26962

(Windows platforms only) If the Elasticsearch cluster which is used by CloudBees Analytics is in the red state (in Elasticsearch this means that it only partly functions and some data is unavailable) then upgrade reconfigure or uninstall operations will not work. Because the Elasticsearch service can not be stopped when a cluster is in red state kill the Elasticsearch service process by the task manager before running the installer for these actions.

NMB-28135

The Microsoft Edge browser does not work with SAML 2.0 and a self-signed certificate during redirection from the identity provider to the service provider. Edge is not recommended for login via SAML 2.0.

NMB-28380

Can’t ignore server mismatch and override passkey from Database Configuration page.

NMB-28381

The ec-groovy command line utility fails when invoking createArtifactVersion with error This server cannot handle version <x> messages.

NMB-29391

The warning, WARNING: An illegal reflective access operation has occurred is logged by Java 11 for the cglib library.

NMB-29486

The LANG environment variable must be set to en.US.UTF-8; otherwise, the upgrade fails. See KBEC-00452 - Error installing CloudBees CD 10.0.x when Lang environment variable is different than en.US.UTF-8 for details.

NMB-30263

In some cases, job step diagnostic information is not available and server reports 507 error, Missing sessionId attribute in request, when trying to view job step logs.

CEV-11106

When an application with snapshots created in CloudBees CD/RO 6.1 or earlier is cloned and a project containing this application is imported to CloudBees CD/RO 6.3 or higher the import operation fails.

CEV-12363

Error prompts for runtimes started by a schedule are not visible if the schedule was created with a missed configuration.

CEV-12429

The stage inclusion status in the Release Dashboard changes color after a stage is renamed.

CEV-14689

No error prompt appears for failed tasks and retry tasks during a pipeline runtime.

CEV-15122

If an application process step cannot expand to its child steps (because of an invalid run condition or an invalid formal parameter) then the step is not retried even if it uses "retry on error" error handling. The job eventually completes with an error.

CEV-15829

The retry count for group tasks or rules using "automated retry on error" is missing from the Pipeline runtime page.

CEV-16245

Multiple mapped environments with the same name from different projects are not supported in email notifications.

CEV-16250

A project import might not include the path-to-production view.

CEV-16930

Jobs might not appear upon drill-down into the "Clusters With Most Deployments" widget in the CloudBees Analytics Microservices Dashboard if the service does not contain a deploy step in the process.

CEV-17164

When you do a full import from version 8.0 to version 8.2 or newer and two or more releases have the same name (under different projects) and are associated to the same pipeline then after import the runs for all releases might become associated to the first imported release. This is because CloudBees CD/RO cannot differentiate runs between the releases since all runs are under the same pipeline project and have the same name. To work around this issue rename releases in the export file so that all their occurrences (in deploymentHistoryItem flowRuntime and so on) are unique.

CEV-18531

All subreleases of a release must appear before the release in the DSL for the release-to-subrelease link to be created.

CEV-19239 CEV-19259

The ability to search by assignee in a Deployment Report is not available in the CloudBees Analytics report editor.

CEV-21426

If Release Command Center was setup for JIRA for user-stories and defects and the JIRA project name was mapped to the release project name using the following field mapping: ` projectName:releaseProjectName` then before upgrading to 10.0 the field mapping must be updated to mention the actual release project name using the following field mapping format: "release-project-name-in-CloudBees CD/RO":releaseProjectName

CEV-22379

Long custom labels in email notifications do not render correctly.

CEV-23624

Approval by email on manual tasks should not expect parameters.

CEV-22644

Navigation to a sub-release editor takes user to the parent release editor. As a workaround, select the subrelease from the left-hand navigation in the parent’s release editor.

CEV-23957

When you use the Deploy UI to edit a resource pool and add a tag while renaming it at the same time, the operation fails with the following error: Resource pool 'oldPoolName' does not exist . To work around this issue, rename the resource pool, then save the change, and then add the tag to the resource pool.

CEV-23958

Running an application process with a parallel manual application process step or running an application process with a parallel manual application and component process steps fails to delete the project.

CEV-23960

If you are signed in to the Deploy UI and upgrade to CloudBees CD/RO 10.0, the version 10.0 sign-in page for the Automation Platform UI goes into an infinite redirect. This is because the version 10.0 Automation Platform UI thinks that your sign-in session expired even though it is active. To work around this issue, do one of the following:

Before upgrading, sign out from the Deploy UI and the Automation Platform UI.
After upgrading, sign out from the Deploy UI and then sign in.
After upgrading, clean your browser cache and cookies to kill the pre-upgrade sign-in session.

As documented at KBEC-00443 - Login page goes to infinite redirect after upgrading Flow to 10.0

CEV-24712

Attempt to delete a project containing a CIBuildDetails object (injected by Jenkins or CloudBees CI) results in the error: The DELETE statement conflicted with the REFERENCE constraint.

CEV-24710

Users will not be able to delete a project if there are Jenkins builds associated with this project that are references in releases not in the project.

CEV-24617

Attempt to delete a build from a pipeline run via buildname and flowRuntimeId results in the error deleted object would be re-saved by cascade.

CEV-25150

If you use the ectool export to export your system configuration from a previous release and then use ectool import to import the same configuration to a CloudBees CD 10.0 server, some out-of-the-box content introduced in the releases since the version from which the full export was done, such as new or updated plugins, new catalog items, and persona based menu items, may be missing in the CD server UI. It is recommended to use ectool export and ectool import only between servers at the same version.

CEV-25400

These service catalog items are disabled because underlying plugin has been removed.

Azure Container Service Discovery
Amazon ECS Dynamic Cluster
GCE Dynamic Cluster

CEV-26700

Single Sign on does not work unless PHP configuration is changed due to a security related request. Workaround: change session.cookie_samesite to "Strict" in /opt/electriccloud/electriccommander/apache/conf/php.ini and restart web server.

CEV-28073

Because the default installation directories have changed in v10.1, running a jobStep may fail with the following error:

Step error [AgentFailedCreateWorkspace]: unable to run the command on the agent The agent was unable to create the workspace directory: Error in mkdir(/opt/electriccloud): Permission denied▼

When using a v10.1 agent with a CloudBees CD/RO server version earlier than 10.1 or with a CloudBees CD/RO server upgraded to 10.1 or newer, the default workspace is defined as /opt/electriccloud/electriccommander/workspace (Linux) or <%COMMON_APPDATA%>\Electric Cloud\ElectricCommander\workspace (Windows), which does not exist any longer on the agent system.

Workaround: Until this is handled in a future release, if you are going to register a v10.1, or newer, Linux agent with a CD Server older than v10.1, create a /opt/electriccloud/electriccommander/workspace folder on the agent system and give the agent service user permissions to read, write and, execute permissions on this folder. Similarly for Windows agents create a <%COMMON_APPDATA%>\Electric Cloud\ElectricCommander\workspace folder and give the agent service user the read, write, and execute permissions on this folder.
When using an agent version earlier than v10.1, with a v10.1 CloudBees CD/RO server, where the CloudBees CD/RO server is a fresh install rather than an upgrade, the default workspace folder is defined as /opt/cloudbees/sda/workspace, which does not exist on the agent system.

Workaround: Until this is handled in a future release, if you are going to register a agent older than v10.1 with a v10.1 or newer CD Server, where the CD server is a fresh install and not an upgrade, create a /opt/cloudbees/sda/workspace (Linux) or <%COMMON_APPDATA%>\CloudBees\Software Delivery Automation\workspace (Windows) folder on the agent system and give the agent service user read, write, and execute permissions on this folder.

If the CloudBees CD/RO server is upgraded to v10.1 or newer from version older than v10.1 no action is needed unless or until you receive the above workspace directory error.

N/A

You can revert changes only for high-level design objects such as applications procedures procedure steps workflow definitions and state definitions.

Restarting the CloudBees CD/RO server while new records are created for all tracked objects might take at least as long as an export or import of all projects (10 to 40 minutes for a large project).

N/A

Enabling Recursively Traverse Group Hierarchy might impact system performance when the LDAP group hierarchy is traversed. The amount of impact varies with the configurations of the CloudBees CD/RO and LDAP servers the depth of group hierarchy in the LDAP server and the network latency between the servers. Make sure that your directory provider can handle the additional load for supporting nested group hierarchy traversal.

N/A

System performance might decrease if you disable change tracking at the server level and then re-enable it. (Change tracking is enabled by default.) For details about using change tracking see change tracking.