CloudBees CD/RO v10.4.0

Released: 2022-01-31

Software Delivery Automation CD/RO v10.4

CloudBees is pleased to announce the CloudBees CD/RO 10.4 LTS release of the Software Delivery Automation platform. With this release, we added several new features and system improvements, including:

  • CloudBees Analytics DORA dashboards

  • GitOps SCM to Server Sync

  • Support for PostgreSQL database

  • Data Retention for CI objects

  • New Java agent on macOS for x86

Security fixes

This release includes the following security updates

  • PHP is upgraded from 7.4.22 to 7.4.26. For details, refer to https://www.php.net/releases/7_4_26.php. [BEE-14512]

  • OpenSSL is upgraded from 1.1.1k to 1.1.1l. [BEE-14512]

  • Apache web server is upgraded from 2.4.48 to 2.4.51. For details, refer to https://httpd.apache.org/download.cgi#apache24. [BEE-14512]

  • TLS v1.3 support update TLS v1.3 support is now available for all CloudBees CD Server components as well as the CD agents. The following PERL based CloudBees CD/RO plugins have been updated to use the cb-perl and support TLS v1.3: EC-JIRA, EC-Jenkins, EC-Servicenow. All Groovy based plugins support TLS v1.3. The only limitation is that PERL based plugins that use the older ec-perl do NOT support TLS v1.3.

New features

CloudBees Analytics DORA dashboards

The DevOps Research and Assessment metrics are a popular way of assessing an organization’s software delivery effectiveness. This release includes the dashboards and guidance on how to collect the necessary information to populate the four reports on these dashboards.

For more information, refer to DORA metrics.

Feature enhancements

PostgreSQL support added

CloudBees CD/RO now features support for PostgreSQL database version 12 and 13. PostgreSQL is a robust database management system that includes a large set of the SQL standard and features robust support for complex queries and multiversion concurrency, as well as extensibility features, including:

  • adding new data types

  • creating custom functions

  • using custom operators

Source code synchronization API and UI

CloudBees CD/RO now includes support for source code synchronization using domain-specific language (DSL) to store, manage and author their automation as code, while also maintaining version control of the code in a source code management (SCM) tool. Using this feature, teams can:

  • Modify DSL code and make commits to the SCM, which automatically applies the DSL changes to the server.

New Java agent on Mac x86

A new CloudBees CD/RO Java agent is now available for macOS on x86. This new agent on Mac fixes the following issues:

  • Job step output can be viewed in the UI for mac agents (BEE-9570)

  • Local (disconnected) workspaces are now supported for macOS (NMB-18791)

In addition, Mac agents also support proxy agents as well as gateways.

Microservice support for YAML based deployments

CloudBees CD/RO now includes support for deploying microservices using YAML based deployments. Both YAML and helm based deployments can be configured in the CloudBees CD/RO user interface and can be used to deploy to kubernetes clusters on various cloud providers.

Data retention includes CI build information

CloudBees CD/RO now includes CloudBees CI builds as a supported object in data retention strategies. Both the API and UI have been updated to include CloudBees CI build objects (data, deployments, pipeline runs, and releases) for storage and archive, which enhances data retention strategies for performance, compliance, and analytics benefits.

Installing a plugin from a file

You can now install jar files in the CloudBees CD/RO UI. From the Installed plugins tab on the Plugin management page, install the plugin from a file that is either on the CloudBees CD/RO server host or from a URL.

Other features and enhancements

  • The Reason field for retrieving the passwords from Cyberark CCP now is configurable. This field will be audited in the Credential Provider audit log.

  • Added support for external proxies (squid/nginx). To use external proxy, the Server IP address server property should be set.

  • A customizable banner can now be displayed. The setting for this is available in the Server setting.

Performance improvements

ActiveMQ performance update

ActiveMQ has been updated from 2.18 to 2.64 and features performance improvements, including:

  • Replace Last-Value-Queue (LVQ) usage with ring-queues, which improves throughput

  • Increase thread pool and batch sizes ins Quartz

  • Speed up bootstrap of systems with a lot of schedules

  • Reduce the amount of paging in ActiveMQ by reducing AMQ222038 warnings

New ActiveMQ option in wrapper.conf

A new wrapper.conf configuration option has been added that allows larger ActiveMQ memory allocations for large CloudBees CD/RO deployments:

wrapper.java.additional.99=-DCOMMANDER_ACTIVEMQ_GLOBAL_MAX_SIZE=5368709120
▼
New importDslFromGitNew procedure added to EC-DslDeploy

A new procedure called importDslFromGitNew has been added to EC-DslDeploy which is based on the EC-Git plugin. This procedure replaces the importDslFromGit procedure that used the ECSM-Git plugin which was deprecated and subsequently removed from new version of CloudBees CD/RO installers.

Database performance improvements

The performance of the CloudBees CD/RO platform has been improved significantly when using both the MySQL and MS-SQL databases when compared with previous versions.

Plugin enhancements

CloudBees CD/RO plugin catalog

The CloudBees CD/RO plugin catalog is available on the main product documentation site.

Plugin updates

Plugin and version

Changes

EC-AWS-EC2 1.0.7

Third party libraries

EC-Bitbucket 1.2.0

BEE-8588 Three procedures were made public such as commit, Approving pull and creating PRs

EC-GCP-ComputeEngine 2.5.5

Third Party libraries

EC-Git 1.10.2

BEE-8187 Support git log report

Added proxy support.

Upgrade Jackson-Databind to v2.13.1.

Updated the Source Provider procedure to provide metadata to the Report Inventory Procedure.

EC-FeatureFlags 1.2.2

Upgraded Jackson-Databind to v2.13.1

EC-FileOps 2.1.1

Fixed a bug when the "Move" has been cleaning up destination folder if source is a file.

EC-Helm 1.5.0

Enabled support of new configurations.

EC-Kubectl 1.3.0

Enabled support of new configurations.

Added Microservices support.

Added proxy support.

Upgrade Jackson-Databind to v2.13.1.

Updated the Source Provider procedure to provide metadata to the Report Inventory Procedure.

EC-Github 4.3.3

BEE-8177 Added procedures Get Files, Upload Files, Set commit status, Create release, create pull request.

Add Reviewers field for procedure CreatePullRequest containing list of usernames and team names to request review for a PR.

Upgrade Jackson-Databind to v2.13.1.

SetupWebhook procedure to support triggers for scmSync.

EC-JIRA 2.0.2

BEE-7453 Upgrade Perl (cb-perl) which enables TLS1.3 support.

Added "ignore SSL errors" flag and allowed accepting self-signed certificates.

EC-Nexus 1.2.2

FLOWPLUGIN-7968 Upgrade Perl

Fixed downloading snapshots from Nexus v3 repository maven2.

EC-Rest 2.2.2

BEE-5264 Added file download support, Support for Bearer access Token, Option to set timeout for rest calls.

Authorize using custom prefix/header and reduce authentication visibility.

Added an "Ignore SSL issues" flag.

Upgraded Jackson-Databind to v2.13.1

EC-SendEmail 1.0.6

FLOWPLUGIN-7868 Exchange Server bug.

EC-ServiceNow 3.0.1

BEE-7452 Upgrade Perl.

The minimum required CloudBees CD version has been set to 10.3.

Trigger-Property 1.1.0 (new)

FLOWPLUGIN-9510 Bundle Trigger Property Plugin.

The plugin allows configuring CloudBees CD/RO to poll the specified source control repository for updates with a preconfigured schedule. If an update is detected, the pipeline or release object will run.
Plugin support changes

The following plugins are deprecated and are no longer supported with CloudBees CD/RO.

ECSCM-Git Replaced by EC-Git, EC-GitHub, and EC-Bitbucket

Resolved issues

BEE-10167

Fixed issue of DSL exporting large properties as an expression concatenated string.

NMB-31049

To avoid massive number of SQL requests to the DB, cache groups that automatically verified on existence and created in case of their absence during AD/LDAP login operation.

BEE-14567

Changed the text message in console mode installation to ask user if they want to "install or upgrade" software instead of just "install", alleviating confusion to those who want to upgrade.

BEE-14510

An issue was resolved where ectool and PERL API’s not working across reverse proxies when using cb-perl.

BEE-14483

An issue where using search function did not display all matching pipelines has been addressed by increasing the limit of pipelines shown from 20 to 50. The search has also been enhanced to allow search for more specific occurrences across all pipelines without limiting to the set displayed initially.

BEE-14472

The Last Run Status in the CloudBees Analytics release report object is now being populated from the status of the last pipeline that was run for that Release.

BEE-14361

Problems publishing to the AWS Artifact Manager in CloudBees CD version 10.3 and earlier versions have been addressed by upgrading the AWS SDK client from version 1.10.60 to 1.12.136 in this release.

BEE-13344

Added support for evalDSL --clientFiles to work across external proxy (squid/nginx).

BEE-12675

The issue of sending emails with an empty body without --html argument when using ectool sendMail has been fixed in this release.

BEE-12515

Performance issues with plugin procedure usage statistics have been addressed.

BEE-10580

Fixed issue with procedure renaming that caused breaking of references to a procedure from another procedure and/or application process.

BEE-10484

Truncated reporting output parameters sent to DevOps Insight to decrease its size and impact on ActiveMQ and ElasticSearch.

BEE-10454

A problem has been corrected where the User Interface would incorrectly show (for example) "Passwords should be equal” message for optional empty fields.

BEE-10420

The problem of not being able to set a resource pool for a procedure from the User Interface has been addressed.

BEE-10418

A problem of slow display on the UI when navigating to the Event logs page has been fixed by improving the performance of countObjects/findObjects API calls.

BEE-10416

A problem with the upgrade of a CloudBees Analytics component (DevOps Insight) has been addressed.

BEE-10409

A problem has been addressed where getProperty in args.each { conf DSL section was not working.

BEE-10408

An issue has been fixed where running evalDsl or createPipeline API multiple times creates duplicate pipelines in the same project.

BEE-10407

The functionality of customizing the CloudBees Platform User Interface has been restored in the Commander UI.

BEE-10404

Fixed the "No credentials/session found in this request" error when installing a local jar plugin file using the plugin manager UI.

BEE-10403

The problem of missing information in custom personas during CloudBees CD/RO server upgrade has been addressed.

BEE-10205

The problem with displaying duration in milliseconds for all bar graph widgets in CloudBees Analytics server has been addressed and it is now displayed in time format.

BEE-10136

Fixed a communication issue with proxy requests between agent and server, agent and gateway in multi-zone environments when using Gateways.

BEE-9828

The problem of advisory messages being mixed in with API output has been addressed. By default advisory messages are suppressed in the ectool output.

BEE-9570

Job step logs in local workspace are now accessible through the Web UI for Mac Agents.

BEE-9133

The problem of selecting a manual task parameter in the standard view of a pipeline or release has been addressed.

Behavior changes

  • Importing Domain Specific Language (DSL) from Git service catalog now uses the EC-Git plugin and new plugin configurations, replacing the ECSCM-Git plugin that is marked as deprecated as of the CloudBees CD/RO 10.4 release.

    To support backward compatibility, CloudBees has not removed the legacy procedure using the EC-DslDeploy plugin that uses the ECSCM-Git plugin. Customers that have ECSCM-Git plugin and want to continue using it can still use it.

  • The DSL Editor now supports editing resources.

  • The PDK package is now self contained including all of its dependencies.

Installation notes

For complete installation and upgrade information, refer to CloudBees CD/RO installation guide.

Legacy services applications and container entities

In CloudBees CD/RO v10.3, the legacy Services applications and Traditional applications with containers were deprecated and removed. Before you upgrade to CloudBees CD/RO v10.3 or later, you must migrate your applications to the current microservices application model.

Also, before upgrading from CloudBees CD/RO v10.2 or earlier you must delete all legacy services and containers. This will prevent upgrade failure, a database consistency break or inability to run the validateDatabase API.

CloudBees CD/RO on Kubernetes

Sample CloudBees CD/RO server and agent Helm chart values, found here, provide CloudBees’s default installation values. The CloudBees CD/RO images.tag value associated with version 10.4 is 10.4.0.153077_3.0.47_20220126.

Upgrading gateway agents

All gateway agents that meet these criteria must be updated to CloudBees CD/RO v10.2+:

  • Your enterprise implements a multi-zone environment.

  • Agent versions are a combination of pre-v10.2 and v10.2+.

  • The access route to a v10.2+ agent is configured through a pre-v10.2 gateway agent.

Configuring autostart services for Linux installations

Linux installations that you perform as a non-root user or without sudo permissions cannot automatically start the CloudBees CD/RO server, web server, repository server, or agents. This means that you must set up service autostart after installation is complete. Learn more here.

Upgrading your CloudBees CD/RO environment

IMPORTANT: Before starting an upgrade, make sure to back up your existing CloudBees CD/RO data.

Upgradable versions

Upgrades to CloudBees CD/RO 10.x are supported only from ElectricCommander 5.0. For upgrade instructions, refer to Upgrade on traditional platforms.

Updating the MySQL configuration before upgrading

Since release 8.0.1, CloudBees has instructed customers using a MySQL database to use the following two lines in their MySQL configuration:

init_connect='SET collation_connection = utf8_unicode_ci, NAMES utf8'
skip-character-set-client-handshake

Before upgrading CloudBees CD/RO, you must remove these lines or comment them out. Otherwise, jobs will not start.

Ensuring the correct default MySQL default collation

Make sure that the default collation for the MySQL database schema is set to utf8_unicode_ci or utf8_general_ci and that no table in the schema overrides this. The CloudBees CD/RO server checks this configuration on startup and logs errors in the server log if it is not set correctly.

If the collation is not configured correctly, then entering non-ASCII text into CloudBees CD/RO might cause errors. For example, setting a release name to a non-ASCII value and attempting a search causes an exception.

If your MySQL database schema or any tables in it are set to a non-UTF-8 collation order, refer to Knowledge Base article KBEC-00385 - Converting a MySQL Database From Latin-1 to UTF-8 for detailed instructions about safely converting your schema to UTF-8. [NMB-26521, NMB_27459]

Upgrading agents that run the ec-groovy job step in multizone deployments

In multizone CloudBees CD/RO deployments, CloudBees CD/RO agents that are in a different zone than the CloudBees CD/RO server must be upgraded to version 9.0 or later for the ec-groovy job step to run successfully on those agents. You must also upgrade the gateway agents that lead back to the server’s zone including those in any zones in between the agent’s zone and the server’s zone. [NMB-27490]

For details about multiple zones and gateway agents, refer to Zones and gateways.

Removing the SSL 2.0 Client Hello or SSLv2Hello protocol from your security configurations

CloudBees recommends removing the SSL 2.0 Client Hello or SSLv2Hello protocol from your security configurations for all components. [NMB-27934, NMB-29326]

  1. Upgrade agents older that fall into this category for security reasons:

    • Windows, Linux: 6.0.3 or older; 6.2 or older

    • Mac OS: 8.4 or older

  2. If this warning appears on the Automation Platform UI:

    Note: We recommend removing `SSL 2.0 Client Hello` format from server configuration and upgrade older agents as indicated on the Cloud/Resources Page to avoid security risk.
    ▼

    then enter the following command on the CloudBees CD/RO server:

    $ ecconfigure --serverTLSEnabledProtocol=TLSv1.2
    ▼
Upgrading the CloudBees Analytics server

This section provides information about upgrading the CloudBees Analytics server.

Potential breaking change: Elasticsearch update

The Elasticsearch version shipped with CloudBees Analytics version 10.2 has been updated from v6.6.2 to v7.10.2. As such, this update may create breaking changes in your custom reports. All changes related to the new version are described in Elasticsearch documentation. The following change may effect your reports the most. [BEE-2717]

  • Accessing missing document values throws an error. The doc['field'].value throws an exception if the document is missing a value for the field field.

    To check if a document is missing a value, you can use doc['field'].size() == 0.

  • It is not possible to upgrade CloudBees Analytics version 9.0.1 and below to CloudBees Analytics version 10.2.0 and above. Installer exits with an error and an appropriate message when such an update is attempted. If user needs to upgrade CloudBees Analytics version 9.0.1 and below, then user must first upgrade to a version between 9.1.0 and 10.1.0, or 9.0.2 and above. After that, the user can upgrade CloudBees Analytics to version 10.3.0 or higher. [NMB-31030]

  • For previous CloudBees Analytics upgrades from version 9.0.1 and below: CloudBees Analytics data may contain obsolete indices that are incompatible with CloudBees Analytics version 10.2.0 and above. To work correctly, it is necessary to re-index these indexes before an upgrade. The installer prompts the user to do this before upgrading.

    • In console mode and Ui mode, the installer displays the following prompt if outdated indexes are detected:

      One or more Elasticsearch indices were created in an obsolete version of Elasticsearch.
These indexes must be re-indexed for the upgrade to be successful.

Do you want to start the reindexation? [n/Y]
      ▼

      After an affirmative answer, the installer automatically re-indexes and continues the upgrade.

    • In silent mode, the installer reindexes automatically.

  • Backing up and restoring custom settings

    The CloudBees Analytics installer overwrites the elasticsearch.yml configuration file with a new file. This file includes a Custom Settings section, which lets you add Elasticsearch settings not managed by the CloudBees Analytics server without being lost during an upgrade. The installer preserves the settings in the Custom Settings section. [NMB-25850]

  • Upgrading CloudBees Analytics clusters

    The principle of forming a cluster in CloudBees Analytics has changed in v10.2 due to the update of Elasticsearch v7.10.2. In this regard, an additional action is required to upgrade to CloudBees Analytics v10.2 or later:

    When updating the first master node, the user must explicitly specify that it is the first node to be updated. If this action is not performed, a cluster being updated is placed out of service.

    All installers have been instrumented to accommodate this change. For details, refer to Upgrade the CloudBees Analytics server. [BEE-2717]

  • CloudBees Analytics server configuration notes

    For a production environment, CloudBees recommends that you install the CloudBees Analytics server on a system other than systems running other CloudBees CD/RO components (such as the CloudBees CD/RO server, web server, repository server, or agent). If you must install it on the same system (such as for testing or other non-production or trial-basis situations), refer to CloudBees Analytics server with other components for details.

CloudBees CI operations center configurations

After upgrading to CloudBees CD/RO v10.4 from v10.0.x, your CloudBees CI operations center configurations may need to be reworked.

  • In v10.0.x, CloudBees CI operations center URLs specified in configurations are silently appended at runtime with the /cjoc path component.

  • In v10.1, URLs are used as defined in configurations. The /cjoc component is not appended.

In order to maintain pre-v10.1 runtime compatibility, the v10.1 upgrade process modifies CloudBees CI operations center URLs in existing configurations by hardcoding the /cjoc path component. You need to rework existing URLs in configurations where appending the /cjoc path component is inappropriate.

Configuration notes

Performing a full import

During a full import, the import operation might hang in the following scenarios. To import successfully into CloudBees CD/RO 8.0 and newer versions, perform the appropriate workarounds [CEV-15447, CEV-11873]:

  • A manual process step in a process has formal parameters. The workaround is to remove the entry related to the property sheet for the job step that is associated with the manual process step.

  • In the exported XML file from the earlier release, two pipelines are in different projects, and both pipelines have no gate tasks. The flow associated with the pipeline is duplicated under both projects. The workaround is to remove the flow element under the projects.

Limitations

When an application is cloned from one project (the original project) to another (the destination project), the tier maps for the application point to the environments with the same names in the destination project. To deploy the application to the environments in the original project, you must create tier maps connecting the application to those environments.

Known issues

Issue number

Description of Issue

BEE-14902

The fix introduced by BEE-10167 splits escaped single quotes into two different strings, separated by the Groovy concatenation operator.

BEE-13922

If you encounter unexpected SSL errors related to communications with CloudBees CD server while using ectool utility, you can use a legacy perl version for ectool. It can be achieved by patching the wrapper script with the following command:

$ sed -i '/SRCDIR=/a [ $BASE = ectool ] && PERLDIR="$ECHOME/perl-legacy"' <install directory>/bin/ec-perl
▼
This workaround does not work on Windows agents.

BEE-14581

MeanLeadTime reports working incorrectly when Elastic search only has pipeline runs but no release runs

BEE-14889

The UI add a new line when performing a Save Changes on Source code synchronization, causing Include objects to not be applied.

BEE-10612

CI build detail are not be present after project import.

BEE-7512

With CloudBees CD/RO v 10.2.1 and earlier, the DSL Import service catalog fails for grouped tasks.

NMB-30095

Browser redirects to port 2080 during first navigation to CD deployed from SDA and Flow Helm charts.

NMB-24734

SyncArtifactVersions procedure completes with success, rather than showing a warning, when manifest is missing and overwrite = false.

NMB-24949

When you use the Automation Platform UI to upload and publish artifact files with non-English characters in their file names the operation fails with the following error: Upload file: Exit code 1: ERROR: Publish failure: Unexpected retrieval exception for repository error .

NMB-26021

Modifications of LDAP user data (such as email addresses) on an Active Directory server after registration in CloudBees CD/RO do not appear properly in user details (in the Automation Platform UI, the Deploy UI, or ectool) until the CloudBees CD/RO server is restarted.

NMB-26962

(Windows platforms only) If the Elasticsearch cluster, which is used by CloudBees Analytics, is in the red state (in Elasticsearch this means that it only partly functions and some data is unavailable) then upgrade reconfigure or uninstall operations will not work. Because the Elasticsearch service can not be stopped when a cluster is in red state kill the Elasticsearch service process by the task manager before running the installer for these actions.

NMB-28135

The Microsoft Edge browser does not work with SAML 2.0 and a self-signed certificate during redirection from the identity provider to the service provider. Edge is not recommended for sign-in via SAML 2.0.

NMB-29486

The LANG environment variable must be set to en.US.UTF-8; otherwise, the upgrade fails. For details, refer to KBEC-00452 - Error installing CloudBees CD/RO 10.0.x when Lang environment variable is different than en.US.UTF-8.

CEV-11106

When an application with snapshots created in CloudBees CD/RO 6.1 or earlier is cloned and a project containing this application is imported to CloudBees CD/RO 6.3 or higher the import operation fails.

CEV-12363

Error prompts for runtimes started by a schedule are not visible if the schedule was created with a missed configuration.

CEV-12429

The stage inclusion status in the Release Dashboard changes color after a stage is renamed.

CEV-14689

No error prompt appears for failed tasks and retry tasks during a pipeline runtime.

CEV-15122

If an application process step cannot expand to its child steps (because of an invalid run condition or an invalid formal parameter) then the step is not retried even if it uses "retry on error" error handling. The job eventually completes with an error.

CEV-15829

The retry count for group tasks or rules using "automated retry on error" is missing from the Pipeline runtime page.

CEV-16245

Multiple mapped environments with the same name from different projects are not supported in email notifications.

CEV-16250

A project import might not include the path-to-production view.

CEV-16930

Jobs might not appear upon drill-down into the "Clusters With Most Deployments" widget in the CloudBees Analytics Microservices Dashboard if the service does not contain a deploy step in the process.

CEV-18531

All subreleases of a release must appear before the release in the DSL for the release-to-subrelease link to be created.

CEV-19239 CEV-19259

The ability to search by assignee in a Deployment Report is not available in the CloudBees Analytics report editor.

CEV-21426

If Release Command Center was setup for JIRA for user-stories and defects and the JIRA project name was mapped to the release project name using the following field mapping: ` projectName:releaseProjectName` then before upgrading to 10.0 the field mapping must be updated to mention the actual release project name using the following field mapping format: "release-project-name-in-CloudBees CD/RO":releaseProjectName

CEV-23624

Approval by email on manual tasks should not expect parameters.

CEV-25150

If you use the ectool export to export your system configuration from a previous release and then use ectool import to import the same configuration to a CloudBees CD/RO 10.0 server, some out-of-the-box content introduced in the releases since the version from which the full export was done, such as new or updated plugins, new catalog items, and persona based menu items, may be missing in the CD server UI. It is recommended to use ectool export and ectool import only between servers at the same version.

CEV-26700

Single Sign on does not work unless PHP configuration is changed due to a security related request. Workaround: change session.cookie_samesite to "Strict" in /opt/electriccloud/electriccommander/apache/conf/php.ini and restart web server.

CEV-28704

CloudBees CD/RO v10.1 introduced new triggers and an updated UI for them. Pre-v10.1 triggers will continue to work but there is no UI to review or run them.

CEV-28779

Before using the export command to perform a full data export from the CloudBees CD/RO database, delete any legacy definitions and references to service objects from applications and releases.

N/A

You can revert changes only for high-level design objects such as applications procedures procedure steps workflow definitions and state definitions.

Restarting the CloudBees CD/RO server while new records are created for all tracked objects might take at least as long as an export or import of all projects (10 to 40 minutes for a large project).

N/A

Enabling Recursively Traverse Group Hierarchy might impact system performance when the LDAP group hierarchy is traversed. The amount of impact varies with the configurations of the CloudBees CD/RO and LDAP servers the depth of group hierarchy in the LDAP server and the network latency between the servers. Make sure that your directory provider can handle the additional load for supporting nested group hierarchy traversal.

N/A

System performance might decrease if you disable change tracking at the server level and then re-enable it. (Change tracking is enabled by default.) For details about using change tracking, refer to change tracking.