Upgrade traditional CloudBees Analytics environments to OpenSearch

In CloudBees CD/RO v2024.06.0, CloudBees Analytics was upgraded from using Elasticsearch to OpenSearch. The data formats of these search engines are not fully compatible. To preserve data from your legacy CloudBees Analytics servers, you must migrate it to an updated server when upgrading from a v2024.03.0 or earlier to v2024.06.0 or later.

This page describes the general processes to test and upgrade legacy CloudBees Analytics servers:

Before you upgrade
Test upgrade process
Production upgrade process

In the following content, these servers are referenced as:

Source server: v2024.03.0 or earlier CloudBees Analytics servers using Elasticsearch.
Destination server: v2024.06.0 or later CloudBees Analytics servers using OpenSearch.

Before you upgrade

If you are upgrading from CloudBees CD/RO v2023.10.0 or earlier to v2024.06.0 or later, before upgrading, you must update specific role permissions for Elasticsearch in your current environment. This is required to successfully perform the upgrade and data migration. CloudBees has provided a script to automatically update the required permissions.

The specific role permissions are for the kibanauser user. This user was included with CloudBees Analytics installations to visualize data for environments using Kibana. If you do not update these permissions, the data migration from your previous Elasticsearch environment to the updated OpenSearch installation may fail.

Linux

To update the role permissions for Linux:

Download update-es-role-permissions.sh from CloudBees examples.

The following commands use the script name update-es-role-permissions.sh. If you change the name, update the commands with your script name.

If your environment does not use the default configuration values for the CloudBees Analytics server, update the following default values as needed:

ES_DIR_CONF_DEF="/opt/cloudbees/sda/conf/reporting/elasticsearch"
## Directory with DOIS configuration files. To use a different value, set ES_DIR_CONF.

ES_DIR_BINS_DEF="/opt/cloudbees/sda/reporting/elasticsearch/bin"
## Directory with DOIS binaries. To use a different value, set ES_DIR_BINS.

ES_JAVA_HOME_DEF="/opt/cloudbees/sda/reporting/jre"
## Java Home directory. To use a different value, set ES_JAVA_HOME.

ES_PORT_DEF=9300
## Transport port used for communication between nodes. To use a different value, set ES_PORT.

CONFIG_DIR_TMP_DEF="/tmp"
## Temporary folder created to patch configuration files. To use a different value, set CONFIG_DIR_TMP.

### Example to run script with new values:
ES_DIR_CONF=/new/install/dir ES_PORT=9302 ./update-es-role-permissions.sh▼

CONFIG_DIR_TMP is a temporary directory (default /tmp and then /var/tmp) used to patch configuration files. If your system has any restrictions against these paths, ensure you assign a new path to the CONFIG_DIR_TMP variable. If you run the script without updating the variable, and neither directory is writable, the script fails with a message instructing you to assign a writable path to CONFIG_DIR_TMP.

Example:

CONFIG_DIR_TMP=/new/dir/tmp ./update-es-role-permissions.sh▼

This script must be executed by the system user assigned to run CloudBees Analytics services in your environment. More specifically, the user that is assigned to the reporting directory in the installation directory. To determine this user, navigate to your CloudBees Analytics installation directory (default /opt/cloudbees/sda), and run:
```
ls -la▼
```
- If you are not the user assigned to run these services, you can use:
  sudo su <user-assigned-to-reporting-directory>
  ▼
Navigate to the directory with the update-es-role-permissions.sh script.
Run the following command to make the script executable:
```
chmod +x update-es-role-permissions.sh▼
```
The update-es-role-permissions.sh includes a help option. To open the help instructions, run update-es-role-permissions.sh -h.

Execute the script:

./update-es-role-permissions.sh▼

Example, if CloudBees Analytics were installed in /opt/cloudbees/dois:

ES_DIR_CONF=/opt/cloudbees/dois/conf/reporting/elasticsearch \
ES_DIR_BINS=/opt/cloudbees/dois/reporting/elasticsearch/bin \
ES_JAVA_HOME=/opt/cloudbees/dois/reporting/jre \
./update-es-role-permissions.sh▼

A success message is displayed if the role permissions are updated. If the script fails, follow the directions in the error messages, and then rerun the script.

After the script successfully runs, proceed to either:

Test upgrade process
Production upgrade process

Windows

To update the role permissions for Windows:

Download update-es-role-permission.ps1 from CloudBees examples.

The following commands use the script name update-es-role-permission.ps1. If you change the name, update the commands with your script name.

If your environment does not use the default configuration values for the CloudBees Analytics server, update the following default values as needed:

ES_PORT=9300
## Transport port used for communication between nodes. To use a different value, set the -Port option.

ES_DATA="C:/Users/<USERNAME>/Documents/SDA"
## Directory with DOIS binaries. To use a different value, set the -Data option.

ES_INSTALL="C:/Program Files/CloudBees/Software Delivery Automation"
## Directory with DOIS configuration files. To use a different value, set the -Install option.

CONFIG_DIR_TMP="$env:TEMP"
## Temporary folder created to patch configuration files. To use a different value, set -Temp.

ES_JAVA_HOME="C:\Program Files\CloudBees\Software Delivery Automation\reporting\jre"
## Java Home directory. To use a different value, set -JavaHome.

# Example usage:
update-role-permissions.ps1 [-Help] [-Port <number>] [-Data <path>] [-Install <path>] [-Temp <path>] [-JavaHome <path>]

# NOTE: All parameter arguments are passed as strings.

## Example to run script with new values:
update-role-permissions.ps1 -Port 9304 -Data "C:\Users\<username>\CloudBees\SDA"▼

CONFIG_DIR_TMP is a temporary directory, default $env:TEMP (such as C:\Users\<USERNAME>\AppData\Local\Temp), used to patch configuration files. If your system has any restrictions against such paths, ensure you assign a new path to the CONFIG_DIR_TMP variable. If you run the script without specifying CONFIG_DIR_TMP and the default temporary directory is not writable, the script fails.

Example:

update-role-permissions.ps1 -Temp "C:\Users\<USERNAME>\Documents"▼

Open Powershell as an Administrator user.

The update-role-permissions.ps1 script must be executed as an Administrator user. If you attempt to run the script as a non-Administrator user, the script fails with the following error message:

Please run the script with Administrator permissions. Quit.▼

Navigate to the directory with the update-role-permissions.ps1 script.

Execute the script:

update-role-permissions.ps1 [parameters <arguements>...]▼

For example, if CloudBees Analytics were installed with the following:
- Installation directory: C:\Users\User1\cloudbees\dois
- Data directory: C:\Users\User1\Documents\SDA
  update-role-permissions.ps1 ` -Install "C:\Users\User1\cloudbees\dois\conf\reporting\elasticsearch" ` -Data "C:\Users\User1\Documents\SDA"
  ▼

A success message is displayed if the role permissions are updated. If the script fails, follow the directions in the error messages, and then rerun the script.

After the script successfully runs, proceed to either:

Test upgrade process
Production upgrade process

Test upgrade process

CloudBees strongly recommends to test migrating your data from Elasticsearch to OpenSearch on a VM before attempting it in your production environment.

This approach helps to:

Identify and resolve potential issues with data migration.
Accurately estimate the duration of the migration process.
Develop and refine troubleshooting steps for production upgrades.
Minimize or avoid potential system downtime in your production environment.

To test the migration:

Review the CloudBees Analytics Elasticsearch to OpenSearch migration known issues.
Perform a clean installation of a CloudBees Software Delivery Automation server on a VM.

Perform an installation of v2024.06.0 or later CloudBees Analytics server within the same VM as your CloudBees Software Delivery Automation installation.

You can download the latest installer from the CloudBees downloads site.

The following information is critical for this process:

When launching the CloudBees Analytics installer, include the URL of the source remote server, as described in Specify remote data source servers.
When performing the CloudBees Software Delivery Automation server installation, do not choose the option to specify a remote CloudBees Software Delivery Automation server to interact with the CloudBees Analytics server when installing the destination CloudBees Analytics server.

Migrate the data from the source CloudBees Analytics server to the destination server by following the instructions on Migrate CloudBees Analytics data from Elasticsearch to OpenSearch for the preferred method.

Performing a backup of the source data is not required when performing a test run. The source system and its data are still available for normal use during and after the test migration.
To configure the CloudBees Analytics server on the CloudBees Software Delivery Automation server, run:
```
ectool setAnalyticsServerConfiguration --enabled 1▼
```
Log in to your new instance of CloudBees CD/RO.
Verify your CloudBees Analytics data is available and up to date in your upgraded CloudBees CD/RO instance.
1. Navigate to Analytics Dashboards, and verify your dashboards are able to populate accurate data.
2. Navigate to Analytics Reports, and verify your reports are able to populate accurate data.

If you have migrated your data without issues, this VM is no longer needed, and you can proceed to Production upgrade process. If you encounter multiple issues during testing that you cannot troubleshoot, copy the logs from the VM, and contact CloudBees Support.

Production upgrade process

For traditional upgrades from CloudBees Analytics v2024.03.0 or earlier to v2024.06.0 or later, the following steps are required for a successful production upgrade:

Review the CloudBees Analytics Elasticsearch to OpenSearch migration known issues.

Disable the connection between CloudBees Analytics and CloudBees CD/RO servers on your source machine by running:

ectool setDevOpsInsightServerConfiguration --enabled 0▼

For production migrations, once the CloudBees Analytics server is disabled, no data will be available for jobs running in CloudBees CD/RO. Additionally, this data will not be part of the backup created in the next step. If possible, it is suggested for all jobs to complete on the legacy server prior to creating the backup.

Back up the data from your legacy CloudBees Analytics Elasticsearch server to avoid data loss by following the instructions on Maintain CloudBees Analytics server data.

Failing to back up the source CloudBees Analytics data could result in permanent data loss if issues arise during data migration.

Install the updated destination CloudBees Analytics server using either:

The same installation and data directories.

A clean installation using new directories, other environments, or virtual machines.

When performing the installation, do not choose the option to specify a remote CloudBees Software Delivery Automation server to interact with the CloudBees Analytics server when installing the destination CloudBees Analytics server.

This is critical to prevent unexpected issues while migrating CloudBees Analytics data from Elasticsearch to OpenSearch. Once the data has been successfully migrated, use setAnalyticsServerConfiguration to configure the CloudBees Analytics server on the CloudBees Software Delivery Automation.

Refer to Upgrade on traditional platforms for more information.

Migrate the data from the source CloudBees Analytics server to the destination server by following the instructions on Migrate CloudBees Analytics data from Elasticsearch to OpenSearch.

Ensure you review the information in Traditional migrations overview before starting. This section contains important information needed during data migration.

Additionally, if your source and destination servers are different machines, ensure you follow the instructions in Specify remote data source servers.

To configure the CloudBees Analytics server on the CloudBees Software Delivery Automation server, run:
```
ectool setAnalyticsServerConfiguration --enabled 1▼
```
Verify your data is available and up to date on the destination CloudBees Analytics server.
Uninstall the source CloudBees Analytics server by following the instructions for your platform:
- Uninstall the CloudBees Analytics server on Linux
- Uninstall the CloudBees Analytics server on Windows