Upgrade a clustered environment

This section describes how to upgrade CloudBees CD/RO when using a cluster configuration. The procedure is the same as when you upgrade to newer versions, except that you need to perform additional tasks to upgrade the cluster.

After upgrading from CloudBees CD/RO 10.8 or an earlier version, delete all .old files, such as .properties.old in the conf subdirectory to finish the encryption process.

To upgrade, use the CloudBeesFlow-<version> installer, which performs the following:

Collects the CloudBees CD/RO service account credentials.
Uninstalls the current release.
Installs the latest CloudBees CD/RO release.
Configures the system with all property values mined.
Restores custom files and data.

After preparing for the upgrade, make sure to shut down the CloudBees CD/RO server service before installing CloudBees CD/RO.

These instructions presume you are upgrading from CloudBees CD/RO v5.x or later. To upgrade from v4.2.x or earlier, contact CloudBees Support directly for additional instructions.

Upgrading clusters

In a cluster configuration, make sure to review the tasks in Upgrade testing. To ensure that all important settings are saved, back up the following subdirectories in DATA_DIR:

The entire conf subdirectory, which contains the CloudBees CD/RO server and agent configuration files
Apache web server configuration files in the apache/conf subdirectory

During the upgrade, CloudBees CD/RO is not available after the web server is stopped or after the server service on the last CloudBees CD/RO server node is stopped.

When upgrading the nodes in a CloudBees CD/RO cluster, you must keep the other nodes stopped until the primary node upgrade is complete.

Also, after upgrading from CloudBees CD/RO 10.8 or earlier, delete all .old files, such as .properties.old in the conf subdirectory to finish the encryption process.

To upgrade clusters:

Stop the CloudBees CD/RO server service on all nodes except on the primary CloudBees CD/RO server node. You can keep the server service on your primary CloudBees CD/RO server node up, because the installer stops the server service automatically on that node when it upgrades the node.

CloudBees CD/RO server version 10.5.0 and later is only compatible with ZooKeeper server v3.8.0 and later. The installer will return an error if the CloudBees CD/RO server being upgraded is not compatible with your ZooKeeper version. Refer to the steps in Updating the ZooKeeper server before you continue your upgrade process.

On the primary CloudBees CD/RO server, do the following:
1. Upgrade the CloudBees CD/RO server. This will also do the following:
  - Connect the server to the database
  - Upgrade the plugins
  - Start the server
2. Ensure that any custom wrapper.conf settings have been retained, or update accordingly.
3. Restart the CloudBees CD/RO servers service.
On the CloudBees CD/RO web server, do the following:
1. Upgrade the node running any web server. This also upgrades the agents on these nodes.
2. Ensure that any custom httpd.conf settings have been retained or updated accordingly.
  
  Before proceeding, test that you can sign in to the server from your webserver or from the command line with ectool login.
Upgrade the remaining CloudBees CD/RO server nodes one at a time.

Your CloudBees CD/RO system is now available.

Verifying the upgrade

Issues during the upgrade may cause some settings to be lost. Verify the following settings before connecting to the CloudBees CD/RO system:

httpd.conf settings for redirecting. The following lines should be commented out:

# Redirect http to https
# RewriteCond %\{HTTPS\} !=on
# RewriteRule ^/commander/(.*) \https://%{SERVER_NAME}:443%{REQUEST_URI} [NC,R,L]▼

httpd.conf setting for COMMANDER_SERVER should point to the load balancer:
```
SetEnv COMMANDER_SERVER "<FQDN of your load balancer>"▼
```
wrapper.conf contains the line that points to your Zookeeper instances.

For example:
```
wrapper.java.additional.600=-DCOMMANDER_ZK_CONNECTION=192.168.7.20:2181▼
```

Upgrade testing

In most implementations, CloudBees CD/RO is used in an environment that affects many users. You should test your upgrade on a separate test server to understand all aspects of the upgrade process. This test minimizes the potential impact on downstream users.

Backing up your existing CloudBees CD/RO data

Before upgrading a CloudBees CD/RO server, you must back up your existing CloudBees CD/RO data. Refer to CloudBees CD/RO Server Backups for more information about backups.

These files were uploaded to Apache ZooKeeper from the first node that was clustered as described in Uploading Configuration Files to ZooKeeper. You can download these files from ZooKeeper to a temporary folder and then compare them with those in the \conf folder. You can do so by using the CloudBees CD/RO ZKConfigTool.

For example, complete the following steps to download these files to C:\temp on Windows, where <install_dir> is C:\Program Files\CloudBees\Software Delivery Automation.

Download the files from ZooKeeper by entering the following commands:

cd C:\temp

"C:\Program Files\CloudBees\Software Delivery Automation\jre\bin\java" -DCOMMANDER_ZK_CONNECTION=<ZooKeeper_Server_IP>:2181 -jar "C:\Program Files\CloudBees\Software Delivery Automation\server\bin\zk-config-tool-jar-with-dependencies.jar" com.electriccloud.commander.cluster.ZKConfigTool --readFile /commander/conf/database.properties database.properties

"C:\Program Files\CloudBees\Software Delivery Automation\jre\bin\java" -DCOMMANDER_ZK_CONNECTION=<ZooKeeper_Server_IP>:2181 -jar "C:\Program Files\CloudBees\Software Delivery Automation\server\bin\zk-config-tool-jar-with-dependencies.jar" com.electriccloud.commander.cluster.ZKConfigTool --readFile /commander/conf/keystore keystore

"C:\Program Files\CloudBees\Software Delivery Automation\jre\bin\java" -DCOMMANDER_ZK_CONNECTION=<ZooKeeper_Server_IP>:2181 -jar "C:\Program Files\CloudBees\Software Delivery Automation\server\bin\zk-config-tool-jar-with-dependencies.jar" com.electriccloud.commander.cluster.ZKConfigTool --readFile /commander/conf/passkey passkey

"C:\Program Files\CloudBees\Software Delivery Automation\jre\bin\java" -DCOMMANDER_ZK_CONNECTION=<ZooKeeper_Server_IP>:2181 -jar "C:\Program Files\CloudBees\Software Delivery Automation\server\bin\zk-config-tool-jar-with-dependencies.jar" com.electriccloud.commander.cluster.ZKConfigTool --readFile /commander/conf/commander.properties▼

Ensure the four files in C:\temp are the same as the ones in <data_dir>\conf (a file diff tool can make this easier). If any file in <data_dir>\conf is different, then back up that file and replace it with the one that you downloaded from ZooKeeper.

Backing up custom files

The CloudBees CD/RO files that might have been modified are too numerous to list, so you should back up the entire CloudBees CD/RO data directory and other miscellaneous files that might have changed. However, you must back up the following files:

The plugins directory. The default location is the plugins subdirectory within <data_dir>.
Files that contain custom configuration data. To ensure that important settings are saved, back up the following subdirectories in <data_dir>:

These properties are stored in the server-level property sheet, which can be viewed in the web UI by accessing the Administration tab/Server subtab.
- The entire conf subdirectory, which contains the CloudBees CD/RO server and agent configuration files.
- Apache web server configuration files in the apache/conf subdirectory.
- (If applicable) The local MySQL database configuration file, my.ini, in the mysql subdirectory.
- (If you use an artifact repository) The CloudBees CD/RO repository configuration files in the conf/repository subdirectory.

Any other files where you created custom configurations, specified other custom information, or created any type of modification.

The upgrade process overwrites default custom editor and preflight driver scripts nested property sheets. Therefore, CloudBees recommends backing up the ec_customEditors and ec_preflight property sheets.

Backup the ec_customEditors and ec_preflight property sheets by renaming them. For example, change ec_preflight/clientDrivers/perforce to ec_preflight/clientDrivers/perforce_modified.

Preserving the upgrade installation file

After you back up your CloudBees CD/RO server, create a folder where you can download the CloudBees CD/RO-<version> installation file.

Starting the installation process

Install the upgrade by choosing the correct upgrade method for your environment.

Copying repository contents

After you have updated CloudBees CD/RO, perform the following steps to copy the contents of an existing repository server into a newly installed repository server:

Install the new repository server software.
Stop the existing and new repository servers.
Copy the entire contents of the repository backingstore directory from the existing repository server to the corresponding location on the newly installed repository server.

The default location for the backingstore directory (<datadir>/repository-data) is:
- UNIX — /opt/cloudbees/sda/repository-data
- Windows — C:\ProgramData\CloudBees\Software Delivery Automation\repository-data

Configuring CloudBees CD/RO server, agent, and repository server logging

If you defined custom log settings for CloudBees CD/RO, you must migrate this data from the current log configuration file to the new one.

Use these steps to configure each new file referenced in the table below.

Copy data from current file into new file.

Modify this configuration entry from logger("org.hibernate.SQL", TRACE) to org.hibernate.SQL=TRACE.

Log Type	Current file	New file
CloudBees CD/RO server	/opt/electriccloud/electriccommander/conf/logback-local.groovy /opt/cloudbees/sda/conf/logback-local.groovy	/opt/cloudbees/sda/conf/logging-local.properties
Agents	/opt/electriccloud/electriccommander/conf/agent/logback-local.groovy /opt/cloudbees/sda/conf/agent/logback-local.groovy	/opt/cloudbees/sda/conf/agent/logging-local.properties
Repository Server	/opt/electriccloud/electriccommander/conf/logback-local.groovy /opt/cloudbees/sda/conf/logback-local.groovy	/opt/cloudbees/sda/conf/logging-local.properties

Log Type

Current file

New file

CloudBees CD/RO server

/opt/electriccloud/electriccommander/conf/logback-local.groovy

/opt/cloudbees/sda/conf/logback-local.groovy

/opt/cloudbees/sda/conf/logging-local.properties

Agents

/opt/electriccloud/electriccommander/conf/agent/logback-local.groovy

/opt/cloudbees/sda/conf/agent/logback-local.groovy

/opt/cloudbees/sda/conf/agent/logging-local.properties

Repository Server

/opt/electriccloud/electriccommander/conf/logback-local.groovy

/opt/cloudbees/sda/conf/logback-local.groovy

/opt/cloudbees/sda/conf/logging-local.properties

Bundled plugins updates

Starting with CloudBees CD/RO v2023.08.0, when upgrading your CloudBees CD/RO environment, bundled plugins are only updated or reinstalled if the bundled plugin version changes. This helps decrease upgrade times by skipping reinstalling bundled plugins that had no changes for the CloudBees CD/RO release.

However, after a successful upgrade to v2023.08.0 or later, if you need to reinstall all bundled plugins regardless of whether they have changes, you can invoke a force-update to all bundled plugins.

For CloudBees CD/RO v2023.06.0 or earlier, the following steps are not applicable. For each CloudBees CD/RO v2023.06.0 or earlier upgrade, all bundled plugins are reinstalled automatically.

Prerequisites to force-update bundled plugins

The following prerequisites are required to force all bundled plugins to reinstall:

You must successfully upgrade your current CloudBees CD/RO version to v2023.08.0 or later.
You must have CloudBees CD/RO administrator permissions and password for the flow-server instances where the bundled plugin updates should occur.

Force-update bundled plugins

After you have successfully upgraded your current CloudBees CD/RO version to v2023.08.0 or later, perform the following steps to force all bundled plugins to reinstall:

Log into the flow-server instance as a CloudBees CD/RO administrator using ectool.
```
ectool login admin▼
```
Provide your CloudBees CD/RO administrator password.
Execute the bundled plugin force-update using CloudBees ectool.
```
ectool reloadSetupScripts --force 1▼
```

If no error message is returned, the command installs all bundled plugins, and will silently exit when complete.

MySQL considerations with server upgrades

CloudBees CD/RO upgrades involving a MySQL database can take several hours to complete if you have a significant data set.

To avoid corrupting your database, do not interrupt the upgrade process. A restore from a previous database backup is required if this occurs.

You can use ectool to view the upgrade progress. On a command line, enter:

ectool getServerStatus▼

This command creates an install or upgrade log file named installer.log in the logs subdirectory in <data_dir>.

Uploading configuration files to ZooKeeper if needed

After you upgrade the CloudBees CD/RO server node, you must again compare the <data_dir> \conf\commander.properties file with the file that you downloaded from ZooKeeper (which you saved to C:\temp). To do so, complete the following steps:

Open the <data_dir> \conf\commander.properties file.
Make sure that the COMMANDER_SERVER_NAME property is set to <load_balancer_FQDN>.

If the following line exists, remove it:

COMMANDER_MQ_DISK_SPACE_CHECK_FREQUENCY=60▼

Check whether the following lines exist. If they do not exist, add them:

COMMANDER_CRITICAL_SERVICES_MONITORING_FREQUENCY=60
COMMANDER_CRITICAL_SERVICES_MONITORING_ENABLED=true
COMMANDER_CRITICAL_SERVICES_MAX_ATTEMPTS_TO_BE_IN_PRIMARY_CLUSTER=5▼

Ensure that these properties are not duplicated in the file.

Upload the new file to ZooKeeper as described in Uploading Configuration Files to ZooKeeper.

Updating the CloudBees Analytics server

If CloudBees Analytics is included in your clustered environment, it must be updated separately from CloudBees CD/RO. For information on updating CloudBees Analytics, refer to Upgrade the CloudBees Analytics server.

Updating the ZooKeeper server

CloudBees CD/RO server v10.5.0 and later in cluster mode requires ZooKeeper server v3.8.0 or later. If you have an outdated ZooKeeper server, you must update it to a newer version.

To update the ZooKeeper server:

Run the upgrade for the primary CloudBees CD/RO server node. Use the --skipCheckZookeeperVersion command line parameter in console/silent mode to disable the ZooKeeper server unsupported version message, or confirm the update in the UI.
Complete the upgrade for the primary node.

The CloudBees CD/RO server will not be able to complete its upgrade at this point.
Stop the CloudBees CD/RO server service.
Run the CloudBees CD/RO installer using the --zookeeperDataDownload command line parameter. This command downloads all the necessary configuration files from the current ZooKeeper version and saves them to the CloudBees CD/RO server data directory.
Uninstall the ZooKeeper server and all its associated files. If the ZooKeeper server is installed in cluster mode and uses multiple nodes, you must uninstall all the nodes.
Install the updated ZooKeeper server. For more information, refer to Install ZooKeeper. If you need to run the ZooKeeper server in cluster mode, install and configure each node.
Run the CloudBees CD/RO installer using the --zookeeperDataUpload command line parameter. This command loads all necessary configuration files from the CloudBees CD/RO server data directory into the updated ZooKeeper server instance.
Start the CloudBees CD/RO server service.

The CloudBees CD/RO server update begins, and you can proceed to step two of Upgrade a clustered environment.