Upgrading a clustered environment

7 minute readDeveloper productivity

This section describes how to upgrade CloudBees CD/RO when using a cluster configuration. The procedure is the same as when you upgrade to newer versions, except that you need to perform additional tasks to upgrade the cluster.

To upgrade to {upgrade-release-version}, use the CloudBeesFlow-<version> installer, which performs the following:

  • Collects the CloudBees CD/RO service account credentials.

  • Uninstalls the current release.

  • Installs the latest CloudBees CD/RO release.

  • Configures the system with all property values mined.

  • Restores custom files and data.

After preparing for the upgrade, make sure to shut down the CloudBees CD/RO server service before installing CloudBees CD/RO {upgrade-release-version}.

These instructions presume you are upgrading from CloudBees CD/RO v5.x or later. To upgrade from v4.2.x or earlier, contact CloudBees Support directly for extra instructions.

Upgrading from version 9.1 or earlier

If you are upgrading CloudBees CD/RO from version 9.1 or earlier, to version 9.2 or later, a new passkey with AES 256-bit encryption is automatically generated, replacing the older 128-bit passkey. During the database upgrade phase of the process, the CloudBees CD/RO server re-encrypts the following using the newly generated passkey.

  • All the credentials in the system including the COMMANDER_DB_PASSWORD in database.properties

  • User credentials

  • Passwords in email configurations

  • CloudBees Analytics configuration passwords

For this to work, you must have the original 128-bit passkey in place to allow the conversion to be take place. Finally, the newly generated passkey is uploaded to ZooKeeper along with the updated database.properties file.

If you encounter any issues with your passkey after completing this upgrade, do not apply your passkey from your previous system as this will likely cause your database credential data to become corrupted. Please contact CloudBees Support if you encounter issues to help analyze whether the upgrade can be salvaged or if a repeat of the effort may be required.

Upgrading clusters

In a cluster configuration, make sure to review the tasks in Preparing for your cluster upgrade. To ensure that all important settings are saved, back up the following subdirectories in DATA_DIR :

  • The entire conf subdirectory, which contains the CloudBees CD/RO server and agent configuration files

  • Apache web server configuration files in the apache/conf subdirectory

During the upgrade, CloudBees CD/RO is not available after the web server is stopped or after the server service on the last CloudBees CD/RO server node is stopped.

  1. Stop the CloudBees CD/RO server service on all nodes except on the primary CloudBees CD/RO server node.

    You can keep the server service on your primary CloudBees CD/RO server node up, because the installer stops the server service automatically on that node when it upgrades the node.

    When upgrading the nodes in a CloudBees CD/RO cluster, you must keep the other nodes stopped until the primary node upgrade is complete.
    CloudBees CD/RO server version 10.5.0 and later is only compatible with ZooKeeper server version 3.5.5 and later. If the installer detects that the CloudBees CD/RO server being upgraded is not compatible with the current version of ZooKeeper, you receive an error. Refer to the steps in Updating the ZooKeeper server before you continue your upgrade process.
  2. On the primary CloudBees CD/RO server, do the following:

    1. Upgrade the CloudBees CD/RO server. This will also do the following:

      • Connects the server to the database

      • Upgrades the plugins

      • Starts the server

    2. Ensure that any custom wrapper.conf settings have been retained, or update accordingly.

    3. Restart the CloudBees CD/RO servers service.

  3. On the CloudBees CD/RO web server, do the following:

    1. Upgrade the node running any web server. This also upgrades the agents on these nodes.

    2. Ensure that any custom httpd.conf settings have been retained or updated accordingly.

      Before proceeding, test that you can sign in to the server from your webserver or from the command line with ectool login.
  4. Upgrade the remaining CloudBees CD/RO server nodes one at a time.

    Your CloudBees CD/RO system is now available.

Issues during the upgrade may cause some settings to be lost. Verify the following settings before connecting to the CloudBees CD/RO system:

  • httpd.conf settings for redirecting: These lines should be commented out:

    # Redirect http to https # RewriteCond %\{HTTPS\} !=on # RewriteRule ^/commander/(.*) \https://%{SERVER_NAME}:443%{REQUEST_URI} [NC,R,L]
  • httpd.conf setting for COMMANDER_SERVER: This should point to the load balancer.

    SetEnv COMMANDER_SERVER "<FQDN of your load balancer>"
  • wrapper.conf contains the line that points to your Zookeeper instances.

    For example:

    wrapper.java.additional.600=-DCOMMANDER_ZK_CONNECTION=192.168.7.20:2181

Preparing for your upgrade

Upgrade testing

In most implementations, CloudBees CD/RO is used in an environment that affects many users. You should test your upgrade on a separate test server to understand all aspects of the upgrade process. This test minimizes the potential impact on downstream users.

Backing up your existing CloudBees CD/RO data

Before upgrading a CloudBees CD/RO server, you must back up your existing CloudBees CD/RO data. See CloudBees CD/RO Server Backups for more information about backups.

Backing up CloudBees CD/RO configuration files

The configuration files for the Commander cluster are in <data_dir>\conf. The default location is:

  • Linux: /opt/cloudbees/sda/conf/

  • Windows: `C:\ProgramData\CloudBees\Software Delivery Automation\conf `

Although the Commander cluster configuration files such as commander.properties, database.properties, keystore, and passkey are present in one of the directories above, they are not actually used by the cluster during runtime.

These files were uploaded to Apache ZooKeeper from the first node that was clustered as described in Uploading Configuration Files to ZooKeeper . You can download these files from ZooKeeper to a temporary folder and then compare them with those in the \conf folder. You can do so by using the CloudBees CD/RO ZKConfigTool, which is discussed in Uploading configuration files to ZooKeeper .

For example, complete the following steps to download these files to C:\temp on Windows, where <install_dir> is C:\Program Files\CloudBees\Software Delivery Automation.

  1. Download the files from ZooKeeper by entering the following commands:

    cd C:\temp "C:\Program Files\CloudBees\Software Delivery Automation\jre\bin\java" -DCOMMANDER_ZK_CONNECTION=<ZooKeeper_Server_IP>:2181 -jar "C:\Program Files\CloudBees\Software Delivery Automation\server\bin\zk-config-tool-jar-with-dependencies.jar" com.electriccloud.commander.cluster.ZKConfigTool --readFile /commander/conf/database.properties database.properties "C:\Program Files\CloudBees\Software Delivery Automation\jre\bin\java" -DCOMMANDER_ZK_CONNECTION=<ZooKeeper_Server_IP>:2181 -jar "C:\Program Files\CloudBees\Software Delivery Automation\server\bin\zk-config-tool-jar-with-dependencies.jar" com.electriccloud.commander.cluster.ZKConfigTool --readFile /commander/conf/keystore keystore "C:\Program Files\CloudBees\Software Delivery Automation\jre\bin\java" -DCOMMANDER_ZK_CONNECTION=<ZooKeeper_Server_IP>:2181 -jar "C:\Program Files\CloudBees\Software Delivery Automation\server\bin\zk-config-tool-jar-with-dependencies.jar" com.electriccloud.commander.cluster.ZKConfigTool --readFile /commander/conf/passkey passkey "C:\Program Files\CloudBees\Software Delivery Automation\jre\bin\java" -DCOMMANDER_ZK_CONNECTION=<ZooKeeper_Server_IP>:2181 -jar "C:\Program Files\CloudBees\Software Delivery Automation\server\bin\zk-config-tool-jar-with-dependencies.jar" com.electriccloud.commander.cluster.ZKConfigTool --readFile /commander/conf/commander.properties
  2. Ensure the four files in C:\temp are the same as the ones in <data_dir>\conf (a file diff tool can make this easier). If any file in <data_dir>\conf is different, then back up that file and replace it with the one that you downloaded from ZooKeeper.

Backing up other files

The CloudBees CD/RO files that might have been modified are too numerous to list, so you should back up the entire CloudBees CD/RO data directory and other miscellaneous files that might have changed. But at a minimum, you must back up the following files:

  • The plugins directory. The default location is the plugins subdirectory within <data_dir>.

  • Files that contain your configuration and custom settings. To ensure that all important settings are saved, back up the following subdirectories in <data_dir> :

  • The entire conf subdirectory, which contains the CloudBees CD/RO server and agent configuration files.

  • Apache web server configuration files in the apache/conf subdirectory.

  • (If applicable) The local MySQL database configuration file, my.ini, in the mysql subdirectory.

  • Any other files where you created custom configurations, specified other custom information, or created any type of modification.

  • (If you use an artifact repository) The CloudBees CD/RO repository configuration files in the conf/repository subdirectory.

  • (If modified) The custom editor or preflight driver script properties (installed by default).

These properties are stored in the server-level property sheet, which you can view in the web UI in the Administration > Server subtab.

Custom editors are stored in the nested sheet named ec_customEditors. Preflight driver scripts are stored in the nested sheet named ec_preflight. The upgrade process overwrites default custom editor and preflight driver scripts with current versions. You should back up any custom properties that you created by renaming those properties. For example, change ec_preflight/clientDrivers/perforce to ec_preflight/clientDrivers/perforce_modified.

Upgrade installer preservation

After you back up your CloudBees CD/RO server, create a folder where you can download the CloudBees CD/RO-<version> installation file.

Starting the installation process

Install the upgrade by choosing the correct upgrade method for your environment.

Copying repository contents

After you have updated CloudBees CD/RO, perform the following steps to copy the contents of an existing repository server into a newly installed repository server:

  1. Install the new repository server software.

  2. Stop the existing and new repository servers.

  3. Copy the entire contents of the repository backingstore directory from the existing repository server to the corresponding location on the newly installed repository server.

    The default location for the backingstore directory (<datadir>/repository-data) is: UNIX/opt/cloudbees/sda/repository-data WindowsC:\ProgramData\CloudBees\Software Delivery Automation\repository-data

MySQL considerations with server upgrades

CloudBees CD/RO upgrades involving a MySQL database can take several hours to complete if you have a significant data set.

To avoid corrupting your database, do not interrupt the upgrade process. A restore from a previous database backup would be required if this occurs.

You can use ectool to view the upgrade progress. On a command line, enter

ectool getServerStatus

An install or upgrade log file named installer.log is created in the logs subdirectory in <data_dir>.

Uploading configuration files to ZooKeeper if needed

After you upgrade the CloudBees CD/RO server node, you must again compare the <data_dir> \conf\commander.properties file with the file that you downloaded from ZooKeeper (which you saved to c:\temp). To do so, complete the following steps.

  1. Open the <data_dir> \conf\commander.properties file.

  2. Make sure that the COMMANDER_SERVER_NAME property is set to <load_balancer_FQDN> .

  3. If the following line exists, remove it:

    COMMANDER_MQ_DISK_SPACE_CHECK_FREQUENCY=60
  4. Check whether the following lines exist. If they do not exist, add them:

    COMMANDER_CRITICAL_SERVICES_MONITORING_FREQUENCY=60 COMMANDER_CRITICAL_SERVICES_MONITORING_ENABLED=true COMMANDER_CRITICAL_SERVICES_MAX_ATTEMPTS_TO_BE_IN_PRIMARY_CLUSTER=5

    Ensure that these properties are not duplicated in the file.

  5. Upload the new file to ZooKeeper as described in the Uploading Configuration Files to ZooKeeper section in the "Clustering" chapter.

Updating the ZooKeeper server

CloudBees CD/RO server version 10.5.0 and later in cluster mode requires ZooKeeper server version 3.5.5 or later. If you have an outdated ZooKeeper server, you must update it to a newer version.

To update the ZooKeeper server:

  1. Run the upgrade for the primary CloudBees CD/RO server node. Use the --skipCheckZookeeperVersion command line parameter in console/silent mode to disable the ZooKeeper server unsupported version message, or confirm the update in the UI.

  2. Complete the upgrade for the primary node.

    The CloudBees CD/RO server will not be able to complete its upgrade at this point.
  3. Stop the CloudBees CD/RO server service.

  4. Run the CloudBees CD/RO installer using the --zookeeperDataDownload command line parameter. This command downloads all the necessary configuration files from the current ZooKeeper version and saves them to the CloudBees CD/RO server data directory.

  5. Uninstall the ZooKeeper server and all its associated files. If the ZooKeeper server is installed in cluster mode and uses multiple nodes, you must uninstall all the nodes.

  6. Install the updated ZooKeeper server. For more information, refer to Installing ZooKeeper. If you need to run the ZooKeeper server in cluster mode, install and configure each node.

  7. Run the CloudBees CD/RO installer using the --zookeeperDataUpload command line parameter. This command loads all necessary configuration files from the CloudBees CD/RO server data directory into the updated ZooKeeper server instance.

  8. Start the CloudBees CD/RO server service.

    The CloudBees CD/RO server update begins, and you can proceed to step two of Upgrading a clustered environment.