Upgrading a clustered environment

6 minute read

This section describes how to upgrade CloudBees CD when using a cluster configuration. The procedure is the same as when you upgrade to newer versions, except that you need to perform additional tasks to upgrade the cluster.

To upgrade to 9.x, use the CloudBeesFlow-<version> installer, which performs the following:

  • Collects the CloudBees CD service account credentials.

  • Uninstalls the current release.

  • Installs the latest CloudBees CD release.

  • Configures the system with all property values mined.

  • Restores custom files and data.

After preparing for the upgrade, make sure to shut down the CloudBees CD server service before installing CloudBees CD 9.x.

These instructions presume you are upgrading from CloudBees CD v5.x or later. To upgrade from v4.2.x or earlier, contact CloudBees Support directly for extra instructions.

Upgrading from version 9.1 or earlier

If you are upgrading CloudBees CD from version 9.1 or earlier, to version 9.2 or later, a new passkey with AES 256-bit encryption is automatically generated, replacing the older 128-bit passkey. During the database upgrade phase of the process, the CloudBees CD server re-encrypts the following using the newly generated passkey.

  • All the credentials in the system including the COMMANDER_DB_PASSWORD in database.properties

  • User credentials

  • Passwords in email configurations

  • DevOps Insight configuration passwords

For this to work, you must have the original 128-bit passkey in place to allow the conversion to be take place. Finally, the newly generated passkey is uploaded to ZooKeeper along with the updated database.properties file.

If you encounter any issues with your passkey after completing this upgrade, do not apply your passkey from your previous system as this will likely cause your database credential data to become corrupted. Please contact CloudBees Support if you encounter issues to help analyze whether the upgrade can be salvaged or if a repeat of the effort may be required.

Upgrading clusters

In a cluster configuration, make sure to review the tasks in Preparing for your cluster upgrade. To ensure that all important settings are saved, back up the following subdirectories in DATA_DIR :

  • The entire conf subdirectory, which contains the CloudBees CD server and agent configuration files

  • Apache web server configuration files in the apache/conf subdirectory

During the upgrade, CloudBees CD is not available after the web server is stopped or after the server service on the last CloudBees CD server node is stopped.

  1. Stop the CloudBees CD server service on all nodes except on the primary CloudBees CD server node.

    You can keep the server service on your primary CloudBees CD server node up, because the installer stops the server service automatically on that node when it upgrades the node.

    When upgrading the nodes in a CloudBees CD cluster, you must keep the other nodes stopped until the primary node upgrade is complete.
  2. On the primary CloudBees CD server, do the following:

    1. Upgrade the CloudBees CD server. This will also do the following:

      • Connects the server to the database

      • Upgrades the plugins

      • Starts the server

    2. Ensure that any custom wrapper.conf settings have been retained, or update accordingly.

    3. Restart the CloudBees CD servers service.

  3. On the CloudBees CD web server, do the following:

    1. Upgrade the node running any web server. This also upgrades the agents on these nodes.

    2. Ensure that any custom httpd.conf settings have been retained or updated accordingly.

      Before proceeding, test that you can sign in to the server from your webserver or from the command line with ectool login.
  4. Upgrade the remaining CloudBees CD server nodes one at a time.

    Your CloudBees CD system is now available.

Issues during the upgrade may cause some settings to be lost. Verify the following settings before connecting to the CloudBees CD system:

  • httpd.conf settings for redirecting: These lines should be commented out:

    # Redirect http to https # RewriteCond %\{HTTPS\} !=on # RewriteRule ^/commander/(.*) \https://%{SERVER_NAME}:443%{REQUEST_URI} [NC,R,L]
  • httpd.conf setting for COMMANDER_SERVER: This should point to the load balancer.

    SetEnv COMMANDER_SERVER "<FQDN of your load balancer>"
  • wrapper.conf contains the line that points to your Zookeeper instances.

    For example:

    `wrapper.java.additional.600=-DCOMMANDER_ZK_CONNECTION=192.168.7.20:2181`

Preparing for your upgrade

Upgrade testing

In most implementations, CloudBees CD is used in an environment that affects many users. You should test your upgrade on a separate test server to understand all aspects of the upgrade process. This test minimizes the potential impact on downstream users.

Backing up your existing CloudBees CD data

Before upgrading a CloudBees CD server, you must back up your existing CloudBees CD data. See CloudBees CD Server Backups for more information about backups.

Backing up CloudBees CD configuration files

The configuration files for the Commander cluster are in <data_dir>\conf. The default location is:

  • Linux: /opt/electriccloud/electriccommander/conf/

  • Windows: `C:\ProgramData\Electric Cloud\ElectricCommander\conf `

Although the Commander cluster configuration files such as commander.properties, database.properties, keystore, and passkey are present in one of the directories above, they are not actually used by the cluster during runtime.

These files were uploaded to Apache ZooKeeper from the first node that was clustered as described in Uploading Configuration Files to ZooKeeper . You can download these files from ZooKeeper to a temporary folder and then compare them with those in the \conf folder. You can do so by using the CloudBees CD ZKConfigTool, which is discussed in Uploading configuration files to ZooKeeper .

For example, complete the following steps to download these files to C:\temp on Windows, where <install_dir> is C:\Program Files\Electric Cloud\ElectricCommander.

  1. Download the files from ZooKeeper by entering the following commands:

    cd C:\temp "C:\Program Files\Electric Cloud\ElectricCommander\jre\bin\java" -DCOMMANDER_ZK_CONNECTION=<ZooKeeper_Server_IP>:2181 -jar "C:\Program Files\Electric Cloud\ElectricCommander\server\bin\zk-config-tool-jar-with-dependencies.jar" com.electriccloud.commander.cluster.ZKConfigTool --readFile /commander/conf/database.properties database.properties "C:\Program Files\Electric Cloud\ElectricCommander\jre\bin\java" -DCOMMANDER_ZK_CONNECTION=<ZooKeeper_Server_IP>:2181 -jar "C:\Program Files\Electric Cloud\ElectricCommander\server\bin\zk-config-tool-jar-with-dependencies.jar" com.electriccloud.commander.cluster.ZKConfigTool --readFile /commander/conf/keystore keystore "C:\Program Files\Electric Cloud\ElectricCommander\jre\bin\java" -DCOMMANDER_ZK_CONNECTION=<ZooKeeper_Server_IP>:2181 -jar "C:\Program Files\Electric Cloud\ElectricCommander\server\bin\zk-config-tool-jar-with-dependencies.jar" com.electriccloud.commander.cluster.ZKConfigTool --readFile /commander/conf/passkey passkey "C:\Program Files\Electric Cloud\ElectricCommander\jre\bin\java" -DCOMMANDER_ZK_CONNECTION=<ZooKeeper_Server_IP>:2181 -jar "C:\Program Files\Electric Cloud\ElectricCommander\server\bin\zk-config-tool-jar-with-dependencies.jar" com.electriccloud.commander.cluster.ZKConfigTool --readFile /commander/conf/commander.properties
  2. Ensure the four files in C:\temp are the same as the ones in <data_dir>\conf (a file diff tool can make this easier). If any file in <data_dir>\conf is different, then back up that file and replace it with the one that you downloaded from ZooKeeper.

Backing up other files

The CloudBees CD files that might have been modified are too numerous to list, so you should back up the entire CloudBees CD data directory and other miscellaneous files that might have changed. But at a minimum, you must back up the following files:

  • The plugins directory. The default location is the plugins subdirectory within <data_dir>.

  • Files that contain your configuration and custom settings. To ensure that all important settings are saved, back up the following subdirectories in <data_dir> :

  • The entire conf subdirectory, which contains the CloudBees CD server and agent configuration files.

  • Apache web server configuration files in the apache/conf subdirectory.

  • (If applicable) The local MySQL database configuration file, my.ini, in the mysql subdirectory.

  • Any other files where you created custom configurations, specified other custom information, or created any type of modification.

  • (If you use an artifact repository) The CloudBees CD repository configuration files in the conf/repository subdirectory.

  • (If modified) The custom editor or preflight driver script properties (installed by default).

These properties are stored in the server-level property sheet, which you can view in the web UI in the Administration > Server subtab.

Custom editors are stored in the nested sheet named ec_customEditors. Preflight driver scripts are stored in the nested sheet named ec_preflight. The upgrade process overwrites default custom editor and preflight driver scripts with current versions. You should back up any custom properties that you created by renaming those properties. For example, change ec_preflight/clientDrivers/perforce to ec_preflight/clientDrivers/perforce_modified.

Upgrade installer preservation

After you back up your CloudBees CD server, create a folder where you can download the CloudBees CD-<version> installation file.

Starting the installation process

Install the upgrade by choosing the correct upgrade method for your environment.

Copying repository contents

After you have updated CloudBees CD, perform the following steps to copy the contents of an existing repository server into a newly installed repository server:

  1. Install the new repository server software.

  2. Stop the existing and new repository servers.

  3. Copy the entire contents of the repository backingstore directory from the existing repository server to the corresponding location on the newly installed repository server.

    The default location for the backingstore directory (<datadir>/repository-data) is: UNIX/opt/electriccloud/electriccommander/repository-data WindowsC:\ProgramData\Electric Cloud\ElectricCommander\repository-data

MySQL considerations with server upgrades

CloudBees CD upgrades involving a MySQL database can take several hours to complete if you have a significant data set.

To avoid corrupting your database, do not interrupt the upgrade process. A restore from a previous database backup would be required if this occurs.

You can use ectool to view the upgrade progress. On a command line, enter

ectool getServerStatus

An install or upgrade log file named installer.log is created in the logs subdirectory in <data_dir>.

Uploading configuration files to ZooKeeper if needed

After you upgrade the CloudBees CD server node, you must again compare the <data_dir> \conf\commander.properties file with the file that you downloaded from ZooKeeper (which you saved to c:\temp). To do so, complete the following steps.

  1. Open the <data_dir> \conf\commander.properties file.

  2. Make sure that the COMMANDER_SERVER_NAME property is set to <load_balancer_FQDN> .

  3. If the following line exists, remove it:

    COMMANDER_MQ_DISK_SPACE_CHECK_FREQUENCY=60
  4. Check whether the following lines exist. If they do not exist, add them:

    COMMANDER_CRITICAL_SERVICES_MONITORING_FREQUENCY=60 COMMANDER_CRITICAL_SERVICES_MONITORING_ENABLED=true COMMANDER_CRITICAL_SERVICES_MAX_ATTEMPTS_TO_BE_IN_PRIMARY_CLUSTER=5

    Ensure that these properties are not duplicated in the file.

  5. Upload the new file to ZooKeeper as described in the Uploading Configuration Files to ZooKeeper section in the "Clustering" chapter.