Upgrading a clustered environment

5 minute read

This section describes how to upgrade CloudBees Flow and cluster configurations at the same time. The procedure is the same as when you upgrade to newer versions, except that you need to perform additional tasks to upgrade the cluster.

To upgrade to 9.x, use the CloudBeesFlow-<version> installer, which performs the following:

  • Collects the CloudBees Flow service account credentials

  • Uninstalls the current release

  • Installs the latest CloudBees Flow release

  • Configures the system with all property values mined

  • Restores custom files and data

After preparing for the upgrade, make sure to shut down the CloudBees Flow server service before installing CloudBees Flow 9.x.

These instructions presume upgrading from CloudBees Flow v5.x or newer. If you are looking to upgrade from v4.2.x or older, please contact CloudBees Support directly for extra instructions.

Upgrading clusters

In a cluster configuration, make sure to review the tasks in Preparing for your cluster upgrade. To ensure that all important settings are saved, back up the following subdirectories in DATA_DIR :

  • The entire conf subdirectory, which contains the CloudBees Flow server and agent configuration files

  • Apache web server configuration files in the apache/conf subdirectory

During the upgrade, CloudBees Flow is not available after the web server is stopped or after the server service on the last CloudBees Flow server node is stopped.

  1. Perform either of these actions:

    • Stop the CloudBees Flow server service on all nodes.

    • Stop the CloudBees Flow server service on all nodes except on the primary CloudBees Flow server node.

      You can keep the server service on your primary CloudBees Flow server node up, because the installer stops the server service automatically on that node when it upgrades the node.

      When upgrading the nodes in a CloudBees Flow cluster, you must keep the other nodes stopped until the primary node upgrade is complete.
  2. On the primary CloudBees Flow server, do the following:

    1. Upgrade the CloudBees Flow server. This will also do the following:

      • Connects the server to the database

      • Upgrades the plugins

      • Starts the server

    2. Ensure that any custom wrapper.conf settings have been retained, or update accordingly. For example, the settings for the line wrapper.java.additional.600=.

    3. Restart the CloudBees Flow servers service.

  3. On the CloudBees Flow web server, do the following:

    1. Upgrade the node running any web server. This also upgrades the agents on these nodes.

    2. Ensure that any custom httpd.conf settings have been retained, or update accordingly.

  4. Upgrade the remaining CloudBees Flow server nodes one-at-a-time.

    Your CloudBees Flow system is now available.

  5. Upgrade the remaining CloudBees Flow server nodes.

Issues during the upgrade may cause some settings to be lost. Verify the following settings before connecting to the CloudBees Flow system:

  • httpd.conf settings for redirecting—These lines should be commented out:

    # Redirect http to https # RewriteCond %\{HTTPS\} !=on # RewriteRule ^/commander/(.*) \https://%{SERVER_NAME}:443%{REQUEST_URI} [NC,R,L]
  • httpd.conf setting for COMMANDER_SERVER –This should point to the load balancer:

    SetEnv COMMANDER_SERVER "<FQDN of your load balancer>"
  • wrapper.conf contains the line that points to your Zookeeper instances.

    For example:

    `wrapper.java.additional.600=-DCOMMANDER_ZK_CONNECTION=192.168.7.20:2181`

Preparing for your upgrade

Upgrade testing

In most implementations, CloudBees Flow is used in an environment that affects many users. You should test your upgrade on a separate test server to understand all aspects of the upgrade process. This test minimizes the potential impact on downstream users.

Backing up your existing CloudBees Flow data

Before upgrading a CloudBees Flow server, you must back up your existing CloudBees Flow data. See CloudBees Flow Server Backups for more information about backups.

Backing up CloudBees Flow configuration files

The configuration files for the Commander cluster are in <data_dir>\conf. The default location is:

  • Linux: /opt/electriccloud/electriccommander/conf/

  • Windows: `C:\ProgramData\Electric Cloud\ElectricCommander\conf `

Although the Commander cluster configuration files such as commander.properties, database.properties, keystore, and passkey are present in one of the directories above, they are not actually used by the cluster during runtime.

These files were uploaded to Apache ZooKeeper from the first node that was clustered as described in Uploading Configuration Files to ZooKeeper . You can download these files from ZooKeeper to a temporary folder and then compare them with those in the \conf folder. You can do so by using the CloudBees Flow ZKConfigTool, which is discussed in Uploading configuration files to ZooKeeper .

For example, complete the following steps to download these files to C:\temp on Windows, where <install_dir> is C:\Program Files\Electric Cloud\ElectricCommander.

  1. Download the files from ZooKeeper by entering the following commands:

    cd C:\temp "C:\Program Files\Electric Cloud\ElectricCommander\jre\bin\java" -DCOMMANDER_ZK_CONNECTION=<ZooKeeper_Server_IP>:2181 -jar "C:\Program Files\Electric Cloud\ElectricCommander\server\bin\zk-config-tool-jar-with-dependencies.jar" com.electriccloud.commander.cluster.ZKConfigTool --readFile /commander/conf/database.properties database.properties "C:\Program Files\Electric Cloud\ElectricCommander\jre\bin\java" -DCOMMANDER_ZK_CONNECTION=<ZooKeeper_Server_IP>:2181 -jar "C:\Program Files\Electric Cloud\ElectricCommander\server\bin\zk-config-tool-jar-with-dependencies.jar" com.electriccloud.commander.cluster.ZKConfigTool --readFile /commander/conf/keystore keystore "C:\Program Files\Electric Cloud\ElectricCommander\jre\bin\java" -DCOMMANDER_ZK_CONNECTION=<ZooKeeper_Server_IP>:2181 -jar "C:\Program Files\Electric Cloud\ElectricCommander\server\bin\zk-config-tool-jar-with-dependencies.jar" com.electriccloud.commander.cluster.ZKConfigTool --readFile /commander/conf/passkey passkey "C:\Program Files\Electric Cloud\ElectricCommander\jre\bin\java" -DCOMMANDER_ZK_CONNECTION=<ZooKeeper_Server_IP>:2181 -jar "C:\Program Files\Electric Cloud\ElectricCommander\server\bin\zk-config-tool-jar-with-dependencies.jar" com.electriccloud.commander.cluster.ZKConfigTool --readFile /commander/conf/commander.properties
  2. Ensure the four files in C:\temp are the same as the ones in <data_dir>\conf (a file diff tool can make this easier). If any file in <data_dir>\conf is different, then back up that file and replace it with the one that you downloaded from ZooKeeper.

Backing up other files

The CloudBees Flow files that might have been modified are too numerous to list, so you should back up the entire CloudBees Flow data directory and other miscellaneous files that might have changed. But at a minimum, you must back up the following files:

  • The plugins directory. The default location is the plugins subdirectory within <data_dir>.

  • Files that contain your configuration and custom settings. To ensure that all important settings are saved, back up the following subdirectories in <data_dir> :

  • The entire conf subdirectory, which contains the CloudBees Flow server and agent configuration files.

  • Apache web server configuration files in the apache/conf subdirectory.

  • (If applicable) The local MySQL database configuration file, my.ini, in the mysql subdirectory.

  • Any other files where you created custom configurations, specified other custom information, or created any type of modification.

  • (If you use an artifact repository) The CloudBees Flow repository configuration files in the conf/repository subdirectory.

  • (If modified) The custom editor or preflight driver script properties (installed by default).

These properties are stored in the server-level property sheet, which you can view in the web UI in the Administration > Server subtab.

Custom editors are stored in the nested sheet named ec_customEditors. Preflight driver scripts are stored in the nested sheet named ec_preflight. The upgrade process overwrites default custom editor and preflight driver scripts with current versions. You should back up any custom properties that you created by renaming those properties. For example, change ec_preflight/clientDrivers/perforce to ec_preflight/clientDrivers/perforce_modified.

Upgrade installer preservation

After you back up your CloudBees Flow server, create a folder where you can download the CloudBees Flow-<version> installation file.

Starting the installation process

Install the upgrade by choosing the correct upgrade method for your environment.

Copying repository contents

After you have updated CloudBees Flow, perform the following steps to copy the contents of an existing repository server into a newly installed repository server:

  1. Install the new repository server software.

  2. Stop the existing and new repository servers.

  3. Copy the entire contents of the repository backingstore directory from the existing repository server to the corresponding location on the newly installed repository server.

    The default location for the backingstore directory (<datadir>/repository-data) is: UNIX/opt/electriccloud/electriccommander/repository-data WindowsC:\ProgramData\Electric Cloud\ElectricCommander\repository-data

MySQL considerations with server upgrades

CloudBees Flow upgrades involving a MySQL database can take several hours to complete if you have a significant data set.

To avoid corrupting your database, do not interrupt the upgrade process. A restore from a previous database backup would be required if this occurs.

You can use ectool to view the upgrade progress. On a command line, enter

ectool getServerStatus

An install or upgrade log file named installer.log is created in the logs subdirectory in <data_dir>.

Uploading configuration files to ZooKeeper if needed

After you upgrade the CloudBees Flow server node, you must again compare the <data_dir> \conf\commander.properties file with the file that you downloaded from ZooKeeper (which you saved to c:\temp). To do so, complete the following steps.

  1. Open the <data_dir> \conf\commander.properties file.

  2. Make sure that the COMMANDER_SERVER_NAME property is set to <load_balancer_FQDN> .

  3. If the following line exists, remove it:

    COMMANDER_MQ_DISK_SPACE_CHECK_FREQUENCY=60
  4. Check whether the following lines exist. If they do not exist, add them:

    COMMANDER_CRITICAL_SERVICES_MONITORING_FREQUENCY=60 COMMANDER_CRITICAL_SERVICES_MONITORING_ENABLED=true COMMANDER_CRITICAL_SERVICES_MAX_ATTEMPTS_TO_BE_IN_PRIMARY_CLUSTER=5

    Ensure that these properties are not duplicated in the file.

  5. Upload the new file to ZooKeeper as described in the Uploading Configuration Files to ZooKeeper section in the "Clustering" chapter.