Upgrading in a Clustered Environment

This section describes how to upgrade the CloudBees Flow 5.x, or newer versions, and upgrade cluster configurations at the same time. The procedure is the same as when you upgrade from CloudBees Flow 5.x, or newer versions, except that you need to perform additional tasks to upgrade the cluster.

To upgrade from CloudBees Flow 5. x or newer versions to 9.x , use the ElectricFlow-<version> installer, which collects the CloudBees Flow service account credentials, uninstalls the current release, installs the latest CloudBees Flow release, configures the system with all property values mined, and restores custom files and data.

After preparing for the upgrade, make sure to shut down the CloudBees Flow server service before installing CloudBees Flow 9.x .

Upgrading Clusters

In a cluster configuration, make sure to review the tasks in Preparing for your cluster upgrade . To ensure that all important settings are saved, back up the following subdirectories in DATA_DIR :

  • The entire conf subdirectory, which contains the CloudBees Flow server and agent configuration files

  • Apache web server configuration files in the apache/conf subdirectory

Before the upgrade, you must shut down the CloudBees Flow server service before installing CloudBees Flow 9.x .

During the upgrade, CloudBees Flow is not available after the web server is stopped or after the server service on the last CloudBees Flow server node is stopped.

  1. Perform either of these actions:

    • Stop the CloudBees Flow server service on all nodes.

    • Stop the CloudBees Flow server service on all nodes except on the primary CloudBees Flow server node.

      You can keep the server service on your primary CloudBees Flow server node up, because the installer stops the server service automatically on that node when it upgrades the node.

      When upgrading the nodes in a CloudBees Flow cluster, you must keep the other nodes stopped until the primary node upgrade is complete.
  2. On the primary CloudBees Flow server, do the following:

    1. Upgrade the CloudBees Flow server. This will also do the following:

      • Connects the server to the database

      • Upgrades the plugins

      • Starts the server

    2. Check and restore the wrapper.conf settings. For example, the settings for the line " wrapper.java.additional.600= ”.

    3. Restart the CloudBees Flow servers service.

  3. On the CloudBees Flow web server, do the following:

    1. Upgrade the node on the web server.

    2. Check and restore the httpd.conf settings.

  4. Upgrade any nodes on the repository servers.

    Your CloudBees Flow system is now available.

  5. Upgrade the remaining CloudBees Flow server nodes.

During the upgrade, some settings may be lost. Verify the following settings before connecting to the CloudBees Flow system:

  • httpd.conf settings for redirecting—These lines should be commented out:

    # Redirect http to https

    # RewriteCond %{HTTPS} !=on

    # RewriteRule ^/commander/(.*) https://%{SERVER_NAME}:443%{REQUEST_URI} [NC,R,L]

  • httpd.conf setting for COMMANDER_SERVER –This should point to the load balancer:

    SetEnv COMMANDER_SERVER "<FQDN of your load balancer>"

  • wrapper.conf contains the line that points to your Zookeeper instances.

    For example:

    wrapper.java.additional.600=-DCOMMANDER_ZK_CONNECTION=192.168.7.20:2181

Preparing for Your Upgrade

Review the following information before you upgrade CloudBees Flow.

Upgrade Testing

In most implementations, CloudBees Flow is used in an environment that affects many users. You should test your upgrade on a separate test server to understand all aspects of the upgrade process. This minimizes the potential impact on downstream users.

Backing Up Your Existing CloudBees Flow Data

Before upgrading a CloudBees Flow server, you must back up your existing CloudBees Flow data. See CloudBees Flow Server Backups for more information about backups.

Backing Up Commander Cluster Configuration Files

The configuration files for the Commander cluster are in <data_dir>\conf. The default location is:

  • Linux: /opt/electriccloud/electriccommander/conf/

  • Windows: `C:\ProgramData\Electric Cloud\ElectricCommander\conf `

Although the Commander cluster configuration files such as commander.properties, database.properties, keystore, and passkey are present in one of the directories above, they are not actually used by the cluster during runtime.

These files were uploaded to Apache ZooKeeper from the first node that was clustered as described in Uploading Configuration Files to ZooKeeper . You can download these files from ZooKeeper to a temporary folder and then compare them with those in the \conf folder. You can do so by using the CloudBees Flow ZKConfigTool, which is discussed in Uploading Configuration Files to ZooKeeper .

For example, complete the following steps to download these files to C:\temp on Windows, where <install_dir> is C:\Program Files\Electric Cloud\ElectricCommander.

  1. Download the files from ZooKeeper by entering the following commands:

    cd C:\temp

    "C:\Program Files\Electric Cloud\ElectricCommander\jre\bin\java" -DCOMMANDER_ZK_CONNECTION=<ZooKeeper_Server_IP>:2181 -jar "C:\Program Files\Electric Cloud\ElectricCommander\server\bin\zk-config-tool-jar-with-dependencies.jar" com.electriccloud.commander.cluster.ZKConfigTool --readFile /commander/conf/database.properties database.properties

    "C:\Program Files\Electric Cloud\ElectricCommander\jre\bin\java" -DCOMMANDER_ZK_CONNECTION=<ZooKeeper_Server_IP>:2181 -jar "C:\Program Files\Electric Cloud\ElectricCommander\server\bin\zk-config-tool-jar-with-dependencies.jar" com.electriccloud.commander.cluster.ZKConfigTool --readFile /commander/conf/keystore keystore

    "C:\Program Files\Electric Cloud\ElectricCommander\jre\bin\java" -DCOMMANDER_ZK_CONNECTION=<ZooKeeper_Server_IP>:2181 -jar "C:\Program Files\Electric Cloud\ElectricCommander\server\bin\zk-config-tool-jar-with-dependencies.jar" com.electriccloud.commander.cluster.ZKConfigTool --readFile /commander/conf/passkey passkey

    "C:\Program Files\Electric Cloud\ElectricCommander\jre\bin\java" -DCOMMANDER_ZK_CONNECTION=<ZooKeeper_Server_IP>:2181 -jar "C:\Program Files\Electric Cloud\ElectricCommander\server\bin\zk-config-tool-jar-with-dependencies.jar" com.electriccloud.commander.cluster.ZKConfigTool --readFile /commander/conf/commander.properties

  2. Make sure that the four files in C:\temp are the same as the ones in <data_dir>\conf. (You can use a file diff tool to make this easier.) If any file in <data_dir>\conf is different, then back up that file and replace it with the one that you downloaded from ZooKeeper.

Backing Up Other Files

The CloudBees Flow files that might have been modified are too numerous to list, so you should back up the entire CloudBees Flow data directory and other miscellaneous files that might have changed. But at a minimum, you must back up the following files:

  • The plugins directory. The default location is the plugins subdirectory within <data_dir>.

  • Files that contain your configuration and custom settings. To ensure that all important settings are saved, back up the following subdirectories in <data_dir> :

  • The entire conf subdirectory, which contains the CloudBees Flow server and agent configuration files.

  • Apache web server configuration files in the apache/conf subdirectory.

  • (If applicable) The local MySQL database configuration file, my.ini, in the mysql subdirectory.

  • Any other files where you created custom configurations, specified other custom information, or created any type of modification.

  • (If you use an artifact repository) The CloudBees Flow repository configuration files in the conf/repository subdirectory.

  • (If modified) The custom editor or preflight driver script properties (installed by default).

These properties are stored in the server-level property sheet, which you can view in the web UI in the Administration > Server subtab.

Custom editors are stored in the nested sheet named ec_customEditors. Preflight driver scripts are stored in the nested sheet named ec_preflight. The upgrade process overwrites default custom editor and preflight driver scripts with current versions. You should back up any custom properties that you created by renaming those properties. For example, change ec_preflight/clientDrivers/perforce to ec_preflight/clientDrivers/perforce_modified.

Upgrade Installer Preservation

After you back up your CloudBees Flow server, create a folder where you can download the CloudBees Flow-<version> installation file.

MySQL Upgrades

CloudBees Flow upgrades involving a MySQL database can take several hours to complete if you have a significant data set.

To avoid corrupting your database, do not interrupt the upgrade process. A restore from a previous database backup would be required if this occurs.

You can use ectool to view the upgrade progress. On a command line, enter

ectool getServerStatus

An install or upgrade log file named installer.log is created in the logs subdirectory in <data_dir>.

Uploading Configuration Files to ZooKeeper if Needed

After you upgrade the CloudBees Flow server node, you must again compare the <data_dir> \conf\commander.properties file with the file that you downloaded from ZooKeeper (which you saved to c:\temp). To do so, complete the following steps.

  1. Open the <data_dir> \conf\commander.properties file.

  2. Make sure that the COMMANDER_SERVER_NAME property is set to <load_balancer_FQDN> .

  3. If the following line exists, remove it:

    COMMANDER_MQ_DISK_SPACE_CHECK_FREQUENCY=60
  4. Check whether the following lines exist. If they do not exist, add them:

    COMMANDER_CRITICAL_SERVICES_MONITORING_FREQUENCY=60
    COMMANDER_CRITICAL_SERVICES_MONITORING_ENABLED=true
    COMMANDER_CRITICAL_SERVICES_MAX_ATTEMPTS_TO_BE_IN_PRIMARY_CLUSTER=5

    Ensure that these properties are not duplicated in the file.

  5. Upload the new file to ZooKeeper as described in the Uploading Configuration Files to ZooKeeper section in the "Clustering" chapter.