Configure Disaster Recovery and recover from a disaster

This topic has the steps to configure a Disaster Recovery (DR) setup. It explains what to expect after a successful failover.

Disaster Recovery Environment Setup

As part of a DR environment setup, you need to set up a secondary CloudBees CD/RO site. This site must include a complete setup of all CloudBees CD/RO components including:

Database (Oracle, SQL Server, or MySQL)
CloudBees CD/RO server
Web server
CloudBees Analytics Server
Repository server
Zookeeper
Gateway agent
Agent

CloudBees CD/RO components that can be load balanced include the CloudBees CD/RO server, web server, CloudBees Analytics server, repository server, and gateway agent. Go to Installing and Configuring a Load Balancer for details.

Along with replicating the component setup, data stores also need to be replicated and made available for CloudBees CD/RO to operate the following:

Database (using the vendor-recommended replication)
CloudBees Analytics server (by restoring snapshots of indices)
Repository Server Data Store (replication of shared file locations where artifacts are stored)
Plugins (typically copied to a shared file location)
Certificates signed by a CloudBees CD/RO CA (Certificate Authority)

Configurations and Settings

Follow these guidelines on the primary and secondary sites:

Both the primary and secondary sites should be running the same version of CloudBees CD/RO.
Under normal operation, it should not be possible to do active transactions directly on the secondary (or replicated) database. That is, the secondary database should be in replication mode.
The CloudBees CD/RO server in the secondary site must be configured and must point to the secondary (or replicated) database.
These servers must not automatically restart after a reboot. That is, all CloudBees CD/RO servers in the secondary site should be set to start in Manual mode. This prevents inadvertent write operations into the replicated database. Details on recommended steps for setting up a secondary site appear below.
- CloudBees CD/RO server
- Web server
- CloudBees Analytics server service
We recommend using DNS Failover to minimize the downtime when moving from the primary site to the secondary site. This allows end users accessing the web servers to continue using the same URL. Agents that are running jobs can send their finish job notification to the secondary server, allowing the jobs to succeed.

IPv6 addresses are only supported for Kubernetes platforms. If using an IPv6 address, enclose the address in square brackets. Example: [<IPv6-ADDRESS>].

To configure Disaster Recovery, follow these steps:

Add following lines to the wrapper.conf file for the server nodes:

wrapper.java.additional.1600=DCOMMANDER_IGNORE_SERVER_MISMATCH=1▼

wrapper.java.additional.1601=DCOMMANDER_PRESERVE_SESSIONS=1▼

This avoids the following errors during the failover to the secondary site:

20160815T22:10:52.406 | 10.0.2.206 | DEBUG | bootstrap
|                                    | schemaMaintenance
| OperationInvoker |           Exception: InvalidServer: The ZooKeeper/Exhibitor
setting of the last cluster
('ip100179155:8080,ip100145251:8080,ip100133239:8080') to connect to the
database is different from the ZooKeeper/Exhibitor setting of the current cluster
('ip1008088:8080,ip100196103:8080,ip100515:8080'). Check that the cluster
is configured for the correct database. To allow this server cluster to become the new
owner for this data, update the database configuration with the ignoreServerMismatch
flag set.

20160815T22:10:52.406 | 10.0.2.206 | WARN | bootstrap |
|                                    |
| ServerStatus                 | InvalidServer: The ZooKeeper/Exhibitor setting of
the last cluster ('ip100179155:8080,ip100145251:8080,ip100133239:8080') to
connect to the database is different from the ZooKeeper/Exhibitor setting of the
current cluster ('ip1008088:8080,ip100196103:8080,ip100515:8080'). Check
that the cluster is configured for the correct database. To allow this server cluster
to become the new owner for this data, update the database configuration with the
ignoreServerMismatch flag set.▼

Make sure that the following recommended standard setup steps are performed:

commander.properties in ZooKeeper should have the COMMANDER_SERVER_NAME set to the FQDN (Fully Qualified Domain Name) of the load balancer of the CloudBees CD/RO server cluster. This can be set using the following command:
```
ecconfigure --serverName <FLOW_SERVER_LOAD_BALANCER_FQDN>▼
```

Both the primary and secondary sites should have the same files (including content):

keystoreFile
passkeyFile

commander.properties

In the cluster setup, these files are stored in their respective ZooKeeper instances (the primary and secondary instances).

The ZooKeeper connection is configured using: ecconfigure --serverZooKeeperConnection <ZooKeeper_servers_comma_separated_list> For example: ecconfigure –-serverZooKeeperConnection ip1008088:2181,ip100196103:2181,ip100515:2181

Run the following command on the web servers to ensure that the COMMANDER_SERVER property in the httpd.conf file is set to CloudBees CD/RO server’s load balancer FQDN:
```
ecconfigure --webTargetHostName <FLOW_SERVER_LOAD_BALANCER_FQDN>▼
```
The httpd.conf file is usually in apache/conf on a Linux machine and ProgramData\CloudBees\Software Delivery Automation\apache\conf on a Windows machine.

The --webTargetHostName argument modifies the CloudBees CD/RO web server configuration and therefore also attempts to restart the CloudBees CD/RO web server. If you used the ecconfigure command without sudo as recommended, the commanderApache service will not start and produces an error. Therefore, you must restart it manually afterward using sudo. You can also use the --skipServiceRestart argument to avoid the ecconfigure command’s restart attempt and the error message.

Similarly, on each repository server, run the following command to set COMMANDER_HOST in the server.properties file:

ecconfigure --repositoryTargetHostName <FLOW_SERVER_LOAD_BALANCER_FQDN>▼

These are the default locations for the server.properties file:

In Linux:

/opt/cloudbees/sda/conf/repository/server.properties

In Windows:

C:\ProgramData\CloudBees\Software Delivery Automation\conf\repository\server.properties

Set the Server IP address in the CloudBees CD/RO server property to the CloudBees CD/RO Server Load Balancer FQDN (the same value as <FLOW_SERVER_LOAD_BALANCER_FQDN> ).

IPv6 addresses are only supported for Kubernetes platforms. If using an IPv6 address, enclose the address in square brackets. Example: [<IPv6-ADDRESS>].

Set the Stomp Client URI server property to stomp+ssl://<FLOW_SERVER_LOAD_BALANCER_FQDN>:61613.

If the load balancer does SSL termination, uncheck the Use SSL for Stomp option. All the CloudBees CD/RO server nodes must be restarted to affect this change, because by default, the check box is selected.

Navigate to the Dev Ops Essentials Platform Home Page Cloud Resource page: register all the CloudBees CD/RO local agents (the ones that run on same machine as CloudBees CD/RO server) from both primary and secondary sites. Make sure that for the cluster setup, you do not have local resources.

Create resource pools named local and default. For both, add CloudBees CD/RO local agents.

Both the local and default pools are used by the CloudBees CD/RO standard job processing. For example, sentry jobs run on local pool resources.

(Optional) If trusted agents are used, in addition to copying the keystore file, along with keystore file, copy the conf/confSecurityFolder folder to the secondary site’s ZooKeeper. This folder stores the CloudBees CD/RO Certificate Authority information along with the certificates that are signed by CloudBees CD/RO.

Perform the following steps to copy this folder from the primary to secondary site:

Log into the primary site’s CloudBees CD/RO server, and run the following command to get the conf/confSecurityFolder folder from the primary ZooKeeper into the local /tmp/CloudBees CD/RO Automation Platform/conf/confSecurityFolder folder:

Linux:

COMMANDER_ZK_CONNECTION=<ZooKeeper_Primary_Server_IP>:2181
<install_dir>/jre/bin/java -cp \
        <install_dir>/server/bin/zk-config-tool-jar-with-dependencies.jar
com.electriccloud.commander.zkconfig.ZKConfigTool --readFolder \
        /commander/conf/confSecurityFolder \
        /tmp/{PRODUCT} Automation Platform/conf/confSecurityFolder▼

Windows:

"C:\Program Files\CloudBees\Software Delivery Automation\jre\bin\java.exe" \
       -DCOMMANDER_ZK_CONNECTION=<ZooKeeper_Primary_Server_IP>:2181 -jar \
"C:\Program Files\CloudBees\Software Delivery \
       Automation\server\bin\zk-config-tool-jar-with-dependencies.jar"
com.electriccloud.commander.cluster.ZKConfigTool --readFolder \
       /commander/conf/confSecurityFolder \
       c:\<path>\Automation Platform\conf\confSecurityFolder▼

Log into the secondary site’s CloudBees CD/RO server, and run the following command to upload the conf/confSecurityFolder folder from the local folder to ZooKeeper:

Linux:

COMMANDER_ZK_CONNECTION=<ZooKeeper_Secondary_Server_IP>:2181 \
<install_dir>/jre/bin/java -cp \
    <install_dir>/server/bin/zk-config-tool-jar-with-dependencies.jar
com.electriccloud.commander.zkconfig.ZKConfigTool \
    --writeFolder /commander/conf/confSecurityFolder \
    /tmp/Automation Platform/conf/confSecurityFolder`▼

Windows:

"C:\Program Files\CloudBees\Software Delivery Automation\jre\bin\java.exe" \
     -DCOMMANDER_ZK_CONNECTION=<ZooKeeper_Secondary_Server_IP>:2181 \
     -jar "C:\Program Files\CloudBees\Software Delivery \
     Automation\server\bin\zk-config-tool-jar-with-dependencies.jar"
com.electriccloud.commander.cluster.ZKConfigTool --writeFolder \
     /commander/conf/confSecurityFolder \
     c:\<path>\{PRODUCT} Automation Platform\conf\security▼

Set the repository data store on the primary and secondary sites.

Repository servers set up on primary and secondary sites can share the same repository data store.

The repository data can be replicated, and the repository servers can point to the respective data store locations. For each repository server, set the REPOSITORY_BACKING_STORE in the server.properties file to a UNC path on a network share on the file server.

For example:
```
REPOSITORY_BACKING_STORE=//10.0.109.72/repo_data/repositorydata▼
```
Register the repository server in the CloudBees CD/RO UI. If the repository server cluster is set up, use the load balancer URL. For example, https://<REPOSITORY_SERVER_LOAD_BALANCER_FQDN>:8200.

During the failover to the secondary site, FQDN should point to the repository servers in the secondary site.
It is recommended that the plugins folder in the network share must be accessible from the remote web servers as mentioned in Universal Access to the Plugins Directory .
For the initial installation and setup of the secondary site, perform the following recommended steps:
1. Set up the secondary database in normal or nonreplicated mode.
2. Follow instructions as described in this installation guide to set up all the servers, including the CloudBees CD/RO, web, repository, and Zookeeper servers.
3. Make sure that the database.properties file is set up to point to the correct secondary site’s database server. This file will be stored in ZooKeeper when the cluster setup is used. For the primary site, it should point to the primary database. For the secondary site, it should point to the secondary database.
4. Ensure that all the servers are running properly, including the connection to the database. At this time, the secondary site is not set up for replication, and operates as a separate installation of CloudBees CD/RO.
5. Before setting up the secondary site’s database in replication mode, shut down all secondary CloudBees CD/RO servers, web servers, repository servers, and others. After these steps are performed, there should not be any write transactions to the secondary database.
6. The first time that the secondary database is set up, a schema for database tables is created. Before proceeding to set this database with replication, this schema may have to be deleted. This avoids a schema name conflict when replication is enabled.
7. Based on the disaster recovery option chosen for the database, set up the secondary database in replication mode.

Disaster Recovery Environment Setup for CloudBees Analytics Server

Follow the steps below to include the CloudBees Analytics server in your disaster recovery environment setup.

Setup identical CloudBees Analytics server installations on the primary site and the secondary sites.
Ensure that the CloudBees CD/RO server on the primary site is configured to point to the FQDN (Fully Qualified Domain Name) of the load balancer of the CloudBees Analytics server cluster in the primary site. Similarly, the CloudBees CD/RO server on the secondary site must be configured to point to the FQDN (Fully Qualified Domain Name) of the load balancer of the CloudBees Analytics server cluster in the secondary site.
Use snapshots to create backups of indices from the primary site at regular intervals, for example, daily. The backups can be stored in a shared file system or on AWS S3 storage. See the section CloudBees Analytics Server OpenSearch Data Backups for details on creating snapshots.
Restore the snapshots to the CloudBees Analytics server cluster running on the secondary site at regular intervals, for example, daily or weekly. You need to start the CloudBees Analytics server service on the secondary site for restoring the snapshots. You may choose to move the snapshot files to a different location for backup and archiving purposes once they have been restored on the secondary site.

Steps to Perform During a Disaster Recovery Failover

When a disaster event happens that interrupts the operations on the primary site, follow these steps to move the operations to the secondary site.

Shut down any services that might still be running on the primary site. With the exception of the CloudBees Analytics server, all other components including the database, CloudBees CD/RO server, and repository server can be shut down. Doing this ensures that no more transactions happen on the primary site. For the CloudBees Analytics server see Disaster Recovery Failover Steps for CloudBees Analytics Server below.
Begin switching operations to the secondary site by restoring and updating the secondary site’s database with the latest data. The steps to do this may vary based on the disaster recovery method used for database.
Delete the brokerdata folder on the CloudBees CD/RO server nodes. For example, in Windows, delete the C:\ProgramData\CloudBees\Software Delivery Automation\brokerdata folder.
Follow the DNS failover procedure, and update the DNS entries to point to servers in the secondary site. This includes updating the entries for servers including web server, CloudBees CD/RO server, and repository server. There may also be other servers based on configuration.
Bring up all the servers, with the exception of the CloudBees Analytics server, and infrastructure in the secondary site. Start the CloudBees CD/RO services on different machines, including the CloudBees CD/RO server, web server, repository server, and gateway agents. For CloudBees Analytics server see Disaster Recovery Failover Steps for CloudBees Analytics Server below.
Based on the nature of the disaster event, certain active operations running on the primary site may be interrupted, and need to be restarted. For example, a build or deploy application process may fail and error out. Use the CloudBees CD/RO UI to review such failures, and take the appropriate corrective actions, usually by executing those failed processes again.
The secondary’s site database now acts as the server. We recommend that you set up a database that replicates to another database server.

Disaster Recovery Failover Steps for a CloudBees Analytics Server

If the CloudBees Analytics server service is still running on the primary site, then create a final snapshot before shutting down the service.
Restore any snapshots created since the last scheduled restoration on the secondary site.

The secondary site’s CloudBees Analytics server now acts as the primary cluster. We recommend that you setup a schedule for creating snapshots from this cluster and restoring into another cluster.

Server Maintenance

See Maintenance for details on CloudBees CD/RO server maintenance.