Issue
After restoring a backup from a production controller to an independent Disaster Recovery (DR) controller without using file exclusions, the DR controller fails to connect or operate normally. The logs are flooded with the following errors indicating an Operations Center overlap:
WARNING c.c.o.c.p.OperationsCenterRootAction#invokeServerAction: Received a unhandled status code - HTTP 403. This is probably a bug. Please set up a logger on com.cloudbees.opscenter.client.plugin.OperationsCenterRootAction with level FINE to get request details, then open a support ticket to CloudBees. ... <h2>HTTP ERROR 403 Bad signature for <controller-id></h2> ... [OperationsCenter2 connection to <host>] Local headers refused by remote: The controller <controller-id> is already connected
Explanation
When a full backup is restored without excluding identity and configuration files, the DR controller inherits the production controller’s connection settings and unique identity (identity.key.enc). On boot, it bypasses the DR Operations Center and attempts to register with the production Operations Center using the cloned controller ID.
== Resolution
To resolve the issue, you must remove the copied identity files from the DR controller so it generates a new identity, and then re-register it with the correct DR Operations Center.
Do not delete secret.key or the $JENKINS_HOME/secrets/ directory. These files work together to decrypt passwords and SSH keys. If deleted, you will permanently lose access to all credentials copied from the production environment.
|
-
Stop the DR controller.
-
In the DR controller’s
$JENKINS_HOME, delete the following files to force Jenkins to generate a fresh identity on the next boot:-
identity.key.enc -
com.cloudbees.opscenter.client.plugin.OperationsCenterRootAction.xml -
operations-center-cloud* -
operations-center-client* -
license.xml
-
-
Edit
$JENKINS_HOME/jenkins.model.JenkinsLocationConfiguration.xmland confirm the Jenkins URL points to the DR URL (not the production URL pulled in from the backup). -
Check
$JENKINS_HOME/config.xmlfor any other production Operations Center URL references and update them to the DR Operations Center URL. -
Start the DR controller. It will automatically generate a new
identity.key.enc. -
Reconnect the DR controller to the DR Operations Center by following How to reconnect a client controller stuck on registration.
Prevention and Best Practices
To prevent this issue during future restores, configure the CloudBees Backup plugin to exclude the identity and connection files. Add the following to your restore exclusion list:
identity.key.enc, com.cloudbees.opscenter.client.plugin.OperationsCenterRootAction.xml, operations-center-cloud*, operations-center-client*, license.xml ---