Issue
When an Amazon EC2 cloud is defined as a shared cloud at the operations center and pushed to a controller running in High Availability (active/active) mode, the EC2 cloud intermittently loses all of its templates on the replicas that are not connected to the operations center.
The following symptoms appear:
-
CloudConfigurationSnippetHandler#consume: Applied/updated Clouds configuration […]is logged every 60 seconds on each replica. -
The EC2 cloud’s template count flips between the expected number and
0, as the replicas repeatedly overwrite each other’s cloud configuration. -
A rolling restart of the controller temporarily resolves the symptom until the next operations center context update.
Environment
-
CloudBees CI on modern cloud platforms - managed controller in High Availability (active/active)
-
Operations Center Server EC2 Cloud plugin on the operations center
-
Amazon EC2 plugin on the controller
Resolution
A fix is pending inclusion in a future CloudBees CI release, this article will be updated once the fix is available.
An alternative solution is to not rely on the operations center shared cloud for the EC2 cloud configuration and instead define it directly on the affected controller using controller-level Configuration as Code (CasC).
Workaround
-
Move the EC2 cloud configuration from the operations center shared cloud to controller-level Configuration as Code (CasC) on the affected controller.
-
This is the recommended long-term solution as well, since defining the cloud directly on the controller avoids the operations center shared-cloud replication path entirely.
-
If the EC2 shared cloud cannot be removed from the operations center because other controllers still rely on it:
-
Install the Extension Filter Plugin on the affected controller.
-
Block the
com.cloudbees.opscenter.context.config.CloudConfigurationSnippetHandlerextension to prevent all shared cloud configurations from syncing from the operations center. -
Define the clouds directly on the controller, for example through CasC, once the sync is blocked.
-