Clouds defined in the Operations Center are not pushed to the Connected controllers

2 minute read

Issue

  • A Shared cloud is defined in the Operations Center but it is not recognized by the Connected controllers.

  • Updates on an existing Shared Cloud item are no propagated/pushed to the Connected controllers.

  • A stacktrace similar to the one below appears in the Operations Center logs.

2020-11-19 09:39:04.444+0000 [id=32] WARNING c.c.o.s.model.ConnectedMaster#rebuildRemoteContext: Uncaught exception from com.cloudbees.opscenter.server.config.ConnectedMasterConfigurationOptOutProperty$RemoteContextContributorImpl@7fe1c4b4 on Managedcontroller{id=30, name='controller-jenkins', encodedName='controller-jenkins', idName='30-controller-jenkins', timeStamp=0, error=null, grantId='ddf4ae57-8516-4ee4-b30e-78946b17c854', approved=true, persistedState=PersistedStateImpl{state=APPROVED, futureFilesystem=null, filesystem=null, futureResource=null, resource=KubernetescontrollerResource [namespace=controller-jenkins, fsGroup=1000, getName()=controller-jenkins, getEndpoint()=https://master.cloudbees.example.com/master-jenkins/, getCpus()=1.0, getMemory()=3072.0, getRatio()=0.5, getImage()=DockerImageDefinition{imageTag='cloudbees/cloudbees-core-mm:2.190.3.2', name='CloudBees Core - Managed controller - 2.190.3.2'}]}, localHome='null', localEndpoint=https://master.cloudbees.example.com/master-jenkins/, channel=hudson.remoting.Channel@262ee43e:OperationsCenter2 connection from controller-jenkins-0.controller-jenkins.controller-jenkins.svc.cluster.local/10.200.46.217:39068, identity=X.509, RSA} java.lang.NullPointerException at com.cloudbees.opscenter.context.config.CloudConfigurationSnippetHandler.populate(CloudConfigurationSnippetHandler.java:111) at com.cloudbees.opscenter.context.config.ConfigurationSnippetHandler.contribute(ConfigurationSnippetHandler.java:128) at com.cloudbees.opscenter.server.config.ConnectedMasterConfigurationOptOutProperty$RemoteContextContributorImpl.contribute(ConnectedMasterConfigurationOptOutProperty.java:213) at com.cloudbees.opscenter.server.config.ConnectedMasterConfigurationOptOutProperty$RemoteContextContributorImpl.contribute(ConnectedMasterConfigurationOptOutProperty.java:314) at com.cloudbees.opscenter.server.config.ConnectedMasterConfigurationOptOutProperty$RemoteContextContributorImpl.contribute(ConnectedMasterConfigurationOptOutProperty.java:197) at com.cloudbees.opscenter.server.model.ConnectedMaster.rebuildRemoteContext(ConnectedMaster.java:800) at com.cloudbees.opscenter.server.model.ConnectedMaster.updateRemoteContext(ConnectedMaster.java:757) at com.cloudbees.opscenter.server.model.ConnectedMaster$UpdateRemoteContexts.doRun(ConnectedMaster.java:2354) at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:91) at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)

Resolution

Expected Behaviour

Changes on the Operation Center Shared Cloud (e.g. Kubernetes Shared Cloud)

2020-12-18 15:26:57.439+0000 [id=132655] INFO c.c.o.s.model.ConnectedMaster#updateRemoteContext: Updating context for Managedcontroller{id=49, name='controller-demo', encodedName='controller-demo', ...... as it was out of date 2020-12-18 15:27:36.885+0000 [id=132779] INFO c.c.o.c.c.CloudConfigurationSnippetHandler#consume: Applied/updated Clouds configuration [kubernetes, kubernetes]

are propagated to the Connected Master

2020-12-18 15:24:51.510+0000 [id=56] INFO c.c.o.c.c.CloudConfigurationSnippetHandler#consume: Applied/updated Clouds configuration [kubernetes, Operations Center Agent Provisioning Service]

Most likely Root Cause

In most cases, this issue is produced by a Shared Configuration item (e.g: AWS EC2 Cloud Configuration, Miscellaneous Configuration Container o Kubernetes Cloud Configuration) in the Operation Center which configuration is not filled (empty fields). As a result, the Operations Center fails trying to push the updates towards the controllers.

Workaround

The following steps aim to identify which Shared Configuration item is empty in order to amend or delete it.

1.- At the UI level, run the following script in the Operation Center Manage Jenkins -> Script Console in the Operations Center to list all the Shared Configuration items and their full paths

Jenkins.instance.getAllItems(com.cloudbees.opscenter.server.config.SharedConfiguration).each { println it.getName() } return null

Example output

ec2-cloud-configuration-shared kubernetes-shared-cloud new_cloud email Result: [com.cloudbees.opscenter.clouds.ec2.EC2Configuration@141d831[ec2-cloud-configuration-shared], com.cloudbees.opscenter.clouds.kubernetes.KubernetesConfiguration@41339d90[kubernetes-shared-cloud], com.cloudbees.opscenter.clouds.kubernetes.KubernetesConfiguration@3467e53b[new_cloud], com.cloudbees.opscenter.server.config.SharedConfiguration@1a78fa7a[email]]

2.- At File System level, navigate to the Operation Center jobs folders and run the following

grep -R -s -e "</com.cloudbees.opscenter.server.config.SharedConfiguration>" -e "</com.cloudbees.opscenter.clouds.ec2.EC2Configuration>" -e "</com.cloudbees.opscenter.clouds.kubernetes.KubernetesConfiguration>" . --exclude-dir=build

Example output

./jobs/kubernetes-shared-cloud/config.xml:</com.cloudbees.opscenter.clouds.kubernetes.KubernetesConfiguration> ./jobs/new_cloud/config.xml:</com.cloudbees.opscenter.clouds.kubernetes.KubernetesConfiguration> ./jobs/email/config.xml:</com.cloudbees.opscenter.server.config.SharedConfiguration> ./jobs/ec2-cloud-configuration-shared/config.xml:</com.cloudbees.opscenter.clouds.ec2.EC2Configuration>
grep -R -s -h -e "</com.cloudbees.opscenter.server.config.SharedConfiguration>" -e "</com.cloudbees.opscenter.clouds.ec2.EC2Configuration>" . -e "</com.cloudbees.opscenter.clouds.kubernetes.KubernetesConfiguration>" --exclude-dir=build -C 200`

Example output

<?xml version='1.1' encoding='UTF-8'?> <com.cloudbees.opscenter.clouds.kubernetes.KubernetesConfiguration plugin="operations-center-kubernetes-cloud@2.263.0.1"> <actions/> <description></description> <snippets> <com.cloudbees.opscenter.clouds.kubernetes.KubernetesCloudConfigurationSnippet> <value> <string>&lt;org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud plugin=&quot;kubernetes@1.28.3&quot;&gt; &lt;name&gt;kubernetes-oc&lt;/name&gt; &lt;defaultsProviderTemplate&gt;&lt;/defaultsProviderTemplate&gt; ... </com.cloudbees.opscenter.clouds.kubernetes.KubernetesConfiguration> -- <?xml version='1.1' encoding='UTF-8'?> <com.cloudbees.opscenter.clouds.kubernetes.KubernetesConfiguration plugin="operations-center-kubernetes-cloud@2.263.0.1"> <actions/> <description></description> <snippets> <com.cloudbees.opscenter.clouds.kubernetes.KubernetesCloudConfigurationSnippet> <value> <string>&lt;org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud plugin=&quot;kubernetes@1.28.3&quot;&gt; &lt;name&gt;kubernetes&lt;/name&gt; &lt;defaultsProviderTemplate&gt;&lt;/defaultsProviderTemplate&gt; ... </com.cloudbees.opscenter.clouds.kubernetes.KubernetesConfiguration> -- <?xml version='1.1' encoding='UTF-8'?> <com.cloudbees.opscenter.server.config.SharedConfiguration plugin="operations-center-server@2.235.0.2"> <snippets/> <properties/> </com.cloudbees.opscenter.server.config.SharedConfiguration> -- <?xml version='1.1' encoding='UTF-8'?> <com.cloudbees.opscenter.clouds.ec2.EC2Configuration plugin="operations-center-ec2-cloud@2.235.0.2"> <actions/> <description></description> <snippets> <com.cloudbees.opscenter.clouds.ec2.EC2CloudConfigurationSnippet> <value> <string>&lt;hudson.plugins.ec2.EC2Cloud plugin=&quot;ec2@1.50.3&quot;&gt; &lt;name&gt;ec2-ec2-cloud-configuration-shared&lt;/name&gt; ... </com.cloudbees.opscenter.clouds.ec2.EC2Configuration>

In the provided example, we observe that email was an empty Miscellaneous Configuration Container after removing it the issue was solved.

Once the Shared Configuration items have been fixed, restart the Operation Center. If the problematic Shared Cloud in the Operations Center still not be recognized by the connected controllers, take a BackUp of the Operation Center and then delete all the Shared Configuration items apart from the Shared Cloud you are troubleshooting. In case it fixes the NullPointerException, start adding one by one the required Shared Configuration items to understand which is the problematic one.