Issue
-
A Shared cloud is defined in the Operations Center but it is not recognized by the Connected controllers.
-
Updates on an existing Shared Cloud item are no propagated/pushed to the Connected controllers.
-
A stacktrace similar to the one below appears in the Operations Center logs.
2020-11-19 09:39:04.444+0000 [id=32] WARNING c.c.o.s.model.ConnectedMaster#rebuildRemoteContext: Uncaught exception from com.cloudbees.opscenter.server.config.ConnectedMasterConfigurationOptOutProperty$RemoteContextContributorImpl@7fe1c4b4 on Managedcontroller{id=30, name='controller-jenkins', encodedName='controller-jenkins', idName='30-controller-jenkins', timeStamp=0, error=null, grantId='ddf4ae57-8516-4ee4-b30e-78946b17c854', approved=true, persistedState=PersistedStateImpl{state=APPROVED, futureFilesystem=null, filesystem=null, futureResource=null, resource=KubernetescontrollerResource [namespace=controller-jenkins, fsGroup=1000, getName()=controller-jenkins, getEndpoint()=https://master.cloudbees.example.com/master-jenkins/, getCpus()=1.0, getMemory()=3072.0, getRatio()=0.5, getImage()=DockerImageDefinition{imageTag='cloudbees/cloudbees-core-mm:2.190.3.2', name='CloudBees Core - Managed controller - 2.190.3.2'}]}, localHome='null', localEndpoint=https://master.cloudbees.example.com/master-jenkins/, channel=hudson.remoting.Channel@262ee43e:OperationsCenter2 connection from controller-jenkins-0.controller-jenkins.controller-jenkins.svc.cluster.local/10.200.46.217:39068, identity=X.509, RSA} java.lang.NullPointerException at com.cloudbees.opscenter.context.config.CloudConfigurationSnippetHandler.populate(CloudConfigurationSnippetHandler.java:111) at com.cloudbees.opscenter.context.config.ConfigurationSnippetHandler.contribute(ConfigurationSnippetHandler.java:128) at com.cloudbees.opscenter.server.config.ConnectedMasterConfigurationOptOutProperty$RemoteContextContributorImpl.contribute(ConnectedMasterConfigurationOptOutProperty.java:213) at com.cloudbees.opscenter.server.config.ConnectedMasterConfigurationOptOutProperty$RemoteContextContributorImpl.contribute(ConnectedMasterConfigurationOptOutProperty.java:314) at com.cloudbees.opscenter.server.config.ConnectedMasterConfigurationOptOutProperty$RemoteContextContributorImpl.contribute(ConnectedMasterConfigurationOptOutProperty.java:197) at com.cloudbees.opscenter.server.model.ConnectedMaster.rebuildRemoteContext(ConnectedMaster.java:800) at com.cloudbees.opscenter.server.model.ConnectedMaster.updateRemoteContext(ConnectedMaster.java:757) at com.cloudbees.opscenter.server.model.ConnectedMaster$UpdateRemoteContexts.doRun(ConnectedMaster.java:2354) at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:91) at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)
Environment
-
CloudBees CI (CloudBees Core) on modern cloud platforms - Managed controller
-
CloudBees CI (CloudBees Core) on modern cloud platforms - Operations Center
-
CloudBees CI (CloudBees Core) on traditional platforms - Client controller
-
CloudBees CI (CloudBees Core) on traditional platforms - Operations Center
-
Linux OS
Resolution
Expected Behaviour
Changes on the Operation Center Shared Cloud (e.g. Kubernetes Shared Cloud)
2020-12-18 15:26:57.439+0000 [id=132655] INFO c.c.o.s.model.ConnectedMaster#updateRemoteContext: Updating context for Managedcontroller{id=49, name='controller-demo', encodedName='controller-demo', ...... as it was out of date 2020-12-18 15:27:36.885+0000 [id=132779] INFO c.c.o.c.c.CloudConfigurationSnippetHandler#consume: Applied/updated Clouds configuration [kubernetes, kubernetes]
are propagated to the Connected Master
2020-12-18 15:24:51.510+0000 [id=56] INFO c.c.o.c.c.CloudConfigurationSnippetHandler#consume: Applied/updated Clouds configuration [kubernetes, Operations Center Agent Provisioning Service]
Most likely Root Cause
In most cases, this issue is produced by a Shared Configuration item (e.g: AWS EC2 Cloud Configuration, Miscellaneous Configuration Container o Kubernetes Cloud Configuration) in the Operation Center which configuration is not filled (empty fields). As a result, the Operations Center fails trying to push the updates towards the controllers.
Workaround
The following steps aim to identify which Shared Configuration item is empty in order to amend or delete it.
1.- At the UI level, run the following script in the Operation Center Manage Jenkins -> Script Console in the Operations Center to list all the Shared Configuration items and their full paths
Jenkins.instance.getAllItems(com.cloudbees.opscenter.server.config.SharedConfiguration).each { println it.getName() } return null
Example output
ec2-cloud-configuration-shared kubernetes-shared-cloud new_cloud email Result: [com.cloudbees.opscenter.clouds.ec2.EC2Configuration@141d831[ec2-cloud-configuration-shared], com.cloudbees.opscenter.clouds.kubernetes.KubernetesConfiguration@41339d90[kubernetes-shared-cloud], com.cloudbees.opscenter.clouds.kubernetes.KubernetesConfiguration@3467e53b[new_cloud], com.cloudbees.opscenter.server.config.SharedConfiguration@1a78fa7a[email]]
2.- At File System level, navigate to the Operation Center jobs folders and run the following
grep -R -s -e "</com.cloudbees.opscenter.server.config.SharedConfiguration>" -e "</com.cloudbees.opscenter.clouds.ec2.EC2Configuration>" -e "</com.cloudbees.opscenter.clouds.kubernetes.KubernetesConfiguration>" . --exclude-dir=build
Example output
./jobs/kubernetes-shared-cloud/config.xml:</com.cloudbees.opscenter.clouds.kubernetes.KubernetesConfiguration> ./jobs/new_cloud/config.xml:</com.cloudbees.opscenter.clouds.kubernetes.KubernetesConfiguration> ./jobs/email/config.xml:</com.cloudbees.opscenter.server.config.SharedConfiguration> ./jobs/ec2-cloud-configuration-shared/config.xml:</com.cloudbees.opscenter.clouds.ec2.EC2Configuration>
grep -R -s -h -e "</com.cloudbees.opscenter.server.config.SharedConfiguration>" -e "</com.cloudbees.opscenter.clouds.ec2.EC2Configuration>" . -e "</com.cloudbees.opscenter.clouds.kubernetes.KubernetesConfiguration>" --exclude-dir=build -C 200`
Example output
<?xml version='1.1' encoding='UTF-8'?> <com.cloudbees.opscenter.clouds.kubernetes.KubernetesConfiguration plugin="operations-center-kubernetes-cloud@2.263.0.1"> <actions/> <description></description> <snippets> <com.cloudbees.opscenter.clouds.kubernetes.KubernetesCloudConfigurationSnippet> <value> <string><org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud plugin="kubernetes@1.28.3"> <name>kubernetes-oc</name> <defaultsProviderTemplate></defaultsProviderTemplate> ... </com.cloudbees.opscenter.clouds.kubernetes.KubernetesConfiguration> -- <?xml version='1.1' encoding='UTF-8'?> <com.cloudbees.opscenter.clouds.kubernetes.KubernetesConfiguration plugin="operations-center-kubernetes-cloud@2.263.0.1"> <actions/> <description></description> <snippets> <com.cloudbees.opscenter.clouds.kubernetes.KubernetesCloudConfigurationSnippet> <value> <string><org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud plugin="kubernetes@1.28.3"> <name>kubernetes</name> <defaultsProviderTemplate></defaultsProviderTemplate> ... </com.cloudbees.opscenter.clouds.kubernetes.KubernetesConfiguration> -- <?xml version='1.1' encoding='UTF-8'?> <com.cloudbees.opscenter.server.config.SharedConfiguration plugin="operations-center-server@2.235.0.2"> <snippets/> <properties/> </com.cloudbees.opscenter.server.config.SharedConfiguration> -- <?xml version='1.1' encoding='UTF-8'?> <com.cloudbees.opscenter.clouds.ec2.EC2Configuration plugin="operations-center-ec2-cloud@2.235.0.2"> <actions/> <description></description> <snippets> <com.cloudbees.opscenter.clouds.ec2.EC2CloudConfigurationSnippet> <value> <string><hudson.plugins.ec2.EC2Cloud plugin="ec2@1.50.3"> <name>ec2-ec2-cloud-configuration-shared</name> ... </com.cloudbees.opscenter.clouds.ec2.EC2Configuration>
In the provided example, we observe that email
was an empty Miscellaneous Configuration Container after removing it the issue was solved.
Once the Shared Configuration items have been fixed, restart the Operation Center. If the problematic Shared Cloud in the Operations Center still not be recognized by the connected controllers, take a BackUp of the Operation Center and then delete all the Shared Configuration items apart from the Shared Cloud you are troubleshooting. In case it fixes the NullPointerException
, start adding one by one the required Shared Configuration items to understand which is the problematic one.