Why is my instance not getting the right information from CloudBees Update Center starting on October 19th, 2021?

Article ID:4408359898011
4 minute readKnowledge base

Environment

Issue

We have observed that starting on October 19th 2021 at 18:31:36 GMT, our instance is showing problems related to CloudBees Update Center under Manage Jenkins -> Manage Plugins, and provisioning new instances will fail during the plugin installation step.

This error has no impact on using the instance to create jobs or running builds. All builds will run as usual. However, this could impact you when installing or updating plugins, or creating new controllers.

The workaround documented below does not require an upgrade or restart.

The groovy script from the workaround will only be effective if run after the certificate has expired.

Existing Instances

Online Instances

In these instances, we see an error in the plugin manager like the one shown below:

There were errors checking the update sites: Signature verification failed in the update center &#039;cje-offline&#039; <a href='#' class='showDetails'>(show details)</a><pre style='display:none'>java.io.IOException: Empty input<br> at java.base/sun.security.provider.X509Factory.engineGenerateCertificate(X509Factory.java:111)<br>Caused: java.security.cert.CertificateException
4408359898011 plugin manager

Nevertheless, the plugins are available for installation and update through the online Update Center, though you will usually have to make your web browser window larger, or scale text smaller to be able to see the plugins page.

Offline Instances

In these instances, we see an error in the plugin manager that prevents us from installing any of the CloudBees Assurance Program plugins, even if the plugins are already downloaded and should be available for the installation.

There were errors checking the update sites: Signature verification failed in the update center &#039;cje-offline&#039; <a href='#' class='showDetails'>(show details)</a><pre style='display:none'>java.io.IOException: Empty input<br> at java.base/sun.security.provider.X509Factory.engineGenerateCertificate(X509Factory.java:111)<br>Caused: java.security.cert.CertificateException

or

WARNING [Beekeeper.checkUC [#1]] com.cloudbees.jenkins.plugins.assurance.UpdateCenterRefresher$CheckUC.call Unable to update site core-cm-offline: ERROR: Invalid Root CA in the license key <a href='#' class='showDetails'>(show details)</a><pre style='display:none'>java.security.cert.CertificateExpiredException: NotAfter: Tue Oct 19 14:31:36 EDT 2021<br>    at sun.security.x509.CertificateValidity.valid(CertificateValidity.java:277)<br>

or

hudson.model.UpdateCenter.updateDefaultSite Upgrading Jenkins. Failed to update the default Update Site 'core-cm-offline'. Plugin upgrades may fail.
	ERROR: Invalid Root CA in the license key <a href='#' class='showDetails'>(show details)</a><pre style='display:none'>java.security.cert.CertificateExpiredException: NotAfter: Tue Oct 19 14:31:36 EDT 2021<br>

The web UI will look the same as the above screenshot for online instances, or you may see Default update site connectivity check failed with fatal error: Cannot check connection status of the update site with ID='core-cm-offline'.

New Instances

  • Newly created instances will not be able to install any of the plugins selected in the installation wizard.

  • New instances created using automation will not be able to download any plugins and this will potentially cause an error during the creation process.

new instance installing suggested plugins

Explanation

The JSON contents of the Update Center served to instances are computed several times a day. The contents include CAP definitions plus all compatible plugins found in the Jenkins Update Center according to the existing allowlist rules. In other words, the Update Center data is subject to change, and as such, the Update Center does not store any signature data. The signature is added instantly when serving the Update Center to client instances.

As a signature is added instantly to all Update Center instances, no matter how old its version may be, once CloudBees updates the certificate used by the Update Center application it will be used to sign all served Update Center instances in the next computation (this happens several times a day).

Our instances contain an offline version of the above-mentioned JSON file that allows air-gapped environments to be able to install CAP plugins. The offline version of the JSON file is distributed among the files included in a release. This file is static for a given release, thus a change in our Update Center certificates will not be propagated to these files. In the event that the certificate used to sign the offline Update Center expires, the file will no longer be valid. And you will not be able to use the offline update center to install CAP plugins.

Manual plugin installation will continue working, so you can download a proprietary plugin and its dependencies and manually install it as described in the Advanced plugin installation section.

Does this mean that the instance will stop working? Absolutely not, the instance will continue working but you will observe the problems mentioned in the section above.

Workaround

Existing Instances

Online and Offline instances

In order to perform the actions needed to work around this issue, you will need to run the following script in the impacted instance. To do this, navigate to Manage Jenkins -> Script Console and run the script there. You can also run this script on all online controllers connected to an operations center using a cluster operation using the cluster operation step Execute Groovy Script on Controller.

The script will perform the necessary changes to the instance so it can preserve the functionality related to the Update Center and will also remove these actions once you are running a version that is not affected by the issue.

  • It will remove the offline Update Center for online instances.

  • It will disable certificate validation for offline instances only.

  • It will install the removal script as part of the startup scripts so that it checks whether the mitigation is still needed or not.

This groovy script will only be effective if run after the certificate has expired.

If you have an instance where you cannot access the web UI, stop the instance, and place the Remediation Script in a new file at JENKINS_HOME/init.groovy.d/ucCertRemediation.groovy.

New Instances

For new instances, when prompted for which plugins you want to install, you will need to choose Select plugins to install, then you should deselect all plugins using the None button at the top of the screen:

4408359898011 install none

Then continue with the setup process by clicking "Install" at the bottom right. Once the instance is ready, go to Manage Jenkins -> Script Console and run the Remediation Script so that the instance works under the expected parameters.

Solution

We have released new versions of CloudBees CI and CloudBees Jenkins Platform that resolve the issue and remove the error:

  • After 2.303.2.6 and before 2.346.3.4 on a rolling release

  • 2.277.42.0.3 or newer on the 2.277.x fixed release

  • 2.249.33.0.2 on the 2.249.x fixed release

Refer to the release notes for more information:

We would like to help prepare a customized upgrade plan, and guide you through upgrade testing via an Assisted Upgrade.

References