eMudhra intermediate CA compromise workarounds for CloudBees CI

Issue

The private keys of eight eMudhra intermediate CA certificates were compromised:

Both the operating system (OS) trust store (/etc/pki/ca-trust) and the Java trust store (cacerts) bundled in container images for the CloudBees CI controller, agent, and operations center images include the compromised root, so TLS validation performed by the OS (for example, by git or curl) and TLS validation performed by the JVM are both affected.

Environment

CloudBees CI on traditional platforms - client controller ≤ 2.580.1.38223
CloudBees CI on traditional platforms - operations center ≤ 2.580.1.38223
CloudBees CI on modern cloud platforms - managed controller ≤ 2.580.1.38223
CloudBees CI on modern cloud platforms - operations center ≤ 2.580.1.38223

Resolution

When a release of CloudBees CI that addresses the eMudhra intermediate CA compromise becomes available, this article will be updated to list the version.

Update your OS and JVM to ensure that the compromised root certificates are removed from the trust stores.

On RHEL and UBI, update the ca-certificates package to 2026.2.92_v9.0.319-90.3.el9_8 (RHEL 9) or 2026.2.92_v9.0.319-81.3.el8_10 (RHEL 8) or later. The RHEL JDK reads the OS trust store, so this also fixes the JVM. JDKs that bundle their own cacerts, such as Eclipse Temurin, need the root removed separately.

Patched OS versions are listed in https://emudhra.we-have-a-problem.com/#patched-os . Patched JVM versions are listed in https://emudhra.we-have-a-problem.com/#patched-runtimes .

Workaround

Each workaround removes trust in the compromised root from both the OS-level trust store and the JVM trust store.

CloudBees CI on traditional platforms running on Linux

Contact your OS vendor if you don’t see a fix available, or follow the workaround guidance at: https://emudhra.we-have-a-problem.com/#linux

CloudBees CI on traditional platforms running in containers

Create a Dockerfile that creates a patched version of the affected operations center image:

FROM cloudbees/cloudbees-core-oc:YOUR_CURRENT_TAG

USER root
RUN <<EOF
d=/etc/pki/ca-trust/source/blocklist; [ -d $d ] || d=/etc/pki/ca-trust/source/blacklist
trust extract --overwrite --format=pem-bundle \
  --filter='pkcs11:object=Comodo%20AAA%20Services%20root;type=cert' \
  $d/Comodo_AAA_Services_root.pem &&
trust extract --overwrite --format=pem-bundle \
  --filter='pkcs11:object=emSign%20ECC%20Root%20CA%20-%20G3;type=cert' \
  $d/emSign_ECC_Root_CA_-_G3.pem &&
update-ca-trust
EOF
USER jenkins▼

Replace YOUR_CURRENT_TAG with the tag of the image currently deployed.

Build the patched image:

docker image build -t cloudbees/cloudbees-core-oc:YOUR_CURRENT_TAG-patched .▼

Create a Dockerfile that creates a patched version of the affected controller image:

FROM cloudbees/cloudbees-core-cm:YOUR_CURRENT_TAG

USER root
RUN <<EOF
d=/etc/pki/ca-trust/source/blocklist; [ -d $d ] || d=/etc/pki/ca-trust/source/blacklist
trust extract --overwrite --format=pem-bundle \
  --filter='pkcs11:object=Comodo%20AAA%20Services%20root;type=cert' \
  $d/Comodo_AAA_Services_root.pem &&
trust extract --overwrite --format=pem-bundle \
  --filter='pkcs11:object=emSign%20ECC%20Root%20CA%20-%20G3;type=cert' \
  $d/emSign_ECC_Root_CA_-_G3.pem &&
update-ca-trust
EOF
USER jenkins▼

Replace YOUR_CURRENT_TAG with the tag of the image currently deployed.

Build the patched image:

docker image build -t cloudbees/cloudbees-core-cm:YOUR_CURRENT_TAG-patched .▼

Redeploy your controller and operations center containers using the YOUR_CURRENT_TAG-patched image.

CloudBees CI on modern cloud platforms

Create a Dockerfile that creates a patched version of the affected operations center image:

FROM cloudbees/cloudbees-cloud-core-oc:YOUR_CURRENT_TAG

USER root
RUN <<EOF
d=/etc/pki/ca-trust/source/blocklist; [ -d $d ] || d=/etc/pki/ca-trust/source/blacklist
trust extract --overwrite --format=pem-bundle \
  --filter='pkcs11:object=Comodo%20AAA%20Services%20root;type=cert' \
  $d/Comodo_AAA_Services_root.pem &&
trust extract --overwrite --format=pem-bundle \
  --filter='pkcs11:object=emSign%20ECC%20Root%20CA%20-%20G3;type=cert' \
  $d/emSign_ECC_Root_CA_-_G3.pem &&
update-ca-trust
EOF
USER jenkins▼

Replace YOUR_CURRENT_TAG with the tag of the image currently deployed.

Build the patched image and extract the corrected trust files from it, without deploying it:

docker image build -t cloudbees/cloudbees-cloud-core-oc:YOUR_CURRENT_TAG-patched .
CONTAINER_ID=$(docker create cloudbees/cloudbees-cloud-core-oc:YOUR_CURRENT_TAG-patched)
docker cp ${CONTAINER_ID}:/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ./ca-certificates.crt
docker cp ${CONTAINER_ID}:/etc/pki/ca-trust/extracted/java/cacerts ./cacerts
docker rm ${CONTAINER_ID}▼

Replace YOUR_CURRENT_TAG with the tag of the image currently deployed.

Install and configure sidecar injector if it is not already deployed, following CloudBees CI on modern cloud platforms - Using self-signed certificates in CloudBees CI on Kubernetes using the following helm values:

injectionCaCertificates:
  - /etc/ssl/certs/ca-certificates.crt
  - /etc/pki/tls/certs/ca-bundle.crt
  - /etc/ssl/ca-bundle.pem
  - /etc/pki/tls/cacert.pem
  - /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
  - /etc/pki/ca-trust/extracted/openssl/ca-bundle.trust.crt▼

Label the namespace that runs operations center, controllers, and agents with sidecar-injector=enabled.
Create or replace the ca-bundles config map in that namespace using the extracted files:
```
kubectl delete configmap ca-bundles -n mynamespace --ignore-not-found
kubectl create configmap --from-file=ca-certificates.crt,cacerts ca-bundles -n mynamespace▼
```
If agents run in a separate namespace, label it and create the config map there too.
The hibernation monitor, pluggable storage, CasC bundle service, and SSO relay services embed their Java trust store at build time and ignore the injected file. Point them at it in your Helm values. Hibernation.JavaOpts is a list, the others are strings:
```
Hibernation:
  JavaOpts:
    - -Djavax.net.ssl.trustStore=/etc/pki/java/cacerts
PluggableStorageService:
  javaOpts: -Djavax.net.ssl.trustStore=/etc/pki/java/cacerts
CascBundleService:
  javaOpts: -Djavax.net.ssl.trustStore=/etc/pki/java/cacerts
SsoRelay:
  controller:
    javaOpts: -Djavax.net.ssl.trustStore=/etc/pki/java/cacerts▼
```
If you already set Java options for these services, append the flag rather than replacing them.
Restart the operations center pod and any running managed controller pods so the updated bundle is injected:
```
kubectl delete pod cjoc-0 --namespace mynamespace▼
```
Restart managed controllers from operations center, and restart the services above with kubectl rollout restart deployment.

Windows agents

The sidecar injector does not apply to Windows agent images, which use their own Java trust store. Build a derived Windows agent image that removes the entry:

keytool -delete -cacerts -storepass changeit -alias "cn_emsign_ecc_root_ca__g3,o_emudhra_technologies_limited,ou_emsign_pki,c_in [jdk]"▼

Verify the workaround

First obtain a verification endpoint from your security team, connections to it should now fail with a certificate trust error. For example, there are verification endpoints listed here which we will use:

https://emudhra.we-have-a-problem.com/

This verification must run against both the OS trust store (/etc/pki/ca-trust) and the Java trust store (cacerts), for both the controller and agents:

Verifying the workaround in the OS trust store (/etc/pki/ca-trust):

curl https://dv-g3b-emudhra.we-have-a-problem.com/
curl https://dv-g3b-aaa-emudhra.we-have-a-problem.com/
curl https://dv-g3b-noct-emudhra.we-have-a-problem.com/
curl https://dv-g3b-aaa-noct-emudhra.we-have-a-problem.com/▼

A successful verification of the workaround in the OS trust store will result in curl failing for dv-g3b-emudhra and dv-g3b-noct-emudhra with a certificate trust error SSL certificate problem: unable to get local issuer certificate, as the aaa endpoints fail on RHEL and UBI even without the workaround.

Verifying the workaround in the Java trust store (cacerts):

keytool -list -v -cacerts -storepass changeit | grep -c "^Owner: CN=emSign ECC Root CA - G3,"▼

A result of 0 confirms the root is removed.

To check from the controller, run this in the script console:

try {
  new URL('https://dv-g3b-emudhra.we-have-a-problem.com/').openConnection().connect()
  println 'TRUSTED: the compromised root is still trusted'
} catch (javax.net.ssl.SSLHandshakeException e) {
  println 'REJECTED: the workaround is in effect'
}▼

This article is part of our Knowledge Base and is provided for guidance-based purposes only. The solutions or workarounds described here are not officially supported by CloudBees and may not be applicable in all environments. Use at your own discretion, and test changes in a safe environment before applying them to production systems.