CloudBees CI controller login issues with Java 17 and proxy setup

Issue

If the following conditions are met:

The controller is running CloudBees CI version 2.440.1.3.
The controller is running Java 17.
The controller has a Proxy configuration (even if the operations center hostname has been included in the non proxy list) and a username is configured on the proxy.
Single sign-on via operations center is selected as Security Realm in the controller.

When logging into the controller you see an angry Jenkins page and the following WARNING message in the controller log:

WARNING	o.e.j.s.h.ContextHandler$Context#log: Error while serving $CONTROLLER_URL/securityRealm/finishLogin
java.io.IOException: WWW-Authenticate header missing for response code 401▼

Environment

CloudBees CI on modern cloud platforms - managed controller = 2.440.1.3
CloudBees CI on traditional platforms - client controller = 2.440.1.3
Java 17
Proxy configured at the Controller level

Resolution

Upgrade to CloudBees CI 2.440.1.4.

When running a connected controller on Java 17 with a proxy configured with username/password authentication, the Operations center single sign-on did not work due to a limitation in a Java HTTP client. Now the proxy configuration is ignored and the controller contracts the Operations center directly regardless of proxy settings.

Saving the proxy configuration in the GUI currently saves a blank username. This will be fixed in a future release. Configuring the proxy via Configuration as Code does not enable authentication unless a username is actually defined.

Workaround

If you are currently unable to upgrade, you can apply one of the following workarounds:

CloudBees CI on Traditional Platforms

Switching the controller back to Java 11.
Disabling the proxy configuration at the controller level in case the proxy settings are not required.
Removing authentication from the proxy in case that is not required. Note that saving proxy configuration from the GUI currently enables authentication even if the username is blank; you can remove <username></username> from $JENKINS_HOME/proxy.xml, or use configuration-as-code.
Disabling operations center SSO if possible, copying the security realm configuration from the operations center to the controller.

CloudBees CI on Modern Platforms

Instead of running this version, you can use product versions running on JDK 11, which are CloudBees CI on modern cloud platforms 2.426.3.3 and earlier. Note that if you want to move to an earlier version of CloudBees CI, you first need to restore from your backup taken before the upgrade, and then upgrade again to your new desired release version, downgrades are not supported.

Alternatively, if you want to use this version or any other product version showing this limitation but compatible both with JDK 17 and JDK 11, you can build custom container images with Java version 11.0.22.7 ( or a later JDK 11 version).

Here is an example of how to build a custom controller image for version using Temurin (https://adoptium.net/) 11.0.22-7.2:

For x64 architecture:

#!/bin/sh
wget https://github.com/adoptium/temurin11-binaries/releases/download/jdk-11.0.22%2B7/OpenJDK11U-jdk_x64_linux_hotspot_11.0.22_7.tar.gz
tar -xf OpenJDK11U-jdk_x64_linux_hotspot_11.0.22_7.tar.gz
docker build -t cloudbees-core-mm:2.440.1.3-jdk11.0.22.7 .▼

For arm architecture:

#!/bin/sh
wget https://github.com/adoptium/temurin11-binaries/releases/download/jdk-11.0.22%2B7/OpenJDK11U-jdk_arm_linux_hotspot_11.0.22_7.tar.gz
tar -xf OpenJDK11U-jdk_arm_linux_hotspot_11.0.22_7.tar.gz
docker build -t cloudbees-core-mm:2.440.1.3-jdk11.0.22.7 .▼

FROM cloudbees/cloudbees-core-mm:2.440.1.3
USER root
RUN /bin/sh -c dnf erase java-17-openjdk.x86_64 java-17-openjdk-devel.x86_64 java-17-openjdk-headless.x86_64 && rm -rf /usr/lib/jvm/java-17-openjdk-17.0.10.0.7-2.el8.x86_64
COPY jdk-11.0.22+7 /usr/lib/jvm/jdk-11.0.22+7
RUN alternatives --install /usr/bin/java java /usr/lib/jvm/jdk-11.0.22+7/bin/java 99
RUN alternatives --install /usr/bin/jstack jstack /usr/lib/jvm/jdk-11.0.22+7/bin/jstack 99
RUN alternatives --install /usr/bin/jps jps /usr/lib/jvm/jdk-11.0.22+7/bin/jps 99
RUN alternatives --install /usr/bin/jcmd jcmd /usr/lib/jvm/jdk-11.0.22+7/bin/jcmd 99
USER 1000▼

You’ll have to adjust the image tag when running docker build to one that you can push to your private container registry.

Also, note that this example is for the managed controller, for your operations center you would change the Dockerfile line from:

FROM cloudbees/cloudbees-core-mm:2.440.1.3

FROM cloudbees/cloudbees-cloud-core-oc:2.440.1.3

As well as change the docker tag when running docker build.

Once you push these images to your container registry, to use them:

For your Operations center, override the helm values OperationsCenter.Image.dockerImage to be the container registry location for your custom image, and OperationsCenter.ImagePullSecrets to be the name of the Kubernetes secret to use to pull images from your container registry.
For your managed controllers, follow Using private image repository in Operations Center controller provisioning.

Troubleshooting

The certificates configured in my CloudBees sidecar injector are not used by the controller after updating the image

If a sidecar injector is deployed in the cluster you’ll need to configure the new JDK 11 installation to use the cacerts file injected. The simplest approach would be to create a softlink that overwrites the default cacerts file of the new JDK 11 and targets /etc/pki/java/cacerts. See the following example:

FROM cloudbees/cloudbees-core-mm:2.440.1.3
USER root
RUN /bin/sh -c dnf erase java-17-openjdk.x86_64 java-17-openjdk-devel.x86_64 java-17-openjdk-headless.x86_64 && rm -rf /usr/lib/jvm/java-17-openjdk-17.0.10.0.7-2.el8.x86_64
COPY jdk-11.0.22+7 /usr/lib/jvm/jdk-11.0.22+7
RUN alternatives --install /usr/bin/java java /usr/lib/jvm/jdk-11.0.22+7/bin/java 99
RUN alternatives --install /usr/bin/jstack jstack /usr/lib/jvm/jdk-11.0.22+7/bin/jstack 99
RUN alternatives --install /usr/bin/jps jps /usr/lib/jvm/jdk-11.0.22+7/bin/jps 99
RUN alternatives --install /usr/bin/jcmd jcmd /usr/lib/jvm/jdk-11.0.22+7/bin/jcmd 99
RUN rm /usr/lib/jvm/jdk-11.0.22+7/lib/security/cacerts
RUN ln -s /etc/pki/java/cacerts /usr/lib/jvm/jdk-11.0.22+7/lib/security/cacerts
USER 1000▼

Alternatively, you could customize the system properties javax.net.ssl.trustStore and javax.net.ssl.trustStorePassword of the instance to point to this mount. This can be achieved by passing them as Java startup arguments to the process like in the following example:

-Djavax.net.ssl.trustStore=/etc/pki/java/cacerts -Djavax.net.ssl.trustStorePassword=changeit

Please note that this last approach is considered less secure.