CloudBees CI on modern cloud platforms

Rolling release: 2019-12-19

Based on Jenkins LTS 2.204.1-cb-2

New features

Introducing Configuration as Code (CasC) for Masters v0.3

Configuration as Code (CasC) for Masters simplifies the management of a CloudBees Core cluster by capturing the configuration of CloudBees Core masters in human-readable declarative configuration files which can then be applied to a master in a reproducible way. By capturing the configuration in files, it can be treated as a first class revision-controlled artifact - versioned, tested, validated, and then applied to masters while being centrally managed from CloudBees Core Operations Center.

Configuration as Code (CasC) for Masters is built using the underlying Configuration as Code implementation and extends its functionality for Core customers. The main differences in v0.3 (Parity preview) is the ability to install and upgrade plugins.

Generate Configuration as Code (CasC) for Masters YAML from pre-existing masters (CTR-676)

Users can now generate a Configuration as Code (CasC) for Masters Bundle from an existing master. The generated bundle is not an exact replica of the existing master and needs to be completed manually, but it is a good starting point.

Configuration bundle hot reload (CTR-728)

The Configuration as Code plugin allows users to reload a configuration file without restarting the instance. Since the CloudBees Core manages plugin updates and the plugins catalog, validating a hot reload is a more complex process. Configuration bundles that do not include plugin update requirements can be reloaded without an instance restart. This feature allows users to update Jenkins configurations, install new plugin catalogs, or install new plugins dynamically.

Permit the Helm chart to use alternate default agents (CPLT2-6029)

The Helm chart was using a hard-coded value for the agent image as a default for all pods. The Helm chart has been updated so that it supports alternate default agent images.

Add UBI options to Docker Hub (CPLT2-6127)

A new Docker Hub repository has been added to support the UBI variants of the Core platforms.

Add options to the Helm chart to allow the selection of UBI or Alpine based images for Operations Center and Managed Masters CPLT2-6025)

The Helm chart now defaults to the UBI image. However, if needed for migration, the Helm chart permits the selection of an Alpine-based image by choosing the corresponding value in XXX.Image.dockerImage.

Provide a way to specify new containers in the Operations Center statefulset (CPLT2-5955)

There was a user need for the ability to add a sidecar container to the Operations Center pod (for example, for monitoring purposes). This facility has been added.

Clarify OpenShift support (CPLT2-5694)

The support policies have been clarified to state that Red Hat OpenShift Container Platform (OCP) 3.x is supported starting with version 3.11 and that, starting with CloudBees Core on modern cloud platforms version, OpenShift Container Platform 4.x is supported, starting with version 4.1.

Feature enhancements

Team support for multiple Kubernetes endpoints (CTR-564)

CloudBees Core now allows users to select a cluster endpoint when creating a Team Master. This feature is only available when running CloudBees Core on Kubernetes. Cluster endpoints need to be created from the Manage Jenkins page in the classic UI.

Resolved issues

Kubernetes agents are removed after Jenkins master restart (CPLT2-6142)

Nodes were removed after restarting a Jenkins master with a running pipeline on a node. Idle Kubernetes agents are now retained for at least 5 minutes to permit the pipeline to resume properly.

Pods were stuck in the Pending state on Core Modern (CPLT2-6048)

Some pods were remaining "stuck" in the Pending state and never progressing because they featured events that the kube-agent-management plugin couldn’t interpret or manage. The NoDelayProvisioningStrategy has been corrected so that it triggers events that are expected by the kube-agent-management plugin.

Use the appropriate Remoting version from the Managed Master WAR file (CPLT2-6101)

It was possible for Kubernetes agents to download a potentially old copy of the Jenkins remoting agent. Kubernetes agents now reference the current Managed Master WAR file to ensure that they are using the appropriate version of Remoting for the Jenkins master to which they are connecting.

Set default encoding to UTF-8 (CPLT2-6140)

UBI-based agents were using ASCII as a default file encoding, which could cause problems for some Jenkins operations, like rendering a freestyle build log or running shell steps on agents. The default file encoding is now explicitly set to UTF-8 for Operations Center, Managed Masters, and Agent images.

Git CLI executable missing from UBI images (CPLT2-6141)

UBI-based Operations Center and Managed Master images were missing the Git CLI executable, which caused problems for certain Jenkins operations like Git-based multibranch projects. The images have been updated to include the Git CLI package.

H2 database was not stopped correctly when Jenkins was terminating (CTR-955)

Operations Center was not properly closing the connection to one of the internal databases, potentially causing database corruption. With this fix, the database connection pool is closed when Operations Center is shutdown.

DownloadSettings has been removed from Jenkins core (CTR-740)

When using Operations Center, the "Use browser for metadata download" setting can be pushed to connected masters, but this setting is no longer supported by Jenkins. With this fix, support for pushing the "Use browser for metadata download" setting to connected masters has been removed.

Run user-activity generate report on all Jenkins masters in CJE cluster (CTR-445)

Master names containing spaces are not generating data because the IFS environment variable is not properly located in scripts. This variable would make a space the default value for a line separator. Users should set the proper value for the IFS environment variable to correct the issue.

If an agent spawned by a shared JNLP agent died, it was not relaunched (CTR-747)

When a JNLP agent leased from Operations Center to a master died (for a variety of reasons), the oc-jnlp controller did not attempt to relaunch the connection. With this fix, if a JNLP agent leased from Operations Center to a master dies, a new agent process is launched in order to resume the pipeline.

Agent process never dies if OC is down until the build finished (CTR-789)

Shared agent processes were not properly stopped after the agent was returned to Operations Center. This issue only occurred when Operations Center was restarted while a shared agent was leased to a master. With this fix, leased agent processes are stopped properly when Operations Center starts.

OpenShift and CJE integration issue (CTR-571)

The URL used to download the OpenShift client .tar.gz binary was incorrect for all versions. With this fix, the URL used to download the OpenShift client .tar.gz binary is correct for all versions.

File leak in AbstractEnterpriseRegistrar.getProductId() (CTR-815)

The platform detection contained a file leak, and if it were called a lot, would lead to exhaustion of file handles. With this fix, the file handle is now correctly closed, fixing the leak.

Fix CVE-2019-12402 in commons-compress in infradna-backup (CTR-662)

The commons-compress dependency has been removed from the CloudBees Backup plugin as it’s already a dependency of Jenkins Core.

Cleanup deprecated code and remove old workarounds in nectar-license (CTR-817)

The CloudBees Jenkins Enterprise License Entitlement Check plugin contained several outdated workarounds for defects that have now been corrected. These workarounds have been removed, and the code has been tidied up.

JENKINS-58878 withEnv, withCredentials hang (NGPIPELINE-821)

Block-scope Pipeline steps implemented using GeneralizedNonBlockingStepExecution (such as withCredentials and wrap) could hang indefinitely in some scenarios. After applying this fix, Block-scope Pipeline steps implemented using GeneralizedNonBlockingStepExecution no longer hang indefinitely.

False positives in CPS mismatch warnings from NGPIPELINE-27 (NGPIPELINE-617)

Incorrect CPS method mismatch warnings were being reported for some acceptable code paths in Jenkins pipelines and shared libraries. After applying this fix, those warnings no longer appear.

Possible thread leaks in the metrics thread 'QueueSubTaskMetrics' (FNDJEN-1659)

Certain build failures were showing a large number of 'QueueSubTaskMetrics' threads that persisted without resolution, preventing builds from ending. The Jenkins core instance has been strengthened to ensure that builds always end.

Unable to validate or push a Plugin Catalog (FNDJEN-1787)

When using a Plugin Catalog with settings similar to those recommended in Configuring plugin catalogs - Defining a plugin catalog, an ID error was received. After adjusting credentials the fix the problem, a new error was generated. Both of these errors made it impossible to validate or push a Plugin Catalog. The underlying problem was investigated, and a binary incompatibility that was causing the issue was resolved.

Known issues

Introducing Red Hat UBI based images for Operations Center, Managed Masters, and Linux Agents.

This is the default starting with this release, CloudBees Core on modern cloud platforms Users wishing to continue with Alpine Linux based images need to use the Helm chart options to do so.

Under certain circumstances, Jenkins may “hang” with the following conditions.
  • The Jenkins java process is running in a waiting state.

  • Jenkins is effectively down.

Nothing is logged.

Sometimes, after numerous restarts, the Jenkins service may start up again normally.

The root cause for this issue is that the Jenkins service hangs immediately before it forks the child process that starts Jetty and Jenkins. Although the Java process is running, nothing is logged, because Jenkins has not yet started and is not yet listening on any port.

This issue affects a very small number of CloudBees customers. You only need to take action if you are directly affected by this issue: if you are not experiencing this issue, no action is necessary.

A workaround is available in the CloudBees Support Knowledge Base article Jenkins intermittently fails to restart on RHEL 7 and CentOS 7.

Version cannot start on Microsoft Windows

This release contains some filenames with ( or ,…​ which prevents the WAR_ file from being uncompressed. To workaround this issue, remove the licenses folder from the folder created by the extraction of the WAR file. The extraction of the WAR file is automatically performed inside the $JENKINS_HOME under a directory called war. Removing the content of $JENKINS_HOME/war/license should be enough to workaround the issue.

The extraction of the WAR file can happen in a different directory in case you are using the property --webroot to change the default extraction directory.

Upgrade notes



Revision 3 (2020-01-15)

CloudBees Security Advisory 2020-01-15

Revision 2 (2020-01-08)

Plugin updates