RELEASED: Public: October 30, 2024
New features
- Using per-replica temporary directory for checkouts of Pipeline libraries and scripts in High Availability (HA) controllers (BEE-51927)
-
Starting with the 2.414.2.2 (September 2023) release where High Availability (HA) was introduced, all managed controllers (High Availability (HA) or otherwise) in a Modern installation were started with the
jenkins.plugins.git.AbstractGitSCMSource.cacheRootDirandorg.jenkinsci.plugins.github_branch_source.GitHubSCMSource.cacheRootDirsystem properties set to a temporary directory under/tmp/jenkins; and Traditional installations were also required to set these properties to a per-replica temporary directory. This prevented clashes between replicas trying to use the same cache directory at the same time. Since library checkouts defaulted to a location in$JENKINS_HOME, for the same reason High Availability (HA) controllers were required to use “clone” mode for libraries (enforced with an administrative monitor); the library caching system was not available and non-multibranch Pipeline projects loading the script from SCM with “lightweight mode” disabled were not safe to use in a High Availability (HA) controller.
Now, a series of cache-related system properties is automatically set on all High Availability (HA) controllers, Modern or Traditional, to a per-replica temporary location. (In Traditional, the environment variable $CACHE_DIR can be used to customize the root directory; otherwise, it defaults to the system temporary directory.) In addition to the GitHub and (generic) Git caches, the list includes checkouts of Pipeline Groovy libraries (with or without caching mode enabled), when the library is not configured to use clone mode and heavyweight checkouts of repositories were used for script from SCM. In Modern, since the caches are unique for each pod, builds run on a recently started replica can be slower, but the speed should be improved once the replica has been running and its caches on local disk are warm.
It is no longer required to use clone mode in a Pipeline library in a High Availability (HA) controller. The administrative monitor has been removed.
Non-High Availability (HA) managed controllers (Modern) no longer override GitHub or (generic) Git caches, so these will, by default, revert to keeping caches inside $JENKINS_HOME. You may explicitly select a temporary directory if you prefer.
Due to the expanded usage of the temporary volume, if your High Availability (HA) controller uses many distinct libraries (or other repositories that must be checked out on the controller, due to loading script from SCM and disabling lightweight checkout), or some of these repositories have large histories or working copies, you can run low on temporary directory space. In a managed controller, exhausting the /tmp volume soon causes the pod to be restarted with a fresh volume. Setting a higher resources.limits.ephemeral-storage may be appropriate, but in some environments it can result in the pod being unschedulable; in GKE, as per https://cloud.google.com/kubernetes-engine/docs/how-to/persistent-volumes/local-ssd#example_ephemeral it is possible to request a larger ephemeral volume, over 300Gib rather than the default of around 100Gib. In EKS, the ephemeral volume is around 100Gib. It is possible to configure a generic ephemeral volume mounted on /tmp (or /tmp/jenkins) that can be any size you like, but the performance will not be as good as using a local volume, and if space is exhausted on this volume, the pod will not be restarted automatically, so it would be necessary to set up additional monitoring.