Building on CloudBees Jenkins Enterprise
| CloudBees will no longer be supporting CloudBees Jenkins Enterprise 1.x after July 30, 2020. This end-of-life announcement allows CloudBees to focus on driving new technology and product innovation for CloudBees CI. For information on moving to CloudBees CI, please refer to Migrating from CloudBees Jenkins Enterprise 1.x to CloudBees CI on modern cloud platforms which has been created to help you with the migration process. Existing customers can also contact their CSM to help ensure a smooth transition. |
CloudBees Jenkins Enterprise comes out of the box with a set of agent images in order to run builds on the underlying cluster infrastructure. This is managed by Palace, a scheduler that is launched on the CloudBees Jenkins Enterprise infrastructure.
These agent images are available as items in Operations Center and are pushed to Client Masters located in the same context. You can also create agent images directly on a Client Master.
Palace, scalable agent scheduling framework for Mesos
Why Palace?
CloudBees Jenkins Enterprise used to ship with a shared cloud using the Mesos plugin to provision docker-based agents. While doing scalability tests on CloudBees Jenkins Enterprise, we realized this plugin couldn’t scale as much as we want because each Jenkins master would register one Mesos framework, and Mesos just cannot handle thousands of frameworks.
We therefore built Palace, which is used as an intermediate in order to provision Jenkins agents on Mesos. From Mesos, it is a single framework. From Jenkins masters, it is accessed through a REST API, which removes the need to bundle Mesos binaries within masters.
Comparing architectures
Mesos plugin
When using the Mesos plugin, each Jenkins master hosts a Mesos framework, which requires to have the Mesos package installed in the container running Jenkins. There is a competition between the different masters interacting with Mesos. A faulty master could hog on to Mesos offers and prevent other masters from creating tasks. Latest registered masters are given priority over those which have been there for a long time. When many frameworks are connected, it takes a longer time for a given framework to get the offers it is interested in because they need to be processed and declined by other frameworks.
Palace
Palace is a scheduled application, just like Jenkins masters. In case of failure, it is automatically restarted and it reconciles its state with the currently running Mesos tasks.
Palace works in exclusive mode : since CloudBees Jenkins Enterprise uses workers dedicated to the Palace workload and Palace is a single Mesos framework, it hogs on to offers on these agents and is able to respond instantly whenever a Jenkins agent request is made by one master. Overall it reduces the time to allocate agents.
Having a central place of decision to provision agents also helps to apply more elaborated strategies to do task placement optimizations, e.g.
-
bin-packing (pack tasks into a few hosts as possible)
-
load spread (maintain a similar load on all available hosts) (not yet implemented in CloudBees Jenkins Enterprise)
-
implement resources quotas per master (not yet implemented in CloudBees Jenkins Enterprise)
Jenkins masters connect to Palace using a REST API, which decouple them from Mesos itself : we no longer need to ship Mesos binaries in the docker container, the Mesos native library is no longer called directly from Jenkins which limits impact in case of a crash.
Defining a new agent image
To define a new agent image, just create a new item and select Docker Agent Template.
The name of the item will be the label to use when defining jobs meant to run using this image.
Then you need to configure the agent image to match your requirements.
| Field | Default Value | Description |
|---|---|---|
Use when no label is specified |
Check this if you want your agent image to be used for unassigned jobs. In case several agent templates have this option selected, these jobs will run on any of them, the first available. |
|
Enable this agent template |
Uncheck this to disable a template you don’t want to use (but that you still want to keep around) |
|
CPU shares |
|
Refer to the corresponding Docker option. We recommend you to leave it to its default value unless you really know what you are doing. |
Memory |
|
It is the amount of memory to be allocated for the docker container. If the value is set too low, the container may be killed by the Linux OOM Killer. If it is too high, less tasks will be able to be launched on the infrastructure and you may experience delays when launching jobs. |
JVM Memory |
|
It is the maximum heap size given to the Jenkins agent. You need to account for this value when setting up Memory above. |
JVM Arguments |
|
Allows to specify additional JVM arguments to be applied to the JVM running the Jenkins agent. |
JNLP Arguments |
|
Allows to specify JNLP arguments when launching the Jenkins agent. |
Remote FS Root |
|
It is the working directory of the Jenkins agent inside the container. It must be a path on which the user running the container as must be able to do anything, otherwise the agent will fail to launch. |
Image |
|
This is the docker tag of the image to pull. It needs to have a JDK installed, as this is a prerequisite to launch the Jenkins agent. |
Additional Options |
||
Additional URI |
Fetches a file from an URI and put it in the Mesos sandbox. This is useful to pull an image from a private Docker registry. |
|
Environment Variable |
Add an environment variable to the Docker container. |
|
Force pull image |
Force update of an image if it already exists locally. This option is useful
if the specified image targets a mutable Docker tag (by convention) such as
|
|
Launch in privileged mode |
Launches the Docker container in privileged mode. |
|
Parameter |
Pass a parameter to the Docker container. |
|
Port Mapping |
Allows to map a port from the host to the Docker container. |
|
Use a custom Docker shell command |
Allows to override the default |
|
Volume |
Allows to map a volume inside the Docker container. |
|
Sharing agent images
You can share agent images by creating them on Operations Center. Agent images created in Operations Center will be made available to client masters located in the same folder hierarchy (same folder or a subfolder below it).
Agent images created by the installer are located in the root of Operations Center. Then are then available to any Client Master.
Shared agent images can be browsed from any Client Master within the special folder Operations Center Shared Templates.
Example of restricted agent image
Considering a folder named Organization 1, an agent template named agent1 and a master Master 1.
If we browse the Shared Templates within Master 1, we will see the template agent1 as well as the other templates defined in Operations Center.
However if there are masters in Operations Center root folder or in other folders, they won’t see the template agent1.
Migrating from the Mesos shared cloud to Palace
CloudBees Jenkins Enterprise used to ship with a shared cloud using the Mesos plugin to provision docker-based agents.
Since 1.3.0, CloudBees Jenkins Enterprise ships with a new shared cloud called Palace. It effectively deprecates the Mesos shared cloud by providing a better scalability model as well as a more flexible way to manage docker agent images across your cluster.
We don’t migrate automatically your templates based on the Mesos plugin as we don’t guarantee a 100% compatibility with the Mesos plugin configuration.
Migrating agent configuration
In particular, only Mesos agent configurations using the Docker containerizer are supported by Palace.
Each Mesos agent configuration corresponds to a Docker Agent Template item when using Palace.
| Field in Mesos plugin | Field in Palace |
|---|---|
Label string |
This is the item name |
Jenkins Agent CPUs |
CPU Shares = (Jenkins Agent CPUs) + (Number of executors)*(Jenkins Executor CPUs) |
Jenkins Agent Memory in MB |
JVM Memory |
Minimum number of Executors per Agent |
Not Applicable (always 1) |
Maximum number of Executors per Agent |
Not Applicable (always 1) |
Jenkins Executor CPUs |
See Jenkins Agent CPUs |
Jenkins Executor Memory in MB |
Memory = (Jenkins Agent Memory) + (Number of executors) * (Jenkins Executor Memory) |
Remote FS Root |
Remote FS Root |
Idle Termination Minutes |
Not Applicable. Agents are removed as soon as the build is done. |
Mesos Offer Selection Attributes |
Not Applicable |
Additional Jenkins Agent JVM arguments |
JVM Arguments |
Additional Jenkins Agent Agent JNLP arguments |
JNLP Arguments |
Docker Image |
Image |
Docker Privileged Mode |
Options > Launch in privileged mode |
Docker Force Pull Image |
Options > Force pull image |
Docker Image Can Be Customized |
Not supported |
Custom docker command shell |
Use a custom docker shell command |
Networking options |
Not supported |
Volumes |
Options > Volume |
Parameters |
Options > Parameter |
Additional URIs |
Options > Additional URI |
Migrating workers
Palace uses a different strategy to work with Mesos offers than the Mesos plugin does. This allows a quicker time to provision agents as well as a better control on where tasks are getting provisioned.
For this reason and to avoid disrupting your workload when upgrading, we have decided to require specific workers for Palace.
When listing workers using cje run list-workers, you will notice that your existing build workers are now tagged with mp-build.
It means these workers are assigned to workload generated by the Mesos plugin.
worker-1 (master: m4.xlarge) ec2-aa-bb-cc-d1.compute-1.amazonaws.com > ACTIVE worker-3 (mp-build: m4.xlarge) ec2-aa-bb-cc-d2.compute-1.amazonaws.com > ACTIVE worker-2 (master: m4.xlarge) ec2-aa-bb-cc-d3.compute-1.amazonaws.com > ACTIVE
To add new workers that can be used by Palace, run the worker-add operation and choose workload_type = build.
Once you are done with your migration you will be able to remove the old mp-build workers using the worker-remove operation.
Creating Credential Package
To use a private registry, CloudBees Jenkins Enterprise needs credentials to access the private registry.
Create a docker.tar.gz file including the .docker folder and the contained .docker/config.json by:
-
Login to the private registry manually. Login creates a
.dockerfolder and a.docker/config.jsonin the user’s home directory.
$ docker login some.docker.example.com
Username: foo
Password:
Email: foo@example.com-
Tar this folder and its contents.
$ cd ~
$ tar czf docker.tar.gz .docker-
Check you have both files in the tar
$ tar -tvf ~/docker.tar.gz
drwx------ root/root 0 2015-07-28 02:54 .docker/
-rw------- root/root 114 2015-07-28 01:31 .docker/config.json-
Put the gzipped file in a location that can be retrieved from the worker VMs. The URI must be accessible by all workers. Approaches may include storing it on a shared network drive, or for example Amazon S3. It is worth considering the security implications of your chosen approach.
Use Docker Commands in Builds
There are two Docker-related build tasks that are available as build tasks.
In order to be able to use these Docker-related tasks in a build environment, a Palace build agent definition with the Docker client is required.
The Docker image cloudbees/java-with-docker-client can be used for this purpose. It has both Java (for the Jenkins agent process)
and Docker client pre-installed.
The Docker socket of the enclosing CloudBees Jenkins Enterprise worker also needs to be mounted through a volume inside the container.
To do so, in the agent image configuration, add a Volume entry, with Host Path and Container Path set to /var/run/docker.sock.
| Configuration changes to agent images are only applied to new containers, not to existing ones. |
Be aware that by letting a build access /var/run/docker.sock the host machine is exposed without protection to the build process, so only enable this option for trusted usages.
|
Use Docker Pipeline
In order to use Docker pipeline syntax like docker.image.inside, besides the requirements in previous section,
the workspace needs to be shared across Jenkins nodes.
This can be done by setting the agent filesystem remote root directory to /mnt/Mesos/sandbox/jenkins,
a Docker volume that is automatically injected into all agents.
Although the data in the /mnt/Mesos/sandbox/jenkins is garbage collected, care must be taken to cleanup after jobs to ensure the space is reclaimed as soon as possible.
|
One-Shot provisioner API
Since 1.5.1, Palace uses by default the One-Shot executor API instead of the Cloud API. This API is designed to handle ephemeral agents running in Docker containers, whereas the Cloud API was designed at a time virtual machine provisioning was the norm. As a net result, agent provisioning is snappier, and agents are guaranteed to be used by a single build.
In case this new API causes incompatibilities with the installation, it can be disabled through the Global settings of a Jenkins instance. In this case Palace will revert to the Cloud API to provision agents.
