|CloudBees will no longer be supporting CloudBees Jenkins Enterprise 1.x after July 30, 2020. This end-of-life announcement allows CloudBees to focus on driving new technology and product innovation for CloudBees CI. For information on moving to CloudBees CI, please refer to CloudBees Jenkins Enterprise 1.x to CloudBees CI on modern cloud platforms migration guide which has been created to help you with the migration process. Existing customers can also contact their CSM to help ensure a smooth transition.|
CloudBees Jenkins Enterprise comes out of the box with a set of agent images in order to run builds on the underlying cluster infrastructure. This is managed by Palace, a scheduler that is launched on the CloudBees Jenkins Enterprise infrastructure.
These agent images are available as items in operations center and are pushed to Client Masters located in the same context. You can also create agent images directly on a Client Master.
CloudBees Jenkins Enterprise used to ship with a shared cloud using the Mesos plugin to provision docker-based agents. While doing scalability tests on CloudBees Jenkins Enterprise, we realized this plugin couldn’t scale as much as we want because each Jenkins master would register one Mesos framework, and Mesos just cannot handle thousands of frameworks.
We therefore built Palace, which is used as an intermediate in order to provision Jenkins agents on Mesos. From Mesos, it is a single framework. From Jenkins masters, it is accessed through a REST API, which removes the need to bundle Mesos binaries within masters.
When using the Mesos plugin, each Jenkins master hosts a Mesos framework, which requires to have the Mesos package installed in the container running Jenkins. There is a competition between the different masters interacting with Mesos. A faulty master could hog on to Mesos offers and prevent other masters from creating tasks. Latest registered masters are given priority over those which have been there for a long time. When many frameworks are connected, it takes a longer time for a given framework to get the offers it is interested in because they need to be processed and declined by other frameworks.
Palace is a scheduled application, just like Jenkins masters. In case of failure, it is automatically restarted and it reconciles its state with the currently running Mesos tasks.
Palace works in exclusive mode : since CloudBees Jenkins Enterprise uses workers dedicated to the Palace workload and Palace is a single Mesos framework, it hogs on to offers on these agents and is able to respond instantly whenever a Jenkins agent request is made by one master. Overall it reduces the time to allocate agents.
Having a central place of decision to provision agents also helps to apply more elaborated strategies to do task placement optimizations, e.g.
bin-packing (pack tasks into a few hosts as possible)
load spread (maintain a similar load on all available hosts) (not yet implemented in CloudBees Jenkins Enterprise)
implement resources quotas per master (not yet implemented in CloudBees Jenkins Enterprise)
Jenkins masters connect to Palace using a REST API, which decouple them from Mesos itself : we no longer need to ship Mesos binaries in the docker container, the Mesos native library is no longer called directly from Jenkins which limits impact in case of a crash.
To define a new agent image, just create a new item and select Docker Agent Template.
The name of the item will be the label to use when defining jobs meant to run using this image.
Then you need to configure the agent image to match your requirements.
Use when no label is specified
Check this if you want your agent image to be used for unassigned jobs. In case several agent templates have this option selected, these jobs will run on any of them, the first available.
Enable this agent template
Uncheck this to disable a template you don’t want to use (but that you still want to keep around)
Refer to the corresponding Docker option. We recommend you to leave it to its default value unless you really know what you are doing.
It is the amount of memory to be allocated for the docker container. If the value is set too low, the container may be killed by the Linux OOM Killer. If it is too high, less tasks will be able to be launched on the infrastructure and you may experience delays when launching jobs.
It is the maximum heap size given to the Jenkins agent. You need to account for this value when setting up Memory above.
Allows to specify additional JVM arguments to be applied to the JVM running the Jenkins agent.
Allows to specify JNLP arguments when launching the Jenkins agent.
Remote FS Root
It is the working directory of the Jenkins agent inside the container. It must be a path on which the user running the container as must be able to do anything, otherwise the agent will fail to launch.
This is the docker tag of the image to pull. It needs to have a JDK installed, as this is a prerequisite to launch the Jenkins agent.
Fetches a file from an URI and put it in the Mesos sandbox. This is useful to pull an image from a private Docker registry.
Add an environment variable to the Docker container.
Force pull image
Force update of an image if it already exists locally. This option is useful
if the specified image targets a mutable Docker tag (by convention) such as
Launch in privileged mode
Launches the Docker container in privileged mode.
Pass a parameter to the Docker container.
Allows to map a port from the host to the Docker container.
Use a custom Docker shell command
Allows to override the default
Allows to map a volume inside the Docker container.
You can share agent images by creating them on operations center. Agent images created in operations center will be made available to client masters located in the same folder hierarchy (same folder or a subfolder below it).
Agent images created by the installer are located in the root of operations center. Then are then available to any Client Master.
Shared agent images can be browsed from any Client Master within the special folder Operations Center Shared Templates.
Considering a folder named Organization 1, an agent template named agent1 and a master Master 1.
If we browse the Shared Templates within Master 1, we will see the template agent1 as well as the other templates defined in operations center.
However if there are masters in operations center root folder or in other folders, they won’t see the template agent1.
CloudBees Jenkins Enterprise used to ship with a shared cloud using the Mesos plugin to provision docker-based agents.
Since 1.3.0, CloudBees Jenkins Enterprise ships with a new shared cloud called Palace. It effectively deprecates the Mesos shared cloud by providing a better scalability model as well as a more flexible way to manage docker agent images across your cluster.
We don’t migrate automatically your templates based on the Mesos plugin as we don’t guarantee a 100% compatibility with the Mesos plugin configuration.
In particular, only Mesos agent configurations using the Docker containerizer are supported by Palace.
Each Mesos agent configuration corresponds to a Docker Agent Template item when using Palace.
|Field in Mesos plugin||Field in Palace|
This is the item name
Jenkins Agent CPUs
CPU Shares = (Jenkins Agent CPUs) + (Number of executors)*(Jenkins Executor CPUs)
Jenkins Agent Memory in MB
Minimum number of Executors per Agent
Not Applicable (always 1)
Maximum number of Executors per Agent
Not Applicable (always 1)
Jenkins Executor CPUs
See Jenkins Agent CPUs
Jenkins Executor Memory in MB
Memory = (Jenkins Agent Memory) + (Number of executors) * (Jenkins Executor Memory)
Remote FS Root
Remote FS Root
Idle Termination Minutes
Not Applicable. Agents are removed as soon as the build is done.
Mesos Offer Selection Attributes
Additional Jenkins Agent JVM arguments
Additional Jenkins Agent Agent JNLP arguments
Docker Privileged Mode
Options > Launch in privileged mode
Docker Force Pull Image
Options > Force pull image
Docker Image Can Be Customized
Custom docker command shell
Use a custom docker shell command
Options > Volume
Options > Parameter
Options > Additional URI
Palace uses a different strategy to work with Mesos offers than the Mesos plugin does. This allows a quicker time to provision agents as well as a better control on where tasks are getting provisioned.
For this reason and to avoid disrupting your workload when upgrading, we have decided to require specific workers for Palace.
When listing workers using
cje run list-workers, you will notice that your existing build workers are now tagged with
It means these workers are assigned to workload generated by the Mesos plugin.
worker-1 (master: m4.xlarge) ec2-aa-bb-cc-d1.compute-1.amazonaws.com > ACTIVE worker-3 (mp-build: m4.xlarge) ec2-aa-bb-cc-d2.compute-1.amazonaws.com > ACTIVE worker-2 (master: m4.xlarge) ec2-aa-bb-cc-d3.compute-1.amazonaws.com > ACTIVE
To add new workers that can be used by Palace, run the
worker-add operation and choose
workload_type = build.
Once you are done with your migration you will be able to remove the old
mp-build workers using the
To use a private registry, CloudBees Jenkins Enterprise needs credentials to access the private registry.
docker.tar.gz file including the
.docker folder and the contained
Login to the private registry manually. Login creates a
.dockerfolder and a
.docker/config.jsonin the user’s home directory.
$ docker login some.docker.example.com Username: foo Password: Email: email@example.com
Tar this folder and its contents.
$ cd ~ $ tar czf docker.tar.gz .docker
Check you have both files in the tar
$ tar -tvf ~/docker.tar.gz drwx------ root/root 0 2015-07-28 02:54 .docker/ -rw------- root/root 114 2015-07-28 01:31 .docker/config.json
Put the gzipped file in a location that can be retrieved from the worker VMs. The URI must be accessible by all workers. Approaches may include storing it on a shared network drive, or for example Amazon S3. It is worth considering the security implications of your chosen approach.
There are two Docker-related build tasks that are available as build tasks.
In order to be able to use these Docker-related tasks in a build environment, a Palace build agent definition with the Docker client is required.
The Docker image
cloudbees/java-with-docker-client can be used for this purpose. It has both Java (for the Jenkins agent process)
and Docker client pre-installed.
The Docker socket of the enclosing CloudBees Jenkins Enterprise worker also needs to be mounted through a volume inside the container.
To do so, in the agent image configuration, add a Volume entry, with Host Path and Container Path set to
|Configuration changes to agent images are only applied to new containers, not to existing ones.|
Be aware that by letting a build access |
In order to use Docker pipeline syntax like
docker.image.inside, besides the requirements in previous section,
the workspace needs to be shared across Jenkins nodes.
This can be done by setting the agent filesystem remote root directory to
a Docker volume that is automatically injected into all agents.
Although the data in the |
Since 1.5.1, Palace uses by default the One-Shot executor API instead of the Cloud API. This API is designed to handle ephemeral agents running in Docker containers, whereas the Cloud API was designed at a time virtual machine provisioning was the norm. As a net result, agent provisioning is snappier, and agents are guaranteed to be used by a single build.
In case this new API causes incompatibilities with the installation, it can be disabled through the Global settings of a Jenkins instance. In this case Palace will revert to the Cloud API to provision agents.