Building on CloudBees Jenkins Enterprise

CloudBees will no longer be supporting CloudBees Jenkins Enterprise 1.x after July 30, 2020. This end-of-life announcement allows CloudBees to focus on driving new technology and product innovation for CloudBees CI. For information on moving to CloudBees CI, please refer to CloudBees Jenkins Enterprise 1.x to CloudBees CI on modern cloud platforms migration guide which has been created to help you with the migration process. Existing customers can also contact their CSM to help ensure a smooth transition.

CloudBees Jenkins Enterprise comes out of the box with a set of agent images in order to run builds on the underlying cluster infrastructure. This is managed by Palace, a scheduler that is launched on the CloudBees Jenkins Enterprise infrastructure.

These agent images are available as items in operations center and are pushed to Client Masters located in the same context. You can also create agent images directly on a Client Master.

Palace, scalable agent scheduling framework for Mesos

Why Palace?

CloudBees Jenkins Enterprise used to ship with a shared cloud using the Mesos plugin to provision docker-based agents. While doing scalability tests on CloudBees Jenkins Enterprise, we realized this plugin couldn’t scale as much as we want because each Jenkins master would register one Mesos framework, and Mesos just cannot handle thousands of frameworks.

We therefore built Palace, which is used as an intermediate in order to provision Jenkins agents on Mesos. From Mesos, it is a single framework. From Jenkins masters, it is accessed through a REST API, which removes the need to bundle Mesos binaries within masters.

Comparing architectures

Mesos plugin

Architecture Diagram: Mesos plugin

When using the Mesos plugin, each Jenkins master hosts a Mesos framework, which requires to have the Mesos package installed in the container running Jenkins. There is a competition between the different masters interacting with Mesos. A faulty master could hog on to Mesos offers and prevent other masters from creating tasks. Latest registered masters are given priority over those which have been there for a long time. When many frameworks are connected, it takes a longer time for a given framework to get the offers it is interested in because they need to be processed and declined by other frameworks.

Palace

Architecture Diagram: Palace

Palace is a scheduled application, just like Jenkins masters. In case of failure, it is automatically restarted and it reconciles its state with the currently running Mesos tasks.

Palace works in exclusive mode : since CloudBees Jenkins Enterprise uses workers dedicated to the Palace workload and Palace is a single Mesos framework, it hogs on to offers on these agents and is able to respond instantly whenever a Jenkins agent request is made by one master. Overall it reduces the time to allocate agents.

Having a central place of decision to provision agents also helps to apply more elaborated strategies to do task placement optimizations, e.g.

bin-packing (pack tasks into a few hosts as possible)
load spread (maintain a similar load on all available hosts) (not yet implemented in CloudBees Jenkins Enterprise)
implement resources quotas per master (not yet implemented in CloudBees Jenkins Enterprise)

Jenkins masters connect to Palace using a REST API, which decouple them from Mesos itself : we no longer need to ship Mesos binaries in the docker container, the Mesos native library is no longer called directly from Jenkins which limits impact in case of a crash.

Defining a new agent image

To define a new agent image, just create a new item and select Docker Agent Template.

The name of the item will be the label to use when defining jobs meant to run using this image.

Then you need to configure the agent image to match your requirements.

Field Default Value Description

Field	Default Value	Description
Use when no label is specified		Check this if you want your agent image to be used for unassigned jobs. In case several agent templates have this option selected, these jobs will run on any of them, the first available.
Enable this agent template		Uncheck this to disable a template you don’t want to use (but that you still want to keep around)
CPU shares	`0.1`	Refer to the corresponding Docker option. We recommend you to leave it to its default value unless you really know what you are doing.
Memory	`256`	It is the amount of memory to be allocated for the docker container. If the value is set too low, the container may be killed by the Linux OOM Killer. If it is too high, less tasks will be able to be launched on the infrastructure and you may experience delays when launching jobs.
JVM Memory	`128`	It is the maximum heap size given to the Jenkins agent. You need to account for this value when setting up Memory above.
JVM Arguments	`(empty)`	Allows to specify additional JVM arguments to be applied to the JVM running the Jenkins agent.
JNLP Arguments	`(empty)`	Allows to specify JNLP arguments when launching the Jenkins agent.
Remote FS Root	`/jenkins`	It is the working directory of the Jenkins agent inside the container. It must be a path on which the user running the container as must be able to do anything, otherwise the agent will fail to launch.
Image	`java`	This is the docker tag of the image to pull. It needs to have a JDK installed, as this is a prerequisite to launch the Jenkins agent.
Additional Options
Additional URI		Fetches a file from an URI and put it in the Mesos sandbox. This is useful to pull an image from a private Docker registry.
Environment Variable		Add an environment variable to the Docker container.
Force pull image		Force update of an image if it already exists locally. This option is useful if the specified image targets a mutable Docker tag (by convention) such as `latest`.
Launch in privileged mode		Launches the Docker container in privileged mode.
Parameter		Pass a parameter to the Docker container.
Port Mapping		Allows to map a port from the host to the Docker container.
Use a custom Docker shell command		Allows to override the default `/bin/sh` shell command that is used to launch the Jenkins agent inside the container. This option can be used to execute commands before the Jenkins agent process is started.
Volume		Allows to map a volume inside the Docker container.

Use when no label is specified

Check this if you want your agent image to be used for unassigned jobs. In case several agent templates have this option selected, these jobs will run on any of them, the first available.

Enable this agent template

Uncheck this to disable a template you don’t want to use (but that you still want to keep around)

CPU shares

0.1

Refer to the corresponding Docker option. We recommend you to leave it to its default value unless you really know what you are doing.

Memory

256

It is the amount of memory to be allocated for the docker container. If the value is set too low, the container may be killed by the Linux OOM Killer. If it is too high, less tasks will be able to be launched on the infrastructure and you may experience delays when launching jobs.

JVM Memory

128

It is the maximum heap size given to the Jenkins agent. You need to account for this value when setting up Memory above.

JVM Arguments

(empty)

Allows to specify additional JVM arguments to be applied to the JVM running the Jenkins agent.

JNLP Arguments

(empty)

Allows to specify JNLP arguments when launching the Jenkins agent.

Remote FS Root

/jenkins

It is the working directory of the Jenkins agent inside the container. It must be a path on which the user running the container as must be able to do anything, otherwise the agent will fail to launch.

Image

java

This is the docker tag of the image to pull. It needs to have a JDK installed, as this is a prerequisite to launch the Jenkins agent.

Additional Options

Additional URI

Fetches a file from an URI and put it in the Mesos sandbox. This is useful to pull an image from a private Docker registry.

Environment Variable

Add an environment variable to the Docker container.

Force pull image

Force update of an image if it already exists locally. This option is useful if the specified image targets a mutable Docker tag (by convention) such as latest.

Launch in privileged mode

Launches the Docker container in privileged mode.

Parameter

Pass a parameter to the Docker container.

Port Mapping

Allows to map a port from the host to the Docker container.

Use a custom Docker shell command

Allows to override the default /bin/sh shell command that is used to launch the Jenkins agent inside the container. This option can be used to execute commands before the Jenkins agent process is started.

Volume

Allows to map a volume inside the Docker container.

Sharing agent images

You can share agent images by creating them on operations center. Agent images created in operations center will be made available to client masters located in the same folder hierarchy (same folder or a subfolder below it).

Agent images created by the installer are located in the root of operations center. Then are then available to any Client Master.

Shared agent images can be browsed from any Client Master within the special folder Operations Center Shared Templates.

Example of restricted agent image

Considering a folder named Organization 1, an agent template named agent1 and a master Master 1.

If we browse the Shared Templates within Master 1, we will see the template agent1 as well as the other templates defined in operations center.

However if there are masters in operations center root folder or in other folders, they won’t see the template agent1.

Migrating from the Mesos shared cloud to Palace

CloudBees Jenkins Enterprise used to ship with a shared cloud using the Mesos plugin to provision docker-based agents.

Since 1.3.0, CloudBees Jenkins Enterprise ships with a new shared cloud called Palace. It effectively deprecates the Mesos shared cloud by providing a better scalability model as well as a more flexible way to manage docker agent images across your cluster.

We don’t migrate automatically your templates based on the Mesos plugin as we don’t guarantee a 100% compatibility with the Mesos plugin configuration.

Migrating agent configuration

In particular, only Mesos agent configurations using the Docker containerizer are supported by Palace.

Each Mesos agent configuration corresponds to a Docker Agent Template item when using Palace.

Field in Mesos plugin	Field in Palace
Label string	This is the item name
Jenkins Agent CPUs	CPU Shares = (Jenkins Agent CPUs) + (Number of executors)*(Jenkins Executor CPUs)
Jenkins Agent Memory in MB	JVM Memory
Minimum number of Executors per Agent	Not Applicable (always 1)
Maximum number of Executors per Agent	Not Applicable (always 1)
Jenkins Executor CPUs	See Jenkins Agent CPUs
Jenkins Executor Memory in MB	Memory = (Jenkins Agent Memory) + (Number of executors) * (Jenkins Executor Memory)
Remote FS Root	Remote FS Root
Idle Termination Minutes	Not Applicable. Agents are removed as soon as the build is done.
Mesos Offer Selection Attributes	Not Applicable
Additional Jenkins Agent JVM arguments	JVM Arguments
Additional Jenkins Agent Agent JNLP arguments	JNLP Arguments
Docker Image	Image
Docker Privileged Mode	Options > Launch in privileged mode
Docker Force Pull Image	Options > Force pull image
Docker Image Can Be Customized	Not supported
Custom docker command shell	Use a custom docker shell command
Networking options	Not supported
Volumes	Options > Volume
Parameters	Options > Parameter
Additional URIs	Options > Additional URI

Field in Mesos plugin

Field in Palace

Label string

This is the item name

Jenkins Agent CPUs

CPU Shares = (Jenkins Agent CPUs) + (Number of executors)*(Jenkins Executor CPUs)

Jenkins Agent Memory in MB

JVM Memory

Minimum number of Executors per Agent

Not Applicable (always 1)

Maximum number of Executors per Agent

Not Applicable (always 1)

Jenkins Executor CPUs

See Jenkins Agent CPUs

Jenkins Executor Memory in MB

Memory = (Jenkins Agent Memory) + (Number of executors) * (Jenkins Executor Memory)

Remote FS Root

Idle Termination Minutes

Not Applicable. Agents are removed as soon as the build is done.

Mesos Offer Selection Attributes

Not Applicable

Additional Jenkins Agent JVM arguments

JVM Arguments

Additional Jenkins Agent Agent JNLP arguments

JNLP Arguments

Docker Image

Image

Docker Privileged Mode

Options > Launch in privileged mode

Docker Force Pull Image

Options > Force pull image

Docker Image Can Be Customized

Not supported

Custom docker command shell

Use a custom docker shell command

Networking options

Not supported

Volumes

Options > Volume

Parameters

Options > Parameter

Additional URIs

Options > Additional URI

Migrating workers

Palace uses a different strategy to work with Mesos offers than the Mesos plugin does. This allows a quicker time to provision agents as well as a better control on where tasks are getting provisioned.

For this reason and to avoid disrupting your workload when upgrading, we have decided to require specific workers for Palace.

When listing workers using cje run list-workers, you will notice that your existing build workers are now tagged with mp-build. It means these workers are assigned to workload generated by the Mesos plugin.

worker-1 (master: m4.xlarge) ec2-aa-bb-cc-d1.compute-1.amazonaws.com > ACTIVE
worker-3 (mp-build: m4.xlarge) ec2-aa-bb-cc-d2.compute-1.amazonaws.com > ACTIVE
worker-2 (master: m4.xlarge) ec2-aa-bb-cc-d3.compute-1.amazonaws.com > ACTIVE

To add new workers that can be used by Palace, run the worker-add operation and choose workload_type = build. Once you are done with your migration you will be able to remove the old mp-build workers using the worker-remove operation.

Creating Credential Package

To use a private registry, CloudBees Jenkins Enterprise needs credentials to access the private registry. Create a docker.tar.gz file including the .docker folder and the contained .docker/config.json by:

Login to the private registry manually. Login creates a .docker folder and a .docker/config.json in the user’s home directory.

$ docker login some.docker.example.com
  Username: foo
  Password:
  Email: foo@example.com

Tar this folder and its contents.

$ cd ~
$ tar czf docker.tar.gz .docker

Check you have both files in the tar

$ tar -tvf ~/docker.tar.gz

  drwx------ root/root         0 2015-07-28 02:54 .docker/
  -rw------- root/root       114 2015-07-28 01:31 .docker/config.json

Put the gzipped file in a location that can be retrieved from the worker VMs. The URI must be accessible by all workers. Approaches may include storing it on a shared network drive, or for example Amazon S3. It is worth considering the security implications of your chosen approach.

Attaching Credential Package to Custom Build Environment

Then, in the agent image, add an Additional URI entry. In Value, provide the URI to download the file, then check Extract option.

Use Docker Commands in Builds

There are two Docker-related build tasks that are available as build tasks.

In order to be able to use these Docker-related tasks in a build environment, a Palace build agent definition with the Docker client is required. The Docker image cloudbees/java-with-docker-client can be used for this purpose. It has both Java (for the Jenkins agent process) and Docker client pre-installed. The Docker socket of the enclosing CloudBees Jenkins Enterprise worker also needs to be mounted through a volume inside the container.

To do so, in the agent image configuration, add a Volume entry, with Host Path and Container Path set to /var/run/docker.sock.

Configuration changes to agent images are only applied to new containers, not to existing ones.

Be aware that by letting a build access /var/run/docker.sock the host machine is exposed without protection to the build process, so only enable this option for trusted usages.

Use Docker Pipeline

In order to use Docker pipeline syntax like docker.image.inside, besides the requirements in previous section, the workspace needs to be shared across Jenkins nodes. This can be done by setting the agent filesystem remote root directory to /mnt/Mesos/sandbox/jenkins, a Docker volume that is automatically injected into all agents.

Although the data in the /mnt/Mesos/sandbox/jenkins is garbage collected, care must be taken to cleanup after jobs to ensure the space is reclaimed as soon as possible.

One-Shot provisioner API

Since 1.5.1, Palace uses by default the One-Shot executor API instead of the Cloud API. This API is designed to handle ephemeral agents running in Docker containers, whereas the Cloud API was designed at a time virtual machine provisioning was the norm. As a net result, agent provisioning is snappier, and agents are guaranteed to be used by a single build.

In case this new API causes incompatibilities with the installation, it can be disabled through the Global settings of a Jenkins instance. In this case Palace will revert to the Cloud API to provision agents.