Using Jenkins X pipelines
In modern software development environments, the process of moving software from planning and analysis to finished deployed product is known as the development life cycle. In continuous delivery (CD) environments, a pipeline is a process (expressed as a collection of commands or plugins and a configuration file to express the development process) that automates the life cycle from repository source files to production deployment.
Every change to the source in the repository runs through the specified life cycle step until it is released. The pipeline controls the steps and its cadence, as well as progressing the build and using promotion to transition the source from test to staging to deployment.
Jenkins X Pipelines is a serverless pipeline execution engine based on the Tekton Pipelines open source project. Tekton has been designed to be a modern cloud native solution for running pipelines.
CloudBees Jenkins X Distribution pipelines are configured in YAML configuration files. The files can be found in two locations serving distinct purposes:
In the CloudBees Jenkins X Distribution project repository, called
In the build packs for creating applications, if it is specified in the project repository
Formatting for the the pipeline is itself separated into separate pipelines, which are themselves separated into life cycles: distinct development steps that define the build and deployment actions for your development environment. The
Each pipeline YAML file has a number of separate logical pipelines:
releasefor processing merges to the
masterbranch which typically creates a new version and release then triggers promotion
pullRequestfor processing Pull Requests
featurefor processing merges to a feature branch. Consider using trunk based development which is a practice of high performing teams.
CloudBees Jenkins X Distribution has various steps in building, validating, and releasing your application through the development lifecycle. The lifecycle phases in the Jenkins X pipeline YAML configuration are:
setup- Steps to create the build environment, such as checking out code with
git checkoutor generating credentials files for Git provider authentication
preBuild- Steps to perform before a build occurs, such as ensuring a Docker image registry is available for building
build- Steps performed to build your application
postBuild- Steps performed after the build occurs, such as validating for Common Vulnerability Exposure (CVE) in any code changes.
promote- Shifting the state of an application (after build and validation) to another environment, such as
The CloudBees Jenkins X Distribution cluster configuration process creates a YAML-based pipeline configuration file called
jenkins-x.yml. This file configures the default development pipeline for building applications on kubernetes clusters with Jenkins X.
buildPack: none pipelineConfig: pipelines: release: pipeline: agent: image: gcr.io/jenkinsxio/builder-go
buildPack specifies a build pack which contains a
pipeline.yml file that supersedes the
jenkins-x.yml file in the project directory. If
none is specified, there is no build pack and CloudBees Jenkins X Distribution continues with the default pipeline configuration.
The configuration defines the pipeline
agent, in this case a Google Container Registry image for the Go language build tools.
environment: - name: DEPLOY_NAMESPACE value: jx
environment specifies environment variables used in the pipeline configuration. In this instance, the
DEPLOY_NAMESPACE envar is used with a value of
jx for the Jenkins X namespace.
stages: - name: release steps: - name: verify-preintall dir: /workspace/source/env command: jx args: ['step','verify','preinstall']
stages are unique groups of steps (or nested stages sequentially run within a stage) that specify commands, directories, and arguments for a particular pipeline stage. In this instance, there is a step within the
release stage called
verify-preinstall that runs a
jx command that verifies whether cloud infrastructure (such as the presence of the
kubectl binary and the correct version of
git is installed) was setup in the preinstallation process.
- name: install-vault dir: /workspace/source/systems/vault command: jx args: ['step', 'boot','vault']
name calls out a unique step in the pipeline configuration that defines development steps to verify and apply arguments to various commands necessary for the stage under which it is nested. In this instance,
install-vault installs the Hashicorp Vault tool for secrets management.
- name: apply-repositories dir: /workspace/source/repositories command: jx args: ['step','helm','apply', '--name', 'repos']
This step creates and applies the Helm Package Manager for installation and management of helm kubernetes applications.
- name: apply-pipeline-schedulers dir: /workspace/source/prowConfig command: jx args: ['step','scheduler','config', 'apply', '--direct=true']
This step allows the pipeline to work with a scheduler, which executes program jobs unattended in the background.
- name: update-webhooks dir: /workspace/source/repositories command: jx args: ['update','webhooks','--verbose', '--warn-on-fail']
This step updates webhooks, which is a service that listens for GitHub acitivity and trigger jobs, send automated messages to chat clients such as Slack, and other configurable actions.
- name: verify-install dir: /workspace/source/env command: jx args: ['step','verify','install', '--pod-wait-time', '30m']
This step verifies the project installation, downloading and installing or updating components when necessary.
pullRequest: pipeline: agent: image: gcr.io/jenkinsxio/builder-go
pullRequest is a logical pipeline within the project pipeline that specifies how pull requests are managed when changes are made to the project repository in GitHub.
stages: - name: release steps: - name: helm-build dir: /workspace/source/env command: make args: ['build']
pullRequest pipeline contains a stage wherein steps can also be executed. In this instance,
make is run to create a helm chart and validate that a build has been completed.
A pipeline YAML can extend another YAML file. You can reference a base pipeline YAML using the following methods:
fileto reference a relative file path in the same build pack
extends: file: ../jenkins-x.yaml
importto reference a YAML file:
extends: import: classic file: maven/pipeline.yaml
which then refers to a named imported module via
modules: - name: classic gitUrl: https://github.com/jenkins-x-buildpacks/jenkins-x-kubernetes.git gitRef: master
Users can override steps in a pipeline YAML from a base pipeline YAML, similar to overriding classes in languages like Java. This allows users reuse the steps in a base pipeline’s lifecycle, then add additional steps.
By default any steps you define are added after the base pipeline YAML steps. For example:
extends: file: base-pipeline.yaml pipelines: pullRequest: build: steps: - sh: export VERSION=$PREVIEW_VERSION && skaffold build -f skaffold.yaml
You can add steps before the base pipeline steps using the
extends: file: base-pipeline.yaml pipelines: release: setup: preSteps: - sh: echo BEFORE BASE SETUP steps: - sh: echo AFTER BASE SETUP build: replace: true steps: - sh: mvn clean deploy -Pmyprofile comment: this command is overridden from the base pipeline
If you want to completely replace all the steps from a base pipeline for a
particular lifecycle you can use
replace: true steps: - sh: mvn clean deploy -Pmyprofile comment: this command is overridden from the base pipeline
If you have any questions or feedback on the CloudBees Jenkins X Distribution documentation, send them to firstname.lastname@example.org.