Introduction to Pipelines

Pipelines are sets of processes that are used to compile, build, and deploy code into a production environment. With Pipelines you can:

  • Manage automated builds, tests, and deployments as one workflow.

  • Deliver quality products frequently and consistently from testing through staging to production automatically.

  • Automatically promote or prevent build artifacts from being deployed. If errors are discovered anytime during the process the Pipeline stops and alerts are sent to the appropriate team for their attention.

To ensure that the quality of the software being developed is not compromised for speed, appropriate checks and balances are built into Pipeline processes. They can be programmed to trigger with commits to the code thereby enabling frequent releases and consistent behavior.

Pipeline example

Pipeline workflow

Pipeline as Code

Pipeline as Code provides a consistent way to build and deploy services and applications by treating them as code. It is used as a tool to execute underlying tasks for various stages of a software delivery lifecycle and takes away the implementation details.

It is a set of features that allow pipelined job processes to be defined with code, stored, and versioned in a source repository. These features allow the discovery, management, and running of jobs for multiple source repositories and branches, eliminating the need for manual job creation and allowing for change management at job level without using the UI.

Both types of Pipelines as described below are Pipeline as Code. Before Pipelines, jobs were configured using the UI , and were part of the config.xml files rather than the code.

Benefits of Pipeline as Code

  1. Pipeline as Code allows integration of your Pipeline with your code thereby making collaboration on it easier. When you update your Pipeline, your changes are automatically picked up thereby eliminating the need to make changes manually.

  2. Storing Pipelines in a version control system enables tracking of changes to the Pipeline. If a change to the Pipeline causes it to break, then it can easily be reverted. It can also be restored more easily.

Types of Pipelines

There are two main types of Pipelines; declarative and scripted.

Declarative Pipelines

Declarative Pipeline provides a structured hierarchial syntax to simplify the creation of Pipelines and the associated Jenkinsfiles. In its simplest form, a Pipeline runs on an agent and contains stages, while each stage contains steps that define specific actions. Here is an example:

pipeline {
    agent { (1)
        label ''
    }
    stages {
        stage(‘Build’) { (2)
            steps { (3)
                sh ‘mvn install’
            }
        }
    }
}
1 agent
2 stages
3 steps

agent

An agent section specifies where tasks in a Pipeline will run. An agent must be present at the top-level inside the pipeline block and defines the default agent for all stages in the Pipeline. An agent may optionally be specified at the stage level, overriding the default for this stage and any child stages. In this example, an agent is specified with an empty label filter, meaning this Pipeline will run on any available agent.

stages

A stage represents a logical grouping of tasks in the Pipeline. Each stage may contain a steps section with steps to be executed, or stages to be executed sequentially, in parallel, or expanded into a parallel matrix.

It contains commands that run processes like Build, Deploy, or Tests. These commands are included in the steps section in the stage.

It is advisible to have descriptive names for a stage as these names are displayed in the UI and logs.

There can be one or more stage inside a stages section. At least one stage must be defined in the stages section.

steps

`steps`contains a set of commands that each perform a specific action and are executed one by one.

pipeline {
    agent any
    stages {
        stage('Example') {
            steps {
                sh 'mvn compile'
            }
        }
    }
}

For more information on Declarative Pipelines, see Using Declarative Pipeline syntax.

Scripted Pipeline

Scripted Pipeline is a more general form of Pipeline as code. It is a DSL (Domain Specific Language) based on Apache Groovy. Most functionality provided by the Groovy language is made available in Scripted Pipelines. This means Scripted Pipelines can offer flexibility and extensibility to users.

However, the learning curve associated with Scripted Pipelines is very steep and it is possible to do things inside Pipelines that are better done in other tools called as part of Pipelines. CloudBees recommends the use of Declarative Pipelines rather than Scripted Pipelines.

For more information on Scripted Pipelines, see Using Scripted Pipeline syntax.