Pipelines user guide

Implementing continuous delivery with pipelines

Continuous delivery allows organizations to deliver software with lower risk. The path to continuous delivery starts by modeling the software delivery pipeline used within the organization and then focusing on the automation of it all. Early, directed feedback, enabled by pipeline automation enables software delivery more quickly over traditional methods of delivery.

Jenkins is the Swiss army knife in the software delivery toolchain. Developers and operations (DevOps) personnel have different mindsets and use different tools to get their respective jobs done. Since Jenkins integrates with a huge variety of toolsets, it serves as the intersection point between development and operations teams.

Many organizations have been orchestrating pipelines with existing Jenkins plugins for several years. As their automation sophistication and their own Jenkins experience increases, organizations inevitably want to move beyond simple pipelines and create complex flows specific to their delivery process.

These Jenkins users require a feature that treats complex pipelines as a first-class object, and so CloudBees engineers developed and contributed the new Jenkins Pipeline feature to the Jenkins open source project. This Pipeline plugin was built with the community’s requirements for a flexible, extensible, and script-based pipeline in mind.

Selecting a pipeline tool

Continuous delivery is a process - rather than a tool - and requires a mindset and culture that must percolate from the top-down within an organization. Once the organization has bought into the philosophy, the next and most difficult part is mapping the flow of software as it makes its way from development to production.

The root of such a pipeline will always be an orchestration tool like a Jenkins, but there are some key requirements that such an integral part of the pipeline must satisfy before it can be tasked with enterprise-critical processes:

  • Zero or low downtime disaster recovery: A commit, just as a mythical hero, encounters harder and longer challenges as it makes its way down the pipeline. It is not unusual to see pipeline executions that last days. A hardware or a Jenkins failure on day six of a seven-day pipeline has serious consequences for on-time delivery of a product.

  • Audit runs and debug ability: Build managers like to see the exact execution flow through the pipeline, so they can easily debug issues.

To ensure a tool can scale with an organization and suitably automate existing delivery pipelines without changing them, the tool should also support:

  • Complex pipelines: Delivery pipelines are typically more complex than canonical examples (linear process: Dev→Test→Deploy, with a couple of operations at each stage). Build managers want constructs that help parallelize parts of the flow, run loops, perform retries and so forth. Stated differently, build managers want programming constructs to define pipelines.

  • Manual interventions: Pipelines cross intra-organizational boundaries necessitating manual handoffs and interventions. Build managers seek capabilities such as being able to pause a pipeline for a human to intervene and make manual decisions.

The Pipeline plugin allows users to create such a pipeline through a new job type called Pipeline. The flow definition is captured in a Groovy script, thus adding control flow capabilities such as loops, forks and retries. Pipeline allows for stages with the option to set concurrencies, preventing multiple builds of the same pipeline from trying to access the same resource at the same time.

Understanding Pipeline benefits

Jenkins is, fundamentally, an automation engine which supports a number of automation patterns. Pipeline adds a powerful set of automation tools onto Jenkins, supporting use cases that span from simple continuous integration to comprehensive continuous delivery pipelines. By modeling a series of related tasks, users can take advantage of the many features of Pipeline:

  • Code: Pipelines are implemented in code and typically checked into source control, giving teams the ability to edit, review, and iterate upon their delivery pipeline.

  • Durable: Pipelines can survive both planned and unplanned restarts of the Jenkins master.

  • Pausable: Pipelines can optionally stop and wait for human input or approval before continuing the Pipeline run.

  • Versatile: Pipelines support complex real-world continuous delivery requirements, including the ability to fork/join, loop, and perform work in parallel.

  • Extensible: The Pipeline plugin supports custom extensions to its DSL and multiple options for integration with other plugins.

Copyright © 2010-2019 CloudBees, Inc.Online version published by CloudBees, Inc. under the Creative Commons Attribution-ShareAlike 4.0 license.CloudBees and CloudBees DevOptics are registered trademarks and CloudBees Core, CloudBees CodeShip, CloudBees Jenkins Enterprise, CloudBees Jenkins Platform, CloudBees Jenkins Operations Center and DEV@cloud are trademarks of CloudBees, Inc. Oracle and Java are registered trademarks of Oracle and/or its affiliates. Jenkins is a registered trademark of the non-profit Software in the Public Interest organization. Used with permission. See here for more info about the Jenkins project. The registered trademark Jenkins® is used pursuant to a sublicense from the Jenkins project and Software in the Public Interest, Inc. Read more at www.cloudbees.com/jenkins/about. Apache, Apache Ant, Apache Maven, Ant and Maven are trademarks of The Apache Software Foundation. Used with permission. No endorsement by The Apache Software Foundation is implied by the use of these marks.Other names may be trademarks of their respective owners. Many of the designations used by manufacturers and sellers to distinguish their products are claimed as trademarks. Where those designations appear in this content, and CloudBees was aware of a trademark claim, the designations have been printed in caps or initial caps. While every precaution has been taken in the preparation of this content, the publisher and authors assume no responsibility for errors or omissions, or for damages resulting from the use of the information contained herein.