Controlling builds

Restarting aborted builds

CloudBees attempts to bring your instance back up quickly after a crash, power outage, or other system failure that resulted in a loss of environment. If any jobs are in the queue when the crash occurs, they are still present after a restart.

In-progress Freestyle builds cannot be resumed where they left off. Recreating the context of the build is typically too complex. Long-running builds may be used when the context is simple.

Restarting Freestyle builds

If a Freestyle build completes within control of the controller, including build failures, manual aborts, and termination due to a scheduled shutdown, then nothing special is done.

However, after a hard crash, the controller may no longer retain the build record. The CloudBees Restart Aborted Builds Plugin helps to manage exceptional halts like this. It ensures that at least a partial record of every build is saved immediately after it starts and after each configured build step. If the instance is halted suddenly, due to a crash or freeze, the list of all builds that are currently running is recorded. When the instance is restarted, all aborted builds are displayed on an administrative monitor page, where they can be inspected or restarted.

To restart aborted builds:

Ensure you are signed in to the controller as a user with the Administer permission.
When the instance is abruptly terminated after a restart, select Manage Jenkins. If builds are in progress, a warning is displayed.

Figure 1. Administrative warning
Select View details to see a list of all running builds that were interrupted.

Figure 2. Builds list
Select any build to view build details, such as the changelog or console log up to the break point. If the job was parameterized, the list displays the parameter values for that build.
Select Restart build next to a build. A new build of the same job is scheduled, including any associated parameters. Restarting the build removes the item from the list.

Restarting builds after a restore

CloudBees Restart Aborted Builds Plugin, version 1.17 or later, is required to restart builds after a restore.

When a controller is restored from a backup, the Restart Aborted Builds plugin helps to manage:

Aborted Freestyle builds that were interrupted due to a controller crash or sudden halt.
Aborted Pipeline builds that were interrupted due to an agent crash or sudden halt, and the Pipeline build could not be automatically recovered.
Active Pipeline builds that were running when the backup was taken if the builds are still running at the time the page is displayed.

If a Pipeline build is started after a controller backup is taken, when the controller is restored from the backup, new builds may reuse existing build numbers. This is typically harmless because any deployed artifacts or reports should use unique identifiers based on commit, date, or similar. However, some projects may rely on the uniqueness of the build number. In this scenario, a restore script can be used to set a controller RESTORED_FROM_BACKUP environment variable to any identifier. If this variable is defined when the controller starts and the environment variable is either reset or set to a different value during the last startup, a pluggable set of actions are launched to adapt to the restoration and 1000 is added to the next build number of every job. This ensures subsequent build numbers are unlikely to overlap with builds that may have started after the corresponding backup.

If you are using Velero, use the OSS inject-metadata-velero-plugin to automatically set the RESTORED_FROM_BACKUP environment variable during a restore operation.

For information on using Velero to back up and restore Kubernetes cluster resources, refer to Using Velero to back up and restore Kubernetes cluster resources.

To restart builds after a restore:

Ensure you are signed in to the controller as a user with the Administer permission.
After the controller is restored from a backup, select Manage Jenkins. If builds are in progress, a warning is displayed, indicating the controller is newly restored from backup.

Figure 3. Administrative warning
Select View details to see a list of all running builds that were interrupted.

Figure 4. Builds list
Select any build to view build details.
- If a Freestyle build was interrupted due to a crash or sudden halt, you can restart it.
- If a Pipeline build is paused for input, you may be able to resume the build where it left off without interruption.

Restarting pipeline node blocks using the retry option

Retry is a mechanism that allows you to automatically restart the node block within a Pipeline build. If a controller or agent fails within a Pipeline, the retry option recovers data and restarts that piece of the build without interrupting the flow of the build or requiring user intervention. Retries can occur when:

Agents running on virtual machines are abruptly preempted.
Builds are aborted due to loss of the agent infrastructure.
There is a network timeout.

Retries do not occur when sh scripts exit with nonzero code or when a build is aborted or superseded.

You can configure an automatic retry for both Declarative and Scripted Pipelines.

Configuring retry for Declarative Pipelines

You can configure the retry option for Declarative Pipelines within the Pipeline Syntax.

To configure automatic retry for Declarative Pipelines:

Select a controller from the Dashboard. The controller page displays.
Select Pipeline Syntax from the left navigation menu. The Pipeline Syntax page displays.

Figure 5. Configure retry count setting
Select Declarative Directive Generator from the left navigation menu.
Select agent:Agent from the Sample Directive list.
Select label: Run on an agent matching a label from the Agent dropdown list.
Enter a number in the Retry Count field. The number must be over 1.
Select Generate Declarative Directive. The Pipeline code displays in the box below the button.

Figure 6. Pipeline code with retry values

You can copy that code directly into the Pipeline block of your Jenkinsfile for top-level directives or into a stage block for stage directives.

Configuring retry for Scripted Pipelines

To configure retry for a Scripted Pipeline, wrap any node block in a retry block and specify conditions of the nonresumable step (a controller restarted during a pipeline step that cannot resume cleanly). Then specify either an agent (agent connection errors) or kubernetesAgent (errors related to Kubernetes pod agent behaviors). The following is an example of the scripted pipeline:

podTemplate(/*…as usual…*/) {
  retry(count: 2, conditions: [kubernetesAgent(), nonresumable()]) {
    node(POD_LABEL) {
      // … as usual
    }
  }
}

Long-running builds

What happens to builds that were running when Jenkins crashes, or is restarted not in "safe" mode (waiting for running builds to complete)? Whether you are using High Availability to start another Jenkins controller, builds of regular projects that were already running will be aborted. The CloudBees Restart Aborted Builds Plugin allows you to find and reschedule them, but for builds of projects which normally take a long time, perhaps hours or even days, this is not enough.

Builds that need to run for an extended period are best implemented with Pipeline and Inserting checkpoints. If builds cannot be implemented with Pipeline and need to run for an extended period and continue running even if the controller restarts, they can be defined as a "long-running build" type.

To address the needs of people who have legacy builds that are too long to interrupt every time a Jenkins agent is reconnected or Jenkins is restarted for a plugin update, CloudBees CI includes the Long-Running Build plugin that provides a "long-running project" type. The configuration is almost the same as for a standard free-style project, with one difference: the part of your build that you want to run apart from Jenkins should be configured as a (Unix) shell or (Windows) batch step. Of course this script could in turn run Maven, Make, or other tools.

If the agent is reconnected or Jenkins restarted during this "detached build" phase, your build keeps on running uninterrupted on the agent machine (so long as that machine is not rebooted of course). When Jenkins makes contact with the agent again, it will continue to show log messages where it left off, and let the build continue. After the main phase is done, you can run the usual post-build steps, such as archiving artifacts or recording JUnit-style test results.

Make a new job and select Long-Running Project and note the Detached Build section. Pick a kind of build step to run—Bourne shell for Unix agents, or batch script for Windows agents—and enter some commands to run in your build’s workspace.

Figure 7. Long-Running Build Configuration

When the project is built, initially the executor widget will look the same as it would for a freestyle project. During this initial phase, SCM checkouts/updates and similar pre-build steps may be performed. Soon you will see a task in the widget with the (detached) annotation. This means that your main build step is running, and should continue running even if the Jenkins server is halted or loses its connection to the agent. (So long as the connection is open, you should see any new output produced by the detached build step in your build log, with a delay of a few seconds.)

The task label will show post steps while any post-build actions are performed, such as archiving artifacts or recording JUnit test results. This phase does not survive a crash: it requires a constant connection from the Jenkins controller to the agent.

There are a number of limitations and restrictions on what Jenkins features work in long-running builds. Generally speaking, anything that works in a freestyle project in the pre-build or post-build phase should also work. But general build steps are not available except as part of the non-detached pre-build phase, and build wrappers will generally not work. Also surviving a restart can only work if Jenkins can reconnect to the exact same agent without that machine having rebooted, so this will generally not work on cloud-provisioned agents. Consult the release notes for more information.

Skip next build

The Skip Next Build plugin allows you to skip building a job for a short period of time. While you could skip building a job by disabling it from the job configure page, you would need to remember to enable the job.

Use this plugin:

If you are going to take some external resources that the build requires offline for maintenance and you don’t want to see any build failure notices.
If you are merging a major feature branch and you want to prevent builds until after the merge is completed.

The plugin adds a Skip Builds action to all jobs. When a skip is applied to the job, a message with an orange background appears across the main job page as shown in the image below.

Figure 8. The main job screen when a skip has been applied

To apply a skip to a folder or job:

Select a job or folder from your controller dashboard.
Select Skip builds on the left navigation menu. The Skip builds page displays.

Figure 9. Applying a skip to a folder/job
Under Skip build for, select how long to apply the skip. You can select a value from 1 to 8 hours or enter a value up to 100 in the hours field.
Select Apply skip. A message displays across the main job page that builds are skipped until the specified time.

When a skip is applied to a folder, all jobs within the folder will be skipped.

To remove a skip from a folder or job:

Select Skip builds on the left navigation menu.

Figure 10. Removing a skip from a job
Select Remove skip to remove the skip. If the skip was not applied directly to the folder/job but instead is either inherited from a parent folder or originating from a skip group, the screen will look similar to the image below.

Figure 11. Trying to remove a skip from a job where the skip is inherited from a parent folder.

The link(s) in the table of active skips can be used to navigate to the corresponding skip.

Skip groups

Depending on how the jobs in your Jenkins instance are organized and the reasons for skipping builds, it may be necessary to select a disjoint set of jobs from across the instance for skipping. Skip groups can be used to combine jobs from different folders so that they can be skipped as a single group. The Jenkins administrator configures skip groups in the global configuration.

To create a skip group:

From the controller dashboard, select Manage Jenkins on the left navigation menu.
On the Manage Jenkins page, select Configure System.
On the Configure System page, select Skip groups from the breadcrumb bar context menu.

Figure 12. Navigating to the Jenkins Manage Jenkins Configure System Skip groups section using the breadcrumb bar’s context menu.
Select Add.

Configure the skip group.

Figure 13. Adding a skip group to the Jenkins global configuration

Enter a skip group title in the Name field.
Enter information in the Description field that explains to users why the skip has been applied to their jobs.

Use ANT Path expressions in the Include: and Exclude: fields to identify jobs to be included or excluded from the skip group.

Include: An expression to match elements to be added in the skip group, coming from the Jenkins instance.

Exclude: An expression to match elements from the Skip Group that will be excluded from the Skip Group.

To configure the ANT Path expressions for the Include: and Exclude: fields use the following example.

myproject (a folder for your development project):

build jobs for branches and features:
- myproject/build-core
- myproject/build-api
- myproject/build-ui
- myproject/build-data
release jobs for the application:
- myproject/release-test
- myproject/release-qa
- myproject/release-prod

If you are planning on releasing your project and would like to skip all of the build projects, but not the release jobs, you can create a skip group by entering the following in the Include: field:

Include: myproject/build-**

If your project has additional jobs:

testing jobs for the application:
- myproject/test-commons
- myproject/test-ui
- myproject/test-regression

You can create a skip group by entering the following in the Exclude: field:

Include: myproject/**
Exlude: myproject/release-**

When you use the Include: field for ANT Path expressions, you do not have to use the Exclude: field. However, you cannot use the Exclude: field without the Include: field.

Select Add again if you want to create more skip groups.

Adding folders or jobs to a skip group

Once skip groups are defined, you can configure the jobs and/or folder membership from the job/folder configuration page.

Figure 14. Configuring a folder’s skip group membership

1	Select Skip group membership.
2	Select a skip group.

When there is at least one skip group defined in a Jenkins instance, the Jenkins Skip groups page is enabled. The Skip groups page contains a table that lists the available skip groups.

Figure 15. The Skip groups page

To manage the skip state of a skip group, select the skip group name in the table to navigate to that skip group’s details page.

Figure 16. The details page for a specific skip group

The details page displays the current status of the skip group and lists all of the items that are a direct member of this skip group.

Where folders are a member of a skip group, the skip group membership will be inherited by all items in the folder.

Controlling skips with Jenkins CLI

The Skip Next Build plugin adds a number of Jenkins CLI commands for controlling skips.

apply-skip: Enables the skip setting on a job.

Figure 17. Command apply-skip

This command takes two parameters:

The full name of the job.
The number of hours the skip should be active.

apply-folder-skip: Enables the skip setting on a folder.

Figure 18. Command apply-folder-skip

This command takes two parameters:

The full name of the folder.
The number of hours the skip should be active.

skip-group-on: Enables the skip setting on a skip group.

Figure 19. Command skip-group-on

This command takes two parameters:

The name of the skip group.
The number of hours the skip should be active.

remove-skip: Removes the currently active skip setting from a job.

Figure 20. Command remove-skip

This command takes only one parameter: the full name of the job.

remove-folder-skip: Removes the currently active skip setting from a folder.

Figure 21. Command remove-folder-skip

This command takes only one parameter: the full name of the folder.

skip-group-off: Removes the currently active skip setting from a skip group.

Figure 22. Command skip-group-off

This command takes only one parameter: the name of the skip group.